Conda

Conda is a package manager and environment management system, most often used to create virtual python environments.

On all Thayer/CS systems, you can use our system-installed version of Miniconda by loading the module:

module load miniconda3

By default, conda will download all packages and create environments in your home directory. While this is fine for small environments, in most cases this will use a large amount of space and quickly consume your home directory quota. To avoid this, we recommend re-directing this to a shared research or class directory, depending on your use.

To do this, you need to set some variables before creating a conda environment. Here is an example where a conda environment is created in /jumbo/researchgroup/myproject (this directory should already exist) with the latest tensorflow and its dependencies. If you use this template, be sure to adjust PROJECT_DIR as appropriate and understand what these commands are doing - don't just copy and paste.

module load miniconda3
export PROJECT_DIR=/jumbo/researchgroup/myproject
export CONDA_PKGS_DIRS=${PROJECT_DIR}/pkgs
conda create --prefix ${PROJECT_DIR} tensorflow

You can also use this same method if you have an environment file:

conda env create --prefix ${PROJECT_DIR} -f environment.yaml

The created environment can then be activated like this:

conda activate ${PROJECT_DIR}

In subsequent sessions, you will need to remember to load the module before activating the environment unless you have saved it as a default. See Modules for more information.

You will also need to either re-create the directory variable or just specify it explicitly. For example:

module load miniconda3
export PROJECT_DIR=/jumbo/researchgroup/myproject
conda activate ${PROJECT_DIR}

-- or --

module load miniconda3
conda activate /jumbo/researchgroup/myproject

If you create any other conda environments or re-create an existing one, you will also need to be careful to set the CONDA_PKGS_DIRS variable beforehand so that new packages will not end up in your home directory.

If you run conda install within an environment, also be sure that the CONDA_PKGS_DIRS variable is set properly. This does not get set automatically when activating an environment and will not carry over from login to login.

Special note regarding pip

If you plan to use pip to install python packages within your conda environment, you need to install the pip command first for it to run properly. You can either do this in your conda create command initially:

conda create --prefix ${PROJECT_DIR} python=3.10 pip

Or, you can install it later within your environment:

conda activate ${PROJECT_DIR}
conda install pip