Body
Creating a conda virtual environment on HPC (on Hellbender)
Why Conda environments?
Building a Conda virtual environment(s) enables you to install software on a cluster as a regular user (without the root privileges). It gives you freedom and flexibility to create different types of environments for your workflows – for example you can build a Conda environment for working in R/RStudio or you can build one with pytorch installation.
The limitation of Conda environments is that to install certain software it must exist as the Conda (or pip) package. However, sometimes the purpose of the Conda environment might be such that you install all the necessary libraries so that you can compile some software from its source code (for example from github) – i.e. A Conda environment can be created to provide all the libraries necessary for compiling or linking some other software from its source code.
Prerequisites:
Before you create any Conda environment on the Hellbender cluster you should modify a file called .condarc (notice the leading dot in the name of the file) which may not exist in your home directory on the Hellbender.
That file should look like this:
envs_dirs:
- /mnt/pixstor/data/${USER}/miniconda/envs
pkgs_dirs:
- /mnt/pixstor/data/${USER}/miniconda/pkgs
It is very important to have this file in place before creating your first Conda environment. This file instructs Conda to create Conda environments in your data folder rather than in your home folder – which would be the default option. Your data folder can hold 500 GB while your home can store only 50 GB of data, and Conda environments can get quite large.
Before creating the first environment we will describe the workflow that one uses when creating a Conda environment.
-
Get an interactive session on the cluster with lots of memory (40GB or more) and a few hours of running time. Make sure it is NOT in the requeue or gpu_requeue partition. Requesting the general partition works well.
-
Load a module with appropriate Conda software (can be Conda, mamba Anaconda or some similar name – check modules)
-
Create a Conda environment with certain name
-
Activate that environment that you just created
-
Install the packages that you need for your workflow via Conda install or pip install commands
Once you have a Conda environment created and have installed all the packages that you need for future use of that environment, load a Conda module and activate the environment of your choice. Once the environment is active, you can also install additional packages and you can continue modifying your environment at any time.
Creating environment – steps
Get an interactive session on a cluster. Reminder: do not run on a login node. You can do it through Open On Demand via web interface but this article guides you through the command line terminal (make sure to specify the partition as general because the default one is the requeue in which your session can get terminated.)
salloc --partition=general -N 1 -n 8 --mem=40G --time=4:00:00
There are several modules on Hellbender that can be used as a Conda – providing Conda command - we will use mamba (there is also a module called miniconda3)
module load mamba/v2.4.0
after loading the module, you can try
conda env list
Which should list all the environments that are available to you. If this is the first time for you running Conda you will probably see just the base environment.
To create an environment in the default location you should do
conda create -n ENVNAME
Replace ENVNAME with the name that you want for your environment – choose something descriptive so that you know what that environment is about. That environment will be created in the location specified in the .condarc file which we mentioned earlier in this document.
If you would like to create a Conda environment in a specific custom location, you can do it like this:
conda create -p /path/to/location
Now that you created the environment to install additional packages in the environment or just to use the environment you will need to activate it.
In the case of the first environment created in the default location you will do it by doing
source activate ENVNAME
while in the second case if the environment was created in the custom location, you would activate it by doing
source activate /path/to/location
When the environment is activated, you should see it in your command prompt line as an environment name or a path to the environment given in parentheses – something like this:
[pawprint@c021 ~]$ source activate vina
(vina) [pawprint@c021 ~]$
Once you are “inside” your virtual environment you can run Conda (or pip) install for packages you would like to install in the environment and it will install them with their dependencies inside this environment.
For example the command:
conda install conda-forge::rstudio-desktop
For example this will install rstudio-desktop package (which is RStudio) from the Conda-forge channel. Notice that we are not installing R since we are counting on RStudio bringing R as a dependency. After this is done, you can simply install additional R packages through Conda – rather than installing them through R itself – which requires compiling while Conda packages come precompiled, so installation is much faster and simpler. Typically, the names of R packages in Conda are r-name_of_the_package. So, you can find them by searching on Google for “conda r-tidyverse" to find a tidiverse R package. The first result is the Conda r-tidyverse package in Conda-forge channel: https://anaconda.org/channels/conda-forge/packages/r-tidyverse/overview
If you would like to execute a code that depends on the packages you installed, you will need to be sure that you are inside your virtual environment and that the environment is active.
To deactivate environment do:
source deactivate
Submitting a job that uses your Conda environment
A SLURM job submission script (my_job.sh) that would run your python script inside your conda environment should look similar to this:
#!/bin/bash
#SBATCH --job-name=my_job
#SBATCH --output=output.txt
#SBATCH --ntasks=1
#SBATCH --time=01:00:00
#SBATCH --mem=4G
# Load the Mamba module and activate environment
module load mamba
source activate ENVNAME
# Run the Python script
python my_script.py
Submit this script with the sbatch command:
sbatch my_job.sh
More on Conda
For how to install additional packages in your environment, how to delete the existing environment, how to transfer (copy) environment, and how to register environment for the Jupyter Notebook see the article on Handling Conda environment. Article Content - How To: How do I handle Co...