Approved Software Only!
All software, including user installed software in userspace, is subject to UM system policy. UM system Business Policy Manual (BPM) 12004 states that all software must be reviewed and approved by IT. S&T IT provides the ITAM application as a method to request IT software review. Note: Software collections can be entered as one submission for approval.
Python Virtual Environments
Python virtual environments function similarly to lmod on the Mill: they are designed to reduce versioning and dependency conflicts when working on multiple projects. Each virtual environment can have its own packages that are different or the same from other virtual environments as well as the system version. There are two primary methods for managing these virtual environments: conda and python.
There are pros and cons of each method but as a general rule:
Python: Lightweight implementation that only needs python and pip. Only manages python packages.
Conda: Heavier implementation; needs Conda installed (Anaconda/Miniconda/Mamba) but can manage both python packages and system packages (like CUDA, R, etc.).
Conda
Both Anaconda and Miniconda are on the Mill for your use. This example will use Miniconda.
To make a virtual environment with conda, begin by loading the miniconda module with:
module load miniconda
To enter the conda base environment, you can run:
eval "$(conda shell.bash hook)"
This will put you into the conda base environment and you should see (base) added to the front of your terminal command line.
From the (base) environment you can create your own custom conda environment using:
conda create my_environment_name
and enter your newly created environment using:
conda activate my_environment_name
Note: You must load miniconda, run the shell hook command, and enter this environment within your SLURM jobs in order to access it within jobs.
The (base) at the start of your terminal command line should change to (my_environment_name) to confirm that you have entered your environment. Within your environment you can install python or system packages using:
conda install <package>
for example:
conda install scipy
will install the popular scipy package. For more details, consult the conda docs
Python
To create a virtual environment with python's native venv command, we begin by selecting our python version using lmod. Several versions of python are available on the Mill. 3.6 is the default system version and several newer versions are available through lmod. In this example, we will load python 3.11 with:
module load python/3.11.7
We then create a virtual environment, storing the environment in our home directory with:
python3 -m venv ~/my_environment_name
This will create a folder for the environment. To activate the environment, use:
source ~/my_environment_name/bin/activate
Note: You must source this file to enter this environment within your SLURM jobs in order to access it.
You can confirm you are in the environment when your terminal gets the prefix (my_environment_name). From here, you can use pip to install python packages into your virtual environment. For example, to install scipy you can run:
pip -m install scipy