Nvidia and CUDA Policy

Currently we have Volta and Hopper generation GPUs available on the Mill. Please note that the Volta cards are nearing the end of support from Nvidia.

The current policy is to maintain two CUDA drivers on the Mill, one for each GPU type. They can be found as modules in lmod. Currently, those are CUDA 12.9 for use with the Volta cards and CUDA 13.3 for use with the Hopper cards.

Because we are maintaining two different CUDA versions for two different hardwares, the CUDA installs are maintained on the individual GPU nodes. This means that while you can load a CUDA module anywhere, it will only function if you are currently on a GPU node with the hardware that matches that module you loaded.

Old CUDA Versions

If you need support for an old CUDA version, we recommend using conda as an environment manager and installing the CUDA version that you need in your home directory. We recommend using miniconda for this.

High Resource Use

Downloading and building conda environments uses significant compute resources. Do not perform this step on a login node.

To begin, switch to a compute node. This should take about 30 minutes but feel free to request more time if this is your first time setting up a conda environment like this. To request a 30 minute interactive session, use:

salloc --partition=interactive --nodes=1 --ntasks=4 --mem=16g --time=0-00:30:00

To use miniconda, begin by loading the module with:

module load miniconda/miniconda3

Instead of using conda init to automatically activate your shell, we recommend manually activating the base conda environment with:

eval "$(conda shell.bash hook)"

You should see (base) appear before your command line to indicate this step was successful. Next, create a virtual environment for CUDA, in this example CUDA 12.8 and python 3.10, with:

conda create --name cuda_12.8 python=3.10

The CUDA version and the python version can be changed to meet your individual needs. Confirm the selection and let conda download the packages it needs to create your python environment. Once it is done, you can enter the environment with:

conda activate cuda_12.8

You should see the (base) label at the front of your command line change to (cuda_x.x). Once you are in the new environment, you can install a specific CUDA, in this example 12.8.1, with:

conda install cuda=12.8.1 -c nvidia/label/cuda-12.8.1

You can verify that the environment has been successfully set up by using:

nvcc --version

Which should report the version you selected instead of the current system-wide version. To use this in a compute job, ensure they you add commands to your job submission to correctly enter your conda environment.