SLURM is the job scheduler on the cluster. There are many researchers running many jobs at the same time on the cluster. SLURM's job is to utilize the hardware on the cluster as efficiently as possible. To this end, you will rarely run jobs yourself on the cluster. Instead, you will give your job to SLURM and it will run the job for you when it can efficiently match the resources your job needs with hardware on the cluster.

The SLURM Script

For SLURM to run your job, you will write a script that contains two main parts. The first part is information for SLURM that tells it what kind of resources you want for your job. These are called SLURM directives and are at the top of your job submission script, starting with #SBATCH. After the SLURM directives, you can enter terminal commands that you want SLURM to run for you. Generally this will include loading modules with lmod and then executing some program, likely with mpi.

It is common practice to work from a base SLURM script that copy and modify slightly based on the needs of each individual job. Feel free to use the following script as a basis for creating your own SLURM scripts.

An Example Script

#!/bin/bash
#SBATCH --job-name=lammps_example
#SBATCH --nodes=1
#SBATCH --ntasks=4
#SBATCH --mem=12G
#SBATCH --time=0-00:02:00 
#SBATCH --partition=general
#SBATCH --mail-type=begin,end,fail,requeue
#SBATCH --out=slurm_%j.out 


module load lammps/2Aug2023_kokkos

mpirun -np $SLURM_NTASKS lmp -in lammps.in > test_output_$SLURM_JOB_ID.out

This example script requests 1 node with 4 CPU cores (ntasks) and 12GB of memory (mem) to run for 2 minutes in the general partition. SLURM will email me when my job changes state and it will write an output file that has the job number appended to it (%j will be replaced by the job number that is assigned when this script is submitted).

Once it is my job's turn to run (after waiting in the queue), it will load the 2 Aug 2023 version of LAMMPS and run it in parallel. Two environment variables are included in the script for convenience. $SLURM_NTASKSreferences the value that was defined with #SBATCH --ntasks=4. $SLURM_JOB_ID functions the same as %j did in the SLURM directive step, but must be referenced as an environment variable to work outside the SLURM directives section.

Submitting a Job

Once your SLURM script is prepared, the process to submit it is straightforward. If you have a SLURM script named submission.slurm, it can be submitted with

sbatch submission.slurm

If this command is successful, SLURM will output the job ID that was assigned to your terminal. You can also check your jobs using squeue. We include the user flag to filter out all jobs that don't belong to you

squeue -u *name*

This will return a list of all jobs you have submitted that are either waiting in the queue or still running. If you spot an error in your job or get bad output you can cancel jobs using the ID displayed by squeue

scancel *job ID*