Body
SLURM Arrays
SLURM Arrays are a method to take many small jobs that do the same thing and submit them as one big job. An array job has a single job number, and array steps are indicated by a second number after an underscore like Job_Array (1000_1, 1000_2, etc)
Controlling an Array
Array jobs are submitted just like normal jobs but include the sbatch directive #SBATCH --array=[] to define the array.
Array numbering can be given as a list or a range:
| Type |
Command |
Array Jobs |
| Lists |
array=[1,6,12] |
Jobs 1,6, and 12 |
| Sequential |
array=[0-31] |
Jobs 1 through 31 |
| Stepped |
array=[1-31:2] |
Odd numbered jobs from 1 to 31 |
The % symbol can be used after the array numbering to control the maximum number of array jobs that can run at the same time. This can be useful for limiting your usage of the cluster, or for preventing a non-compute resource like API calls or disk I/O from being overwhelmed by many jobs running at the same time. The % symbol is placed after the array list like:
#SBATCH --array=[1-10]%2
Using SLURM Variables with Arrays
An array job runs the same script for each member of the array. Therefore, it us useful to have some unique identifier so that each array job can read a unique input and write a unique output. This can accomplish this using several SLURM variables.
The example SLURM script in the next section shows two kinds of variables.
Within the SBATCH block, SLURM supports the use of variables tagged with % , used in the line #SBATCH --out=slurm_%A-%a.out. The %A symbol will return the job number of the array job and the %a symbol will return the array task number. Combined, these variable can give each SLURM output file a unique filename.
When the job is running, we must use bash environment variables that SLURM sets at the start of the job.
mpirun -np $SLURM_NTASKS lmp -in input_$SLURM_ARRAY_TASK_ID.in > output_$SLURM_ARRAY_TASK_ID.out
will read in an input file that has the same number as the array job (input_1.in and so on) and will write an output file with the same array number (output_1.out and so on).
An Example Array Job
#!/bin/bash
#SBATCH --job-name=lammps_example
#SBATCH --array=[1-10]%2
#SBATCH --nodes=1
#SBATCH --ntasks=4
#SBATCH --mem=12G
#SBATCH --time=0-00:02:00
#SBATCH --partition=requeue
#SBATCH --out=slurm_%A-%a.out
module load lammps/2Aug2023_kokkos
mpirun -np $SLURM_NTASKS lmp -in lammps_$SLURM_ARRAY_TASK_ID.in > test_output_$SLURM_ARRAY_TASK_ID.out