What is Requeue?

Idle compute capacity is available in requeue for non-paid users. Requeue is the default partition on the Mill; it is the queue where jobs are sent if no partition is specified.

If a paid user submits a job that needs resources used by a requeue job, the requeue job will be stopped and sent back to the queue and the paid job will begin immediately.

What is the benefit of Requeue?

Because the Requeue partition has access to idle nodes, there is more total capacity. Using idle resources increases the total efficiency of the Mill and provides a non-paid option for new users to evaluate the Mill for their research workflows.

The trade-off for this is that your job may be interrupted in an unpredictable manner by a paid user.

How can I use Requeue effectively?

There are two main ways to increase your efficacy with the requeue partition:

Submit short jobs

Short jobs make it unlikely for any individual job to be interrupted and minimize the cost associated with a job being interrupted. For example, if you submit 1 minute jobs to requeue, it is highly unlikely that any individual job is interrupted by a paid user submitting a job. Additionally, even if your job is interrupted, you will lose at most 59 seconds of computational work.

Contrast this with a 2-day-long job. It is highly possible that a paid user will run a job within that 2-day window. Compounding this, you potentially stand to lose many hours of computational work if your job is interrupted.

Checkpointing Jobs

Some calculations support the use of checkpointing, where the current state of the computation is saved to a persistent file at pre-determined points/intervals. These files can be used to restart an interrupted calculation and limit the amount of progress that is lost when a job is requeued. Checkpointing is a costly process, as the program must stop work, then package and write the files to disk. You should carefully weigh the cost of writing checkpoint files against the amount of progress that you are willing to lose on a requeue event.

A LAMMPS Requeue Example

The following example demonstrates how to use checkpointing in the MD software LAMMPS to minimize the potential effect of being requeued.

The example consists of two files, the SLURM submission script and the LAMMPS input file. Both must be written in a way that the job can run correctly the first time as well as correctly restart when requeued.

LAMMPS_SLURM.slurm
#!/bin/bash
#SBATCH --job-name=lammps_example
#SBATCH --nodes=1
#SBATCH --ntasks=4
#SBATCH --mem=12G
#SBATCH --time=0-01:00:00 
#SBATCH --partition=requeue
#SBATCH --out=slurm_%j.out

     
module load lammps/2Aug2023_kokkos


if compgen -G "restart.data.*" > /dev/null; then
	mpirun -np $SLURM_NTASKS lmp -var restart 1 -in  requeue.in > requeue.out
else
	mpirun -np $SLURM_NTASKS lmp -var restart 0 -in  requeue.in > requeue.out
fi

The SLURM submission script uses conditional logic to check if a restart file exists. If it does, it passes into LAMMPS a variable flag (restart) using the -var command line switch. LAMMPS will automatically load the restart file with the largest timestep value appended to the end, so we only need to identify if any restart file exists.

On the initial run, there will be no existing restart files, so the restart variable will be set to false. This will trigger special behavior in the LAMMPS input script for the initial run.

LAMMPS_requeue.in
#Many of the following sections are not needed on the restarted runs, as the restart
#file contains some system information. However, it is not problematic to redeclare
#them and it makes the input more readable.
#----------------------  Define Simulation -------------------------
units           metal
atom_style      atomic
boundary        p p p
timestep        0.0005

#This if statement catches the command line variable ${restart} and will either
#continue from the last restart file or load the initial data on the first run

if "${restart} == 1" then "read_restart restart.data.*" else "read_data initial_data.data"


#------------------- Force Field Information ---------------------
pair_style      eam
pair_coeff      * * ../Ni_u3.eam

neighbor        2.0 bin
neigh_modify    every 1 delay 0 check yes

variable max_step equal 15000
variable        t equal 300
#----------------- Advance Timesteps ----------------------------

#The restart command triggers writing a restart file on timesteps that are
#multiples of N restart 5000 restart.data


fix 1 all nvt temp $t $t  $(1000*dt)
thermo 250
run ${max_step} upto

#On step 15,000 write the final data file.
if "${ending_step} == ${max_step}" then "write_data final_data.data"

The LAMMPS input script uses conditional logic to perform a different data loading operation depending on if the restart variable is set to true or false.

The restart command writes a restart file every 5,000 timesteps. These files can be used to restart a requeued job, losing at most 4,999 timesteps of work.

It also uses a "upto" setting for the timesteps to ensure that no matter where it restarts, it will only perform the intended number of maximum timesteps.

Conclusion

With some small adjustments to your job inputs and SLURM scripts, you can write jobs that are resilient to being requeued and minimize the potential data loss. This allows you to take advantage of the idle capacity available in the requeue partition.