Scratch Space

Each node has a local scratch space of at least 1TB located at /tmp. Each node only has access to its own local scratch, and it is intended to be used for temporary file storage during jobs on that node. For jobs running across multiple nodes, we recommend the networked scratch space which is available at /scratch/$USER.

SLURM does not delete local scratch for you; you must configure your job to clean up after itself. We recommend using SLURM environment variables to make it easier to consistently clean up the correct files (see the example below)

How to Use Scratch

Your jobs may need to be configured to use scratch. In this example, an environment variable is used to define the location of the scratch space(local scratch, /tmp). It is injected into the the QE input file using the find-and-replace functionality of sed.

At the end of the calculation, the wavefunction files (*.wfc) are copied to a permanent location in my home directory and the files in scratch are deleted.

#!/bin/bash
#SBATCH --job-name=scratch_example
#SBATCH --nodes=1
#SBATCH --ntasks=4
#SBATCH --mem=12G
#SBATCH --time=0-00:05:00 
#SBATCH --partition=requeue
#SBATCH --mail-type=begin,end,fail,requeue
#SBATCH --out=slurm_%j.out 


module load quantum-espresso/7.2
export OMP_NUM_THREADS=1

local_dir="/tmp/bbkbk"

#replace a string in the QE input with the correct working directory
sed -i "s|__SCRATCH_DIR__|$local_dir|g" QE_pw.in

mpirun -np $SLURM_NTASKS pw.x -in QE_pw.in > QE_pw_${SLURM_JOB_ID}.out


save=~/$SLURM_JOB_ID
mkdir $save

cp QE_pw_${SLURM_JOB_ID}.out $save

cd ${work_dir}/pwscf.save/
cp *wfc* $save

cd ../../
rm -rf {$work_dir}/