Why is my job on Hellbender failing?

There are many reasons that your job may be failing on Hellbender, however, below are some more obvious causes of job failure. 

First, do not confuse your job getting requeued (while running in a requeue partition) with the job really failing.  

One of the very common causes of the job failure is that the job tried to use more memory than it had allocated – so it produced an Out of Memory error and was terminated by SLURM scheduler as the consequence of that.  

There can be myriads of causes created by user that cause the job to fail – the most typical ones being  

• typos in the submission script 

• pointing to the wrong location of the input files 

• having bad parameters in the input files 

• missing to load a module  

• activate a conda environment 

To avoid all such potential errors, read our article on how and why to get an interactive session where we explain how you should eliminate all such potential errors before you submit your job for the first time.  Article - How To: How and why to get... 

While unlikely, but it might happen that the node(s) on which your job is running goes through some glitch during your job’s execution. The glitch can be caused by other users’ jobs or simply by a hardware problem of the node. If you run across multiple nodes using MPI protocol, the glitch in the network hardware might cause the job to fail. All such events are extremely rare but can happen.  

In interactive session, you can have all your job output and errors show up on a screen, but when you submit a job, since there is no “screen”, to be able to analyze what went wrong you should have the following lines in your job submission script. 

#SBATCH --job-name=my_job # Name of the job  

#SBATCH --output=my_job_%j.out # File where STDOUT will be saved (%j inserts Job ID)  

#SBATCH --error=my_job_%j.err # File where STDERR will be saved (%j inserts Job ID) 

In the part of the submission script given above, you should change my_job to some name that is meaningful to you. The %j  will get replaced by the real job ID number once it is assigned to the job. In this way, you can submit multiple jobs from the same directory and have their output files nicely organized with job ID numbers in their names. Notice that the output and error files will appear in the directory in which you were in when you gave an sbatch command! – not where the submission script is located!  - for example, if you do  

sbatch  ../submit_test.sh 

the output and error text files will appear in the current directory from which you are launching the sbatch command, not in the directory in which the submit_test.sh script is located (in this case the parent directory  ../).     

This will also enable you to keep track of what happened with each job independently since each (slurm) output file will have a unique jobID in its name. 

For reference, please check the job state codes.  Slurm Workload Manager - Job State Codes

For additional assistance, please join our Research Facilitators in Office Hours.  Article - Getting Hellbender/RDE Help