Measuring Job Efficiency

Measuring Job Efficiency

Job efficiency is important because the Mill is a shared resource. If you request resources and utilize them poorly, those resources are not available for another user.

Additionally, asking for more resources than your need increases the time that your job will wait in the queue for those resources. Beyond the fact that larger jobs are harder to schedule, each account has a fairshare score, which measures the amount of resources that you use compared to others. The more resources you use, the lower your priority for future jobs.

SLURM Efficiency Tool - seff

SLURM has a tool for reporting the efficiency of a job. It is designed to be run after a job has finished, and may report incorrect results if give a running job. The basic command is:

seff <job #>

Which will output:

[<user>@mill-login-p1 ~]$ seff 2228540
Job ID: 2228540
Cluster: mill
User/Group: <user>/<user>
State: CANCELLED (exit code 0)
Nodes: 1
Cores per node: 6
CPU Utilized: 00:07:32
CPU Efficiency: 0.81% of 15:34:12 core-walltime
Job Wall-clock time: 02:35:42
Memory Utilized: 1.09 GB
Memory Efficiency: 9.09% of 12.00 GB

This output can be used to determine the resources your job actually needs to run. The CPU efficiency measures the percentage of time that each CPU core was active. The memory efficiency is calculated as the maximum memory used by the job divided by the memory request. The standard output does not report time efficiency directly. Time efficiency can be obtained by dividing the wall-clock time by the requested time.

ReportSeff

There is a module that can be loaded using lmod to print reports from seff. Load it with:

module load reportseff

The following command string can be used to print a formatted report of all jobs run by a user in a certain time period (in this example, the last 31 days):

reportseff -u <user> --format=JobID,User,State,Partition,TimeLimit,Elapsed,TimeEff,ReqCPUS,CPUEff,ReqMem,MemEff --since=now-31day

  JobID    User      State      Partition    Timelimit   Elapsed    TimeEff   ReqCPUS   CPUEff   ReqMem   MemEff 
 2222252   <user>   CANCELLED       gpu       06:00:00    00:00:00    0.0%        4       ---      16G      0.0%  
 2223071   <user>    TIMEOUT    interactive   00:30:00    00:30:16   100.9%       4       0.8%     16G      7.8%  
 2225052   <user>   COMPLETED   interactive   00:30:00    00:00:35    1.9%        4       ---      16G      0.2%  
 2225323   <user>   COMPLETED   interactive   03:00:00    01:15:13    41.8%       4      24.8%     16G      7.4%  
 2225406   <user>   CANCELLED       gpu       01:00:00    00:15:09    25.2%       4      10.5%     16G     16.3%  
 2227331   <user>    FAILED     interactive   02:00:00    01:58:45    99.0%       4       0.4%     16G      0.2%  
 2227420   <user>   COMPLETED   interactive   02:00:00    00:00:12    0.2%        4       ---      16G      0.0%  
 2227423   <user>   COMPLETED   interactive   03:20:00    01:31:46    45.9%       4      11.8%     16G      9.0%  
 2227691   <user>    FAILED       requeue     00:10:00    00:00:00    0.0%        4       ---      16G      0.0%  
 2227692   <user>    FAILED         gpu       00:10:00    00:00:00    0.0%        4       ---      16G      0.0%  
 2227693   <user>    FAILED         gpu       00:10:00    00:00:00    0.0%        4       ---      16G      0.0%  
 2227695   <user>    FAILED         gpu       00:10:00    00:00:00    0.0%        4       ---      16G      0.0%  
 2227696   <user>   COMPLETED       gpu       00:10:00    00:00:05    0.8%        4       ---      16G      0.0%  
 2228534   <user>   COMPLETED   interactive   01:00:00    00:02:07    3.5%        4      16.3%     12G     19.2%  
 2228540   <user>   CANCELLED   interactive   04:00:00    02:35:42    64.9%       6       0.8%     12G      9.1%  
 2228741   <user>   COMPLETED     requeue     00:30:00    00:00:44    2.4%        1       ---      800M     0.3%  
 2230651   <user>    TIMEOUT       debug      01:00:00    01:00:09   100.2%       3       0.2%      6G     11.3%

If you have many jobs, it may be more convenient to add the -p flag to make the output parseable and output to a .csv file instead of the terminal:

reportseff -p -u <user> --format=JobID,User,State,Partition,TimeLimit,Elapsed,TimeEff,ReqCPUS,CPUEff,ReqMem,MemEff --since=now-31day > jobs.csv

GPU Efficiency

SLURM lacks the same level of information as CPU/MEM usage when reporting on GPU use. To view your GPU efficiency, you may use nvidia-smi but that is designed to report real time information and is only available when you are logged into a GPU node running a job.

Print Article