Slurm Array Jobs#
If you have a script you need to run many times with different parameters there’s an easy way to do it, a Job Array.
Job arrays offer a mechanism for submitting and managing collections of similar jobs quickly and easily.
Submitting a Job Array#
A job array can be submitted by adding the following to your submission script:
#SBATCH --array=x-y
where x and y are the array bounds
A job array will be created on submission with a number of independent jobs (array tasks) that correspond to the defined array.
Instead of providing a range, you can use a comma-separated list of task numbers. For example, to rerun failed jobs in a previously submitted job array you could use:
#SBATCH --array=2,7,11,12,21,33,40
Limiting the number of simultaneous tasks#
If you have a large number of tasks to run, it’s a good idea to limit the number of tasks that will run at the same time.
You can use the %N suffix where N is the number of active tasks running.
#SBATCH -a 1-100%10
This command will create an array job where only 10 tasks will be running at once.
The $SLURM_ARRAY_TASK_ID VARIABLE#
SLURM will assign a $SLURM_ARRAY_TASK_ID variable to each task specified in the –array directive in your script.
The $SLURM_ARRAY_TASK_ID for the first taks will be equal to 1, the second task equal to 2 and so on, up to the final task in your array.
You can use this variable to handle input and output files for that task.
For example, if you had a series of input files named sample_1 to sample_100, you could replace the line where you specify the input file with sample_$[SLURM_ARRAY_TASK_ID].
Each of the array tasks would then pick a different file as it’s input depending on it’s $SLURM_ARRAY_TASK_ID value.
Output and Error files#
Similar to the Task ID variable, we can use %A and %a as replacement strings for the master job ID and the task ID respectively.
e.g:
#SBATCH --output=test.%A_%a.out
#SBATCH --error=test.%A_%a.error
So if you’d submitted a job with Job ID 10, the output for array task 15 would be:
Test.10.15.out
The error log entry is optional, it will be added to the standard out if not specified.
Job Emails#
Emails in slurm apply to the whole job, not just the individual array tasks
If you want to be emailed about each task in the array use the following in your batch script:
#SBATCH --mail-type=BEGIN,END,FAIL,ARRAY_TASKS