Job Dependencies

Job dependencies are used to defer the start of a job until the specified dependencies have been satisfied. They are specified with the --dependency option to sbatch in the format

sbatch --dependency=<type:job_id[:job_id][,type:job_id[:job_id]]> ...

Dependency types are:

Condition Explanation
after:jobid[:jobid…] job can begin after the specified jobs have started
afterany:jobid[:jobid…] job can begin after the specified jobs have terminated
afternotok:jobid[:jobid…] job can begin after the specified jobs have failed
afterok:jobid[:jobid…] job can begin after the specified jobs have run to completion with an exit code of zero (see the user guide for caveats).
singleton jobs can begin execution after all previously launched jobs with the same name and user have ended. This is useful to collate results of an array job or similar.

To set up pipelines using job dependencies the most useful types are afterany, afterok and singleton. The simplest way is to use the afterok dependency for single consecutive jobs. For example:

$ sbatch <further arguments>
Submitted batch job 253507
$ sbatch <further arguments> --dependency=afterok:253507

Now when job1 ends with an exit code of zero, job2 will become eligible for scheduling. However, if job1 fails (ends with a non-zero exit code), job2 will not be scheduled but will remain in the queue and needs to be canceled manually.

As an alternative, the afterany dependency can be used and checking for successful execution of the prerequisites can be done in the jobscript itself.

The following ${id1##* } constructs are there because sbatch returns a statements like Submitted batch job <JOBID>.

# first job - no dependencies
jid1=$(sbatch -p short -A <your account> --mem=12g --cpus-per-task=4 job1.script)
# multiple jobs can depend on a single job
jid2=$(sbatch  -p short -A <your account> --dependency=afterany:${jid1##* } --mem=20g job2.script)
jid3=$(sbatch  -p short -A <your account> --dependency=afterany:${jid1##* } --mem=20g job3.script)
# a single job can depend on multiple jobs
jid4=$(sbatch  -p short -A <your account> --dependency=afterany:${jid2##* }:${jid3##* } job4.script)
# a single job can depend on an array job
# it will start executing when all arrayjobs have fin.scripted
jid5=$(sbatch -p short -A <your account> --dependency=afterany:${jid4##* } job5.script)
# a single job can depend on all jobs by the same user with the same name
jid6=$(sbatch -p short -A <your account> --dependency=afterany:${jid5##* } --job-name=dtest job6.script)
jid7=$(sbatch -p short -A <your account> --dependency=afterany:${jid5##* } --job-name=dtest job7.script)
sbatch -p short -A <your account> --dependency=singleton --job-name=dtest job8.script
#show dependencies in squeue output:
squeue -u $USER -o "%.8A %.4C %.10m %.20E"

Sometime a dependency condition cannot be satisfied. For example when asking for a predecessor job to be sucessfully done (afterok:<jobid>) and it fails.

In such a case --kill-on-invalid-dep=<yes|no> can be specified to sbatch.

  • start/working_on_mogon/workflow_organization/job_dependeny.txt
  • Last modified: 2020/04/16 11:52
  • by jrutte02