Submitting Jobs

salloc, srun, and sbatch are user to reserve job slots, run tasks interactively or submit a job to SLURM.

Basically, most scripts are submitted using sbatch:

$ sbatch <scriptfile>

where the scriptfile holds several special options preceded with #SBATCH, e.g.

#!/bin/bash # or could be another language
 
#SBATCH -J my_job_name 
 
# etc.

Detailed information on important options follow below.

An extensive documentation on the salloc, srun and sbatch command can be found in the SLURM documentation: salloc, srun, sbatch.

There are some important parameters that are always required or at least recommended to use:

To specify files for output, error and input consider the following:

You may use one or more replacement symbols, which are a percent sign “%” followed by a letter (e.g. %j).
For example, job%4j.out yields job0128.out

Other important parameters / features on mogon include:

• Submitting job arrays and specific considerations
• Using the ramdisk
• Using Local scratch space
• Specifying runtimes runtime
• Using GPU Queues

In order to test or visualize it is sometimes handy to allocate resources for an interactive job. SLURM provides two commands to facilitate this: srun and salloc.

Please note, that our policies forbid using login nodes for prolonged interactive work, which may inhibit the workflow of others. You can use interactive reservations as described in the following paragraphs, instead:

To get an interactive shell you can just run:

$ srun --pty -p <partition name> -A <account name> bash -i

You can reserve more time, memory or cpus as well.

To quote the official documentation: salloc is used to allocate a Slurm job allocation, which is a set of resources (nodes), possibly with some set of constraints (e.g. number of processors per node). When salloc successfully obtains the requested allocation, it then runs the command specified by the user.

An example:

$ salloc -N 2 -p nodeshort -A zdvhpc
salloc: Granted job allocation 3242
salloc: Waiting for resource configuration
salloc: Nodes a[0001-0002] are ready for job
$ # now you can use two nodes and start the desired application
$ 
$ # do not forget to type 'exit' or else you will be working in a subshell
$ exit

You have to prepare a job script to submit jobs using the sbatch command.

You can pass options to sbatch directly on the command-line or specify them in the job script file. Slurm will reject Jobs that do not set -A (account) and -p (partition).

To submit your job use

login21$ sbatch myjobscript

#!/bin/bash
#-----------------------------------------------------------------
# Example SLURM job script to run serial applications on Mogon.
#
# This script requests one core (out of 64) on one node. The job
# will have access to all the memory in the node.  Note that this
# job will be charged as if all 64 cores were requested.
#-----------------------------------------------------------------
 
#SBATCH -J mysimplejob           # Job name
#SBATCH -o mysimplejob.%j.out    # Specify stdout output file (%j expands to jobId)
#SBATCH -p nodeshort             # Queue name
#SBATCH -N 1                     # Total number of nodes requested (16 cores/node)
#SBATCH -n 1                     # Total number of tasks
#SBATCH -t 00:30:00              # Run time (hh:mm:ss) - 0.5 hours
 
#SBATCH -A account               # Specify allocation to charge against
 
# Load all necessary modules if needed (these are examples)
# Loading modules in the script ensures a consistent environment.
module load gcc/6.3.0
 
# Launch the executable
<myexecutable>

#!/bin/bash
#-----------------------------------------------------------------
# Example SLURM job script to run MPI Job on Mogon.
#
# This script requests 128 cores on two node. The job
# will have access to all the memory in the nodes.  
#-----------------------------------------------------------------
 
#SBATCH -J mympijob              # Job name
#SBATCH -o mympijob.%j.out       # Specify stdout output file (%j expands to jobId)
#SBATCH -p nodeshort             # Queue name
#SBATCH -N 2                     # Total number of nodes requested (64 cores/node)
#SBATCH -n 128                   # Total number of tasks
#SBATCH -t 00:30:00              # Run time (hh:mm:ss) - 0.5 hours
 
#SBATCH -A account               # Specify allocation to charge against
 
# Load all necessary modules if needed (these are examples)
# Loading modules in the script ensures a consistent environment.
module load gcc/6.3.0
module load mpi/intelmpi/2017
 
# Launch the executable
srun -n 128 <myexecutable>