User Tools

Site Tools


slurm_submit

Submitting Jobs

With SLURM there are three commands to reserve resource allocaction, resp. to submit jobs: salloc, srun and sbatch. They are used to submit jobs (sbatch), to reserve allocation for interactive tasks (salloc) and to run so-called job-steps (see below) or small interactive jobs (srun).

Extensive command documentation can be found in the man pages for each command, e.g $ man sbatch.

Slurm will reject Jobs that do not set -A (account) and -p (partition). See Accounts and Accounting for details.

Important parameters

There are some important parameters that are always required or at least recommended to use:

Mandatory Without these your jobs are rejected
-A <account> The project account (not your user id) your job should be accounted for.
Use sacctmgr -s list user $USER format=user%10,account%20 to retrieve the associated accounts. See Accounts and Accounting for details.
-p <partition> The partition your job should run in.
Available partitions.

On Mogon II a third important parameter is present: You may select the CPU type to be either skylake or broadwell for the Skylake and Broadwell nodes, respectively. If the architecture is not relevant for your application, select anyarch.

This can be set with:

-C <selection list> or –constrain=<selection list>

to sbatch (on the command line or within a jobscript).

The defaults are1)

If nothing is specified you'll get broadwell except for the himster2-partition where it's going to be skylake. On the bigmem-partition it will depend on your requested memory per node.

Necessary Lack of Options might lead to default settings and/or unexpected allocations
-n <tasks> Controls the number of tasks to be created for the job (=cores, if no advanced topology is given).
-N <nodes> The number of nodes you need
-t <minutes>
-t <hour>:<min>:<sec>
Set the runtime limit of your job (up to the maximum allowed by the selected partition). See the description for host model considerations.
-J <jobname> Sets an arbitrary name for your job that is used for listing of jobs.
Optional
--task-per-node Controls the maximum number of tasks per allocated node.
--cpus-per-task No. of cpus per task
--mem The amount of memory per Node. Different units can be specified using the suffix [K|M|G|T] (default is 'M' for MegaByte). See the Memory reservation page for details and hints, particularly with respect to partition default memory settings.
--mem-per-cpu Amount of memory per CPU. See above for the units.

To specify files for output, error and input consider the following:

-o <filename> Will direct stdout, stderr into one file. 2)
-o <filename>.log -e <filename>.err Will direct stdout to the log file and stderr to the error log file.
-i <filename> Instruct Slurm to connect the batch script's standard input directly to the file name specified.

Receiving mail notifications:

--mail-type= Specifies which types of mails a user wants to receive. Can be any of NONE, BEGIN, END, FAIL, REQUEUE or ALL
--mail-user=<username>@uni-mainz.de Specifies the receiving mail address. We highly recommend taking an internal address rather relying on an a third party service.

You may use one or more replacement symbols, which are a percent sign “%” followed by a letter (e.g. %j).
For example, job%4j.out yields job0128.out

%A Job array's master job allocation number.
%a Job array ID (index) number.
%J jobid.stepid of the running job. (e.g. “128.0”)
%j jobid of the running job.
%s stepid of the running job.
%u User name.
%x Job name.

Other important parameters / features on mogon include:

Once a job has been submitted you can get information on it or control with this list of commands.

Interactive Jobs

In order to test or visualize it is sometimes handy to allocate resources for an interactive job. SLURM provides two commands to facilitate this: srun and salloc.

Please note, that our policies forbid using login nodes for prolonged interactive work, which may inhibit the workflow of others. You can use interactive reservations as described in the following paragraphs, instead:

Simple Interactive Work with ''srun''

To get an interactive shell you can just run:

$ srun --pty -p <partition name> -A <account name> bash -i

You can reserve more time, memory or cpus as well.

Allocation with ''salloc''

To quote the official documentation: salloc is used to allocate a Slurm job allocation, which is a set of resources (nodes), possibly with some set of constraints (e.g. number of processors per node). When salloc successfully obtains the requested allocation, it then runs the command specified by the user.

An example:

$ salloc -N 2 -p nodeshort -A zdvhpc
salloc: Granted job allocation 3242
salloc: Waiting for resource configuration
salloc: Nodes a[0001-0002] are ready for job
$ # now you can use two nodes and start the desired application
$ # e.g. 
$ srun -N1 [other parameters] <some application confined on one node>
$ srun [other parameters] <some application triggered on all nodes>
$ srun [other parameters] <some mpi application>
$ # do not forget to type 'exit' or else you will be working in a subshell
$ exit

During a session with salloc you may login to the allocated nodes (with ssh) and monitor their behaviour. This can be handy to estimate memory usage, too.

Using sbatch

You have to prepare a job script to submit jobs using sbatch (for interactive jobs see srun or salloc).

You can pass options to sbatch directly on the command-line or specify them in the job script file.

To submit your job use

$ sbatch myjobscript

Trivial example - single core job

#!/bin/bash
#-----------------------------------------------------------------
# Example SLURM job script to run single core applications on 
# Mogon.
#
# This script requests one core (out of 64) on one node. The job
# will have access to the default memory of the partition.
#-----------------------------------------------------------------
#SBATCH -J mysimplejob           # Job name
#SBATCH -o mysimplejob.%j.out    # Specify stdout output file (%j expands to jobId)
#SBATCH -p short                 # Queue name 'short' or 'long' on Mogon I
                                 #            'smp' on Mogon II
#SBATCH -n 1                     # Total number of tasks, here explicitly 1
#SBATCH -t 00:30:00              # Run time (hh:mm:ss) - 0.5 hours
 
#SBATCH -A account               # Specify allocation to charge against
 
# Load all necessary modules if needed (these are examples)
# Loading modules in the script ensures a consistent environment.
 
# Launch the executable 64 times
srun <myexecutable>

Trivial example - full node job - threaded application

In contrast to the previous example, the following will launch one executable, with 64 threads. Be careful: Most applications do not scale that far.

#!/bin/bash
#-----------------------------------------------------------------
# Example SLURM job script to run serial applications on Mogon.
#
# This script requests one core (out of 64) on one node. The job
# will have access to all the memory in the node.  Note that this
# job will be charged as if all 64 cores were requested.
#-----------------------------------------------------------------
 
#SBATCH -J mysimplejob           # Job name
#SBATCH -o mysimplejob.%j.out    # Specify stdout output file (%j expands to jobId)
#SBATCH -p nodeshort             # Queue name
#SBATCH -N 1                     # Total number of nodes requested (64 cores/node per Mogon I node)
#SBATCH -n 1                     # Total number of tasks
#SBATCH -c 64                    # Total number of cores for the single task
#SBATCH -t 00:30:00              # Run time (hh:mm:ss) - 0.5 hours
 
#SBATCH -A account               # Specify allocation to charge against
 
# Load all necessary modules if needed (these are examples)
# Loading modules in the script ensures a consistent environment.
module load gcc/6.3.0
 
# Launch the executable 1 times
srun <myexecutable>

Simple MPI Job

#!/bin/bash
#-----------------------------------------------------------------
# Example SLURM job script to run MPI Job on Mogon.
#
# This script requests 128 cores on two node. The job
# will have access to all the memory in the nodes.  
#-----------------------------------------------------------------
 
#SBATCH -J mympijob              # Job name
#SBATCH -o mympijob.%j.out       # Specify stdout output file (%j expands to jobId)
#SBATCH -p nodeshort             # Queue name
#SBATCH -N 2                     # Total number of nodes requested (64 cores/node)
#SBATCH -n 128                   # Total number of tasks
#SBATCH -t 00:30:00              # Run time (hh:mm:ss) - 0.5 hours
 
#SBATCH -A account               # Specify allocation to charge against
 
# Load all necessary modules if needed (these are examples)
# Loading modules in the script ensures a consistent environment.
module load toolchain/iimpi/2017.02-GCC-6.3.0
 
# Launch the executable
srun <myexecutable>

Hybrid MPI Jobs (using GROMACS as an example)

Whereas MPI applications frequently adhere to the standard MPI idea of parallelization by multiprocessing and exchanging messages between the created processes, hybrid applications use internally threaded processes (e.g. either by means of MPI-threads or OpenMP).

For this example we assume you want to run GROMACS on 2 nodes with 32 MPI-ranks per node and 2 threads per rank. The job script could look something like this

#!/bin/bash
#-----------------------------------------------------------------
# Example SLURM job script to run GROMACS with MPI on Mogon.
#
# This script requests 128 cores on two node. The job
# will have access to all the memory in the nodes.  
#-----------------------------------------------------------------
 
#SBATCH -J mygromacsjob          # Job name
#SBATCH -o mygromacsjob.%j.out   # Specify stdout output file (%j expands to jobId)
#SBATCH -p nodeshort             # Queue name
#SBATCH -N 2                     # Total number of nodes requested (64 cores/node)
#SBATCH -t 00:30:00              # Run time (hh:mm:ss) - 0.5 hours
 
#SBATCH -A <account>             # Specify allocation to charge against
 
# Load all necessary modules if needed (these are examples)
# Loading modules in the script ensures a consistent environment.
module load bio/GROMACS/5.1.4-foss-2016b-hybrid
 
# Launch the executable
srun -n 64 -c 2 gmx_mpi mdrun -ntomp 2 -deffnm em 
1)
Actually: “will be” - once a certain plugin is in place.
2)
SLURM writes buffered. Shell based solution do not write buffered.
slurm_submit.txt · Last modified: 2019/05/20 11:29 by meesters