node_local_scheduling

Differences

This shows you the differences between two versions of the page.

Link to this comparison view

Both sides previous revision Previous revision
Next revision 		Previous revision
node_local_scheduling [2016/03/04 20:41]
meesters [An example showing the use of functions, variables and redirection] 		node_local_scheduling [2020/10/02 15:11]
jrutte02 removed
Line 1: Line 1:
 ====== Node-local scheduling ====== ====== Node-local scheduling ======
  
-There are some use cases, where you would want to simply request a **full cluster node** from the LSF batch system and then run **many** //(e.g. much more than 64)// **smaller** //(e.g. only a fragment of the total job runtime)// tasks on this full node. Then of course you will need some **local scheduling** on this node to ensure proper utilization of all cores.+There are some use cases, where you would want to simply request a **full cluster node** from slurm and then run **many** //(e.g. much more than 64)// **small** //(e.g. only a fragment of the total job runtime)// tasks on this full node. Then of course you will need some **local scheduling** on this node to ensure proper utilization of all cores.
  
 To accomplish this, we suggest you use the [[http://www.gnu.org/software/parallel/|GNU Parallel]] program. The program is installed to ''/cluster/bin'', but you can also simply load the [[modules|modulefile]] ''software/gnu_parallel'' so that you can also access its man page. To accomplish this, we suggest you use the [[http://www.gnu.org/software/parallel/|GNU Parallel]] program. The program is installed to ''/cluster/bin'', but you can also simply load the [[modules|modulefile]] ''software/gnu_parallel'' so that you can also access its man page.
Line 23: Line 23:
 </file> </file>
  
-Now of course we could submit 150 jobs using LSF or we could use one job which processes the files one after another, but the most elegant way would be to submit one job for 64 cores (e.g. a whole node) and process the files in parallel. This is especially convenient, since we can then use the ''nodelong'' queue which has better scheduling characteristics than ''long''. +Now of course we could submit 150 jobs using slurm or we could use one job which processes the files one after another, but the most elegant way would be to submit one job for 64 cores (e.g. a whole node on Mogon I) and process the files in parallel. This is especially convenient, since we can then use the ''nodeshort'' queue which has better scheduling characteristics than ''short'' (while both show better scheduling compared to their ''long'' counterparts:
- +
-Let's further assume that our program is able to work in parallel itself using OpenMP. +
-We determined that ''OMP_NUM_THREADS=4'' is the best amount of parallel work for one set of input data. +
-This means we can launch ''64/4=16'' processes using GNU Parallel on the one node we have.+
  
 <file bash parallel_job> <file bash parallel_job>
 #!/bin/bash #!/bin/bash
-# LSF Job parameters (could also be given on the bsub command line) 
-# Job name 
-#BSUB -J parallel_job 
-# Queue 
-#BSUB -q nodelong 
-# Number of cores 
-#BSUB -n 64 
-# Memory reservation 
-#BSUB -app Reserve1800M 
-# Allowed job runtime (maximum) 
-#BSUB -W 7200 
  
-# Store working directory to be safe +#SBATCH --job-name=demo_gnu_parallel 
-SAVEDPWD=`pwd`+#SBATCH --output=res_gnu_parallel.txt 
 +#SBATCH --ntasks=4 
 +#SBATCH --time=10:00 
 +#SBATCH --mem-per-cpu=100 
 +#SBATCH -p short 
 +#SBATCH -A <your account>
  
-# First, we copy the input data files and the program to the local filesystem of our node +# will load the most recent version of ''parallel'' 
-cp "${SAVEDPWD}"/data_*.in "/jobdir/${LSB_JOBID}/" +module load tools/parallel
-cp "${SAVEDPWD}"/program "/jobdir/${LSB_JOBID}/"+
  
-# Change directory to jobdir +# Store working directory to be safe 
-cd "/jobdir/${LSB_JOBID}/"+SAVEDPWD=$(pwd) 
 +# set jobdir 
 +export JOBDIR=/localscratch/$SLURM_JOB_ID
  
-export OMP_NUM_THREADS=4+# suppose we want to process 150 data files, we need to create them for the purpose of the example: 
 +for ((i=0; i < 151; i++)); do 
 +    fname="data_$(printf "%03d" $i).in" 
 +    echo "{0..4}" >> $fname 
 +    echo "{5..9}" >> $fname 
 +done 
 +      
 +# First, we copy the input data files and the program to the local filesystem of our node  
 +# (we pretend it is useful - an actual use case are programs with random I/O) on those files 
 +cp "${SAVEDPWD}"/data_*.in $JOBDIR
  
 +# Change directory to jobdir
 +cd $JOBDIR
 +     
 +# we could set the number of threads for the program to use like this:
 +# export OMP_NUM_THREADS=4
 +# but in this case the program is not threaded
 +     
 # -t enables verbose output to stderr # -t enables verbose output to stderr
 # We could also set -j $((LSB_DJOB_NUMPROC/OMP_NUM_THREADS)) to be more dynamic # We could also set -j $((LSB_DJOB_NUMPROC/OMP_NUM_THREADS)) to be more dynamic
-# The --delay parameter is used to distribute I/O load at the beginning of program execution by+# The --delay parameter should be used to distribute I/O load at the beginning of program execution by
 #   introducing a delay of 1 second before starting the next task #   introducing a delay of 1 second before starting the next task
 # --progress will output the current progress of the parallel task execution # --progress will output the current progress of the parallel task execution
Line 64: Line 71:
 # Both variants will have equal results: # Both variants will have equal results:
 #parallel -t -j 16 --delay 1 --progress "./program {/} > {/.}.out" ::: data_*.in #parallel -t -j 16 --delay 1 --progress "./program {/} > {/.}.out" ::: data_*.in
-find . -name 'data_*.in' | parallel -t -j 16 --delay 1 --progress "./program {/} > {/.}.out"+find . -name 'data_*.in' | parallel -t -j $SLURM_NPROCS  "wc {/} > {/.}.out" 
 # See the GNU Parallel documentation for more examples and explanation # See the GNU Parallel documentation for more examples and explanation
 +     
 # Now capture exit status code, parallel will have set it to the number of failed tasks # Now capture exit status code, parallel will have set it to the number of failed tasks
 STATUS=$? STATUS=$?
 +    
 # Copy output data back to the previous working directory # Copy output data back to the previous working directory
-cp "/jobdir/${LSB_JOBID}"/data_*.out "${SAVEDPWD}/" +cp $JOBDIR/data_*.out $SAVEDPWD/ 
 +     
 exit $STATUS exit $STATUS
 </file> </file>
  
-For this example. the program only sleeps for some seconds and then counts the words in the input data file using ''wc''. +<code bash> 
- +$ sbatch parallel_example_script.sh 
-<file bash> +</code>
-$ bsub < parallel_job +
-</file>+
  
 After this job has run, we should have the results/output data (in this case, it's just the output of ''wc'', for demonstration): After this job has run, we should have the results/output data (in this case, it's just the output of ''wc'', for demonstration):
Line 95: Line 101:
 </file> </file>
  
 +===== Multithreaded Programs =====
  
-==== An example showing the use of functions, variables and redirection ====+Let's further assume that our program is able to work in parallel itself using OpenMP. 
 +We determined that ''OMP_NUM_THREADS=8'' is the best amount of parallel work for one set of input data. 
 +This means we can launch ''64/8=8'' processes using GNU Parallel on the one node we have
  
-This example shows how to use user-defined functions, variables and anonymous pipes in bash. It uses [[http://bio-bwa.sourceforge.net/|bwa]], a bio-informatics tool to map unknown DNA-sequences to a given reference. The reason, why it is chosen, is that it requires different kind of inputs: A reference file or directory and at least two input files. These input files may not be compressed, but, as the script shows, uncompression be means of a gzip / zcat and redirection is a working solution.+<file bash parallel_job2> 
 +#!/bin/bash 
 +#SBATCH --job-name=demo_gnu_parallel 
 +#SBATCH --output=res_gnu_parallel.txt 
 +#SBATCH --cpus-per-task=8 
 +#SBATCH --time=10:00 
 +#SBATCH --mem-per-cpu=100 
 +#SBATCH -p short 
 +#SBATCH -A <your account>
  
-  * Note, that this example sets ''set -x'' to print command traces. This is intended merely to ease comprehension. For a script in production, this should be out-commented. +# will load the most recent version of ''parallel'' 
-  * Also note, that information about the function is carried to the sub-shells with the ''export -f'' statement. +module load tools/parallel
-  * Variables which are not stated upon the call of GNU ''parallel'' can be made available to a function with additional ''export'' statements (here: ''$OUTPUTDIR''). +
-  * Positional variables (here, just ''$1'' in our example function) are given by the call through GNU ''parallel''. This is useful for parameters which should change for every iteration, here: the input file name.+
  
-//In particular//: bwa can be slow on files bigger than a few GB. Hence, the proposed trade-off between threads and concurrently running processes, the load balancing, might not be optimal. (Many smaller files will probably analyzed faster with only 4 threads and 16 concurrently running processes (the ''-j'' option).)+# Store working directory to be safe 
 +SAVEDPWD=$(pwd)
  
-<file bash>+JOBDIR=/localscratch/$SLURM_JOBID 
 +RAMDISK=$JOBDIR/ramdisk 
 + 
 +export OMP_NUM_THREADS=$SLURM_CPUS_PER_TASK 
 + 
 +# -t enables verbose output to stderr 
 +# We could also set -j $((LSB_DJOB_NUMPROC/OMP_NUM_THREADS)) to be more dynamic 
 +# The --delay parameter is used to distribute I/O load at the beginning of program execution by 
 +#   introducing a delay of 1 second before starting the next task 
 +# --progress will output the current progress of the parallel task execution 
 +# {} will be replaced by each filename 
 +# {#} will be replaced by the consecutive job number 
 +# Both variants will have equal results: 
 +#parallel -t -j 16 --delay 1 --progress "./program {/} > {/.}.out" ::: data_*.in 
 +find . -name 'data_*.in' | parallel -t -j 8 --delay 1 --progress "./program {/} > {/.}.out" 
 +# See the GNU Parallel documentation for more examples and explanation 
 + 
 +# Now capture exit status code, parallel will have set it to the number of failed tasks 
 +STATUS=$? 
 + 
 +exit $STATUS 
 +</file> 
 + 
 + 
 + 
 + 
 + 
 + 
 +===== Running on several hosts ===== 
 + 
 +We do not recommend supplying a hostlist to GNU parallel with the ''-S'' option, as GNU parallel attempts to ssh on the respective nodes (inluding the master host) and therefore looses the environment. You can script around this, but you will run into a quotation hell. 
 + 
 +<file bash multi_host>
 #!/bin/bash #!/bin/bash
 +#SBATCH -J <your meaningful job name>
 +#SBATCH -A <your account>
 +#SBATCH -p nodeshort # for Mogon I
 +#SBATCH -p parallel  # for Mogon II
 +#SBATCH --nodes=3 # appropriate number of Nodes
 +#SBATCH -n 24    # example value for Mogon I, see below
 +#SBATCH -t 300
 +#SBATCH -c=8 # we assume an application which scales to 8 threads, but
 +             # -c / --cpus-per-task cat be ommited (default is =1)
 +             # or set to a different value.
 +#SBATCH -o <your logfile prefix>_%j.log
  
-#BSUB -n 64 +#adjust / overwrite those two commands to enhance readability & overview 
-#BSUB -R 'span[ptile=64]' +# parameterize srun 
-#BSUB -q nodelong +srun="srun -N1 -n 1 -c $SLURM_CPUS_PER_TASK  --jobid $SLURM_JOBID --cpu_bind=q --mem-per-cpu=$((SLURM_MEM_PER_NODE / SLURM_NTASKS))" 
-#BSUB -o %J.log +# parameterize parallel 
-#BSUB -e %J.log +parallel="parallel -j $SLURM_NTASKS --no-notice "
-#BSUB -N +
-#BSUB -W 3600 +
-#BSUB -J 'bwa template' +
-#BSUB -R 'rusage[ramdisk=10000,mem=1600, local=1000]'+
  
-# This script is written by Christian Meesters (HPC-team, ZDV, Mainz) +# your preprocessing goes here
-#  +
-# Please note: It is valid for Mogon I. The following restrictions apply: +
-# - if your fastq-files in the defined inputdirectory are big, the given +
-#   memory might not be sufficient. In this case restrict yourself to  +
-#   a data subset.+
  
-# in order to see the output of all commands, we set this: +# start the run with GNU parallel 
-set -x+$parallel $srun <command> ::: <parameter list> 
 +</file>
  
-#1 we purge all possible module to avoid a mangled setup +<WRAP center round info 95%> 
-module purge+The number of tasks (given by ''-n'') times the number of cpus per task (given by ''-c'') needs to be equal the number of nodes (given by ''-N'') times number of CPUs per nodes (to be inferred from ''scontrol show node <nodename>'' or in the [[nodes|wiki]].) Or (in pseudo bash)):
  
-#2 we load out GNU parallel module +<code bash> 
-module load software/gnu_parallel+# ensure 
 +((SLURM_CPUS_PER_TASK * SLURM_NTASKS)) -eq $((SLURM_CPUS_ON_NODE * SLURM_CPUS_ON_NODE)) 
 +</code> 
 +</WRAP>
  
-#3 in order perform our alignment (here: bwa sampe) and subsequent sorting we +====== SLURM multiprog for uneven arrays ======
-#  load bwa and samtools +
-module load software/bioinf/bwa/0.7.13 +
-module load software/bioinf/samtools/1.3+
  
-#4 make the return value of the last pipe command which fails the return value +The [[https://slurm.schedmd.com/srun.html|SLURM multiprog]] option in ''srun'' essentially displays a master-slave setup. You need it to run within a SLURM job allocation and trigger ''srun'' with the ''%%--multi-prog%%'' option and appropriate multiprog file:
-set -o pipefail+
  
-#5 set a path the reference genome, extract its directory path +<file bash master_slave_simple.sh> 
-export REFERENCEGENOME="reference/hg19.fasta" +#!/bin/bash 
-REFERENCEDIR=$(dirname $REFERENCEGENOME)+# 
 +#SBATCH --job-name=test_ms 
 +#SBATCH --output=res_ms.txt 
 +# parameters of this snippet, choose sensible values for your setup 
 +#SBATCH --ntasks=4 
 +#SBATCH --time=10:00 
 +#SBATCH --mem-per-cpu=100
  
-#6 select a base directory for all input and traverse through it: +# for the purpose of this course 
-INPUTBASEDIR=./input +#SBATCH -A <your account> 
-#6b now we gather all input files: +#SBATCH -p short
-#  - a first fastq file (ending on _1.fastq) +
-#  - its mate (ending on _2.fastq) +
-# If your files name use a different scheme, adjust this script +
-FORWARD_READS=$(find -L $INPUTBASEDIR -type f -name '*_1.fastq*')+
  
-#7 create an output directory, here: according to bwa and samtools versions +srun <other parameters> --multi-prog multi.conf 
-BWA_VERSION=$(bwa |& grep Version | cut -d ' ' -f2 | cut -d '-' -f1 ) +</file>
-export OUTPUTDIR="bwa${BWA_VERSION}_samtools1.1_${LSB_JOBID}"+
  
-if [ ! -d "$OUTPUTDIR" ]; then +Then, of course the ''multi.conf'' file has to exist:
-    mkdir -p "/jobdir/${LSB_JOBID}/$OUTPUTDIR" +
-    mkdir -p "$OUTPUTDIR" +
-fi+
  
-#8 copy the reference to the ramdisk +<file bash multi.conf> 
-    #df -h /jobdir/${LSB_JOBID}/ramdisk+0      echo     'I am the Master' 
 +1-3    bash -c 'printenv SLURM_PROCID' 
 +</file>
  
-mkdir -p $REFERENCEDIR /jobdir/${LSB_JOBID}/ramdisk/reference +Indeed, as the naming suggests, you can use such setup to emulate a master-slave environment. But then the processes have to care themselves about there communication (sockets, regular files, etc.). And the most cumbersome aspect is: You have to maintain two files at all times, whenever the setup has to be changed, and all parameters have to match.
-cp -r $REFERENCEDIR /jobdir/${LSB_JOBID}/ramdisk/ +
-REFERENCEGENOME=/jobdir/${LSB_JOBID}/ramdisk/reference/$(basename $REFERENCEGENOME) +
-REFERENCEDIR=/jobdir/${LSB_JOBID}/ramdisk/reference+
  
-#9 create an alignment function with the appropriate calls for bwa and samtools +The configuration file contains three fields, separated by blanks. These fields are :
-function bwa_aln { +
-  TEMPOUT=$(basename $1) +
-  # check file ending: is the file ending on gz? +
-  if [ "$1" == "*.gz" ]; then +
-        #bwa sampe $REFERENCEGENOME <( bwa aln -t 4 $REFERENCEGENOME <(zcat $1) ) \ +
-        #                           <( bwa aln -t 4 $REFERENCEGENOME <(zcat ${1/_1/_2} ) ) \ +
-        #                           <(zcat $1) <(zcat ${1/_1/_2}) | \ +
-        #samtools view -Shb /dev/stdin > "$OUTPUTDIR/${TEMPOUT%_1.fastq.gz}_aligned.bam" +
-        bwa mem -M -t 8 $REFERENCEGENOME <(zcat $1) <(zcat ${1/_1/_2})  | +
-        samtools view -Shb /dev/stdin > "$OUTPUTDIR/${TEMPOUT%_1.fastq}_aligned.bam" +
-  else +
-        #bwa sampe $REFERENCEGENOME <( bwa aln -t 4 $REFERENCEGENOME $1 ) \ +
-        #                           <( bwa aln -t 4 $REFERENCEGENOME ${1/_1/_2} ) \ +
-        #                           $1 ${1/_1/_2} | \+
  
-        bwa mem -M -t 8 $REFERENCEGENOME $1 ${1/_1/_2} | +  *   Task number 
-        samtools view -Shb /dev/stdin > "$OUTPUTDIR/${TEMPOUT%_1.fastq}_aligned.bam" +  *     Executable File 
-  fi +  *     Argument  
-}+  
 +Parameters available :
  
-#9b we need to export this function, such that all subprocesses will see it (only works in bash) +  * ''%t'' - The task number of the responsible task 
-export -f bwa_aln+  * ''%o'' - The task offset (task's relative position in the task range).
  
-# finally we start processing 
-#  we consider taking 4 thread for each call of bwa aln, hence 8 threads 
-#  and 64 / 8 is 8. This results in a little over subscription, as bwa mem 
-#  runs with 8 threads and samtools is run, too. 
-#  Note the ungrouping of output with the -u option. 
-parallel -v -u --env bwa_aln --no-notice -j 8 bwa_aln ::: $FORWARD_READS 
  
-# copy all files to the actual output director +====== The ZDV-taskfarm Script an alternative to multiprog ====== 
-cp -r "/jobdir/${LSB_JOBID}/$OUTPUTDIR/*" "$OUTPUTDIR/."+ 
 +The script is hosted on [[https://github.com/cmeesters/staskfarm|github]], forked from [[https://www.tchpc.tcd.ie/person/paddy-doyle|Paddy Doyle, Trinity College, Dublin]] and adapted for Mogon (I and II). 
 + 
 +The slurm multi-prog setup can be difficult for some scenarios: 
 + 
 +  * only one executable can be specified per task (e.g. no chain of commands or shell loops are possible, such as ''cd dir01; ./my_exec'') 
 +  * limitation on the maximum number of characters per task description (256) 
 +  * building the multi-prog file can be onerous, if you do not have the luxury of using the '%t' tokens in your commands or arguments 
 +  * the number of commands must match exactly the number of slurm tasks (''-n''), which means updating two files if you wish to add or remove tasks 
 + 
 +[[job_arrays|Slurm Job Arrays]] are a better option to multi-prog, unless  using the ''parallel'' (Mogon II) or ''node*'' (Mogon I) partitions, with scalable software, anyway. 
 + 
 +The taskfarm script makes using multi-prog setups easy. Please only use it, if your tasks have +/- the same run time or else huge parts of the reserved nodes can be left idle. 
 + 
 +For a full listing of the command line interface you can load the module and ask the script itself for help: 
 +<code shell> 
 +$ module load tools/staskfarm 
 +$ staskfarm -h 
 +</code> 
 + 
 + 
 +===== Taskfarm: Working with one application on many files ===== 
 + 
 +<file bash taskfarm_file> 
 +#!/bin/bash 
 + 
 +#SBATCH -J taskfarm_example 
 +#SABTCH -o taskfarm_example_%j.out 
 +#SBATCH -N2 # in this example we take 2 nodes 
 +#SBATCH -n 128 # optional argument - the optimal setting (or ommitance) has to be tried on a case basis 
 +#SBATCH -A <your account> 
 +#SBATCH -p nodeshort 
 + 
 +# will load the most recent module version of the taskfarm 
 +module load tools/staskfarm 
 + 
 +# - suppose we have a program which requires 2 intputs: 
 +#    'input_<number>_R1.fastq' and 'input_<number>_R2.fastq' 
 +# - assume further we have 302 such files 
 +# - and we want to work on them in a round robin manner 
 + 
 +# 1st we "produce" such dummy files: 
 +for ((i=0; i < 303; i++)); do 
 +   touch "input_${i}_R1.fastq"  
 +   touch "input_${i}_R2.fastq"  
 +done 
 + 
 +# 3rd, we specify our input command. 
 +#      Instead of a 'real' application we drop in 'echo'. 
 +#      And we use pattern expansion to retrieve the 2nd file name, 
 +#      as we cannot (always) loop over several expressions. 
 +echo '#!/bin/bash' > cmd_file.sh 
 +echo 'echo "working on node $(hostname) on files $1 and ${1%%_*}_R2.fastq"' >> cmd_file.sh 
 +chmod +x cmd_file.sh 
 +cmd=$(pwd)/cmd_file.sh 
 + 
 +# 4th, start the taskfarm: 
 +staskfarm $cmd *_R1.fastq 
 + 
 +# finally, we need to clean up our mess: 
 +rm *fastq cmd_file.sh
 </file> </file>
-===== Running on several hosts ===== 
  
-LSF offers ''$LSB_HOSTS'' within a job to identify the hosts assigned to a particular job. This can be used to execute a command distributed over those hosts:+===== Taskfarm: Screening one application many parameters =====
  
-<code bash> +<WRAP center round alert 80%> 
-parallel --no-notice --onall -S $(echo $LSB_HOSTS | tr ' ' ',') echo ::: foo bar +As stated, the most sensible use case for the taskfarm are application with +/- equal run times for the inputs. When screening parameters in a simulation, is is likely that run times greatly depend on the parameters. Therefore, it might be better to consider using GNU parallel or other parallelisation schemes. 
-</code>  +</WRAP> 
-will print the host and 'D', 'E' and 'F' (not necessarily in order and wrapped around, if more than 3 hosts are requested via ''bsub'').+ 
 +<file bash taskfarm_file2> 
 +#!/bin/bash 
 + 
 +#SBATCH -J taskfarm_example 
 +#SABTCH -o taskfarm_example_%j.out 
 +#SBATCH -N1 # in this example we take <1 node 
 +#SBATCH -n 40 
 +#SBATCH --cpus-per-task=2 
 +#SBATCH -A <your account> 
 +#SBATCH --time 10 
 +#SBATCH -p short 
 + 
 +# will load the most recent module version of the taskfarm 
 +module load tools/staskfarm 
 + 
 +# - suppose we have a program which requires 2 intputs: 
 +#    all natural numbers for arg1 between 1 and 5 and for arg2 just 0 and 1 
 +# - assume further we want to run a permutation test 
 + 
 +# 1st we "produce" such dummy files: 
 +permutations=$(echo {1..5},{0..1}) 
 + 
 +# 3rd, we specify our input command. 
 +#      Instead of a 'real' application we drop in 'echo'. 
 +echo '#!/bin/bash' > cmd_file.sh 
 +echo 'echo "working on node $(hostname) on args $1 and with $OMP_NUM_THREADS threads' >> cmd_file.sh 
 +chmod +x cmd_file.sh 
 +cmd=$(pwd)/cmd_file.sh 
 + 
 +# 4th, start the taskfarm: 
 +#      NOTE: We start each application with 2 threads (-t 2) 
 +# the '--parameters' flag, will skip the test for input files (as parameters aren't files) 
 +staskfarm -t 2 --parameters $cmd $permutations 
 + 
 +# finally, we need to clean up our mess: 
 +rm cmd_file.sh 
 +</file>