Differences
This shows you the differences between two versions of the page.
Both sides previous revision Previous revision Next revision | Previous revision | ||
node_local_scheduling [2019/06/13 14:45] meesters [Running on several hosts] |
— (current) | ||
---|---|---|---|
Line 1: | Line 1: | ||
- | ====== Node-local scheduling ====== | ||
- | There are some use cases, where you would want to simply request a **full cluster node** from slurm and then run **many** //(e.g. much more than 64)// **small** //(e.g. only a fragment of the total job runtime)// tasks on this full node. Then of course you will need some **local scheduling** on this node to ensure proper utilization of all cores. | ||
- | |||
- | To accomplish this, we suggest you use the [[http:// | ||
- | |||
- | For more documentation on how to use GNU Parallel, please read '' | ||
- | |||
- | ===== Mogon Usage Example ===== | ||
- | |||
- | Let's say we have a number of input data files that contain differing parameters that are going to be processed independently by our program: | ||
- | |||
- | <file bash> | ||
- | $ ls data_*.in | ||
- | data_001.in | ||
- | data_002.in | ||
- | [...] | ||
- | data_149.in | ||
- | data_150.in | ||
- | $ cat data_001.in | ||
- | 1 2 3 4 5 | ||
- | 6 7 8 9 0 | ||
- | </ | ||
- | |||
- | Now of course we could submit 150 jobs using slurm or we could use one job which processes the files one after another, but the most elegant way would be to submit one job for 64 cores (e.g. a whole node on Mogon I) and process the files in parallel. This is especially convenient, since we can then use the '' | ||
- | |||
- | <file bash parallel_job> | ||
- | #!/bin/bash | ||
- | |||
- | #SBATCH --job-name=demo_gnu_parallel | ||
- | #SBATCH --output=res_gnu_parallel.txt | ||
- | #SBATCH --ntasks=4 | ||
- | #SBATCH --time=10: | ||
- | #SBATCH --mem-per-cpu=100 | ||
- | #SBATCH -p short | ||
- | #SBATCH -A <your account> | ||
- | |||
- | # will load the most recent version of '' | ||
- | module load tools/ | ||
- | |||
- | # Store working directory to be safe | ||
- | SAVEDPWD=$(pwd) | ||
- | # set jobdir | ||
- | export JOBDIR=/ | ||
- | |||
- | # suppose we want to process 150 data files, we need to create them for the purpose of the example: | ||
- | for ((i=0; i < 151; i++)); do | ||
- | fname=" | ||
- | echo " | ||
- | echo " | ||
- | done | ||
- | |||
- | # First, we copy the input data files and the program to the local filesystem of our node | ||
- | # (we pretend it is useful - an actual use case are programs with random I/O) on those files | ||
- | cp " | ||
- | |||
- | # Change directory to jobdir | ||
- | cd $JOBDIR | ||
- | |||
- | # we could set the number of threads for the program to use like this: | ||
- | # export OMP_NUM_THREADS=4 | ||
- | # but in this case the program is not threaded | ||
- | |||
- | # -t enables verbose output to stderr | ||
- | # We could also set -j $((LSB_DJOB_NUMPROC/ | ||
- | # The --delay parameter should be used to distribute I/O load at the beginning of program execution by | ||
- | # | ||
- | # --progress will output the current progress of the parallel task execution | ||
- | # {} will be replaced by each filename | ||
- | # {#} will be replaced by the consecutive job number | ||
- | # Both variants will have equal results: | ||
- | #parallel -t -j 16 --delay 1 --progress " | ||
- | find . -name ' | ||
- | |||
- | # See the GNU Parallel documentation for more examples and explanation | ||
- | |||
- | # Now capture exit status code, parallel will have set it to the number of failed tasks | ||
- | STATUS=$? | ||
- | | ||
- | # Copy output data back to the previous working directory | ||
- | cp $JOBDIR/ | ||
- | |||
- | exit $STATUS | ||
- | </ | ||
- | |||
- | <code bash> | ||
- | $ sbatch parallel_example_script.sh | ||
- | </ | ||
- | |||
- | After this job has run, we should have the results/ | ||
- | |||
- | <file bash> | ||
- | $ ls data_*.out | ||
- | data_001.out | ||
- | data_002.out | ||
- | [...] | ||
- | data_149.out | ||
- | data_150.out | ||
- | $ cat data_001.out | ||
- | 2 10 20 data_001.in | ||
- | </ | ||
- | |||
- | ===== Multithreaded Programs ===== | ||
- | |||
- | Let's further assume that our program is able to work in parallel itself using OpenMP. | ||
- | We determined that '' | ||
- | This means we can launch '' | ||
- | |||
- | <file bash parallel_job2> | ||
- | #!/bin/bash | ||
- | #SBATCH --job-name=demo_gnu_parallel | ||
- | #SBATCH --output=res_gnu_parallel.txt | ||
- | #SBATCH --cpus-per-task=8 | ||
- | #SBATCH --time=10: | ||
- | #SBATCH --mem-per-cpu=100 | ||
- | #SBATCH -p short | ||
- | #SBATCH -A <your account> | ||
- | |||
- | # will load the most recent version of '' | ||
- | module load tools/ | ||
- | |||
- | # Store working directory to be safe | ||
- | SAVEDPWD=$(pwd) | ||
- | |||
- | JOBDIR=/ | ||
- | RAMDISK=$JOBDIR/ | ||
- | |||
- | export OMP_NUM_THREADS=$SLURM_CPUS_PER_TASK | ||
- | |||
- | # -t enables verbose output to stderr | ||
- | # We could also set -j $((LSB_DJOB_NUMPROC/ | ||
- | # The --delay parameter is used to distribute I/O load at the beginning of program execution by | ||
- | # | ||
- | # --progress will output the current progress of the parallel task execution | ||
- | # {} will be replaced by each filename | ||
- | # {#} will be replaced by the consecutive job number | ||
- | # Both variants will have equal results: | ||
- | #parallel -t -j 16 --delay 1 --progress " | ||
- | find . -name ' | ||
- | # See the GNU Parallel documentation for more examples and explanation | ||
- | |||
- | # Now capture exit status code, parallel will have set it to the number of failed tasks | ||
- | STATUS=$? | ||
- | |||
- | exit $STATUS | ||
- | </ | ||
- | |||
- | |||
- | |||
- | |||
- | |||
- | |||
- | ===== Running on several hosts ===== | ||
- | |||
- | We do not recommend supplying a hostlist to GNU parallel with the '' | ||
- | |||
- | <file bash multi_host> | ||
- | #!/bin/bash | ||
- | #SBATCH -J <your meaningful job name> | ||
- | #SBATCH -A <your account> | ||
- | #SBATCH -p nodeshort # for Mogon I | ||
- | #SBATCH -p parallel | ||
- | #SBATCH --nodes=3 # appropriate number of Nodes | ||
- | #SBATCH -n 24 # example value for Mogon I, see below | ||
- | #SBATCH -t 300 | ||
- | #SBATCH -c=8 # we assume an application which scales to 8 threads, but | ||
- | # -c / --cpus-per-task cat be ommited (default is =1) | ||
- | # or set to a different value. | ||
- | #SBATCH -o <your logfile prefix> | ||
- | |||
- | #adjust / overwrite those two commands to enhance readability & overview | ||
- | # parameterize srun | ||
- | srun=" | ||
- | # parameterize parallel | ||
- | parallel=" | ||
- | |||
- | # your preprocessing goes here | ||
- | |||
- | # start the run with GNU parallel | ||
- | $parallel $srun < | ||
- | </ | ||
- | |||
- | <WRAP center round info 80%> | ||
- | The number of tasks (given by '' | ||
- | |||
- | <code bash> | ||
- | # ensure | ||
- | ((SLURM_CPUS_PER_TASK * SLURM_NTASKS)) -eq $((SLURM_CPUS_ON_NODE * SLURM_CPUS_ON_NODE)) | ||
- | </ | ||
- | </ | ||
- | |||
- | ====== SLURM multiprog for uneven arrays ====== | ||
- | |||
- | The [[https:// | ||
- | |||
- | <file bash master_slave_simple.sh> | ||
- | #!/bin/bash | ||
- | # | ||
- | #SBATCH --job-name=test_ms | ||
- | #SBATCH --output=res_ms.txt | ||
- | # parameters of this snippet, choose sensible values for your setup | ||
- | #SBATCH --ntasks=4 | ||
- | #SBATCH --time=10: | ||
- | #SBATCH --mem-per-cpu=100 | ||
- | |||
- | # for the purpose of this course | ||
- | #SBATCH -A <your account> | ||
- | #SBATCH -p short | ||
- | |||
- | srun <other parameters> | ||
- | </ | ||
- | |||
- | Then, of course the '' | ||
- | |||
- | <file bash multi.conf> | ||
- | 0 echo ' | ||
- | 1-3 bash -c ' | ||
- | </ | ||
- | |||
- | Indeed, as the naming suggests, you can use such setup to emulate a master-slave environment. But then the processes have to care themselves about there communication (sockets, regular files, etc.). And the most cumbersome aspect is: You have to maintain two files at all times, whenever the setup has to be changed, and all parameters have to match. | ||
- | |||
- | The configuration file contains three fields, separated by blanks. These fields are : | ||
- | |||
- | * Task number | ||
- | * | ||
- | * | ||
- | |||
- | Parameters available : | ||
- | |||
- | * '' | ||
- | * '' | ||
- | |||
- | |||
- | ====== The ZDV-taskfarm Script an alternative to multiprog ====== | ||
- | |||
- | The script is hosted on [[https:// | ||
- | |||
- | The slurm multi-prog setup can be difficult for some scenarios: | ||
- | |||
- | * only one executable can be specified per task (e.g. no chain of commands or shell loops are possible, such as '' | ||
- | * limitation on the maximum number of characters per task description (256) | ||
- | * building the multi-prog file can be onerous, if you do not have the luxury of using the ' | ||
- | * the number of commands must match exactly the number of slurm tasks ('' | ||
- | |||
- | [[job_arrays|Slurm Job Arrays]] are a better option to multi-prog, unless | ||
- | |||
- | The taskfarm script makes using multi-prog setups easy. Please only use it, if your tasks have +/- the same run time or else huge parts of the reserved nodes can be left idle. | ||
- | |||
- | For a full listing of the command line interface you can load the module and ask the script itself for help: | ||
- | <code shell> | ||
- | $ module load tools/ | ||
- | $ staskfarm -h | ||
- | </ | ||
- | |||
- | |||
- | ===== Taskfarm: Working with one application on many files ===== | ||
- | |||
- | <file bash taskfarm_file> | ||
- | #!/bin/bash | ||
- | |||
- | #SBATCH -J taskfarm_example | ||
- | #SABTCH -o taskfarm_example_%j.out | ||
- | #SBATCH -N2 # in this example we take 2 nodes | ||
- | #SBATCH -n 128 # optional argument - the optimal setting (or ommitance) has to be tried on a case basis | ||
- | #SBATCH -A <your account> | ||
- | #SBATCH -p nodeshort | ||
- | |||
- | # will load the most recent module version of the taskfarm | ||
- | module load tools/ | ||
- | |||
- | # - suppose we have a program which requires 2 intputs: | ||
- | # ' | ||
- | # - assume further we have 302 such files | ||
- | # - and we want to work on them in a round robin manner | ||
- | |||
- | # 1st we " | ||
- | for ((i=0; i < 303; i++)); do | ||
- | touch " | ||
- | touch " | ||
- | done | ||
- | |||
- | # 3rd, we specify our input command. | ||
- | # Instead of a ' | ||
- | # And we use pattern expansion to retrieve the 2nd file name, | ||
- | # as we cannot (always) loop over several expressions. | ||
- | echo '# | ||
- | echo 'echo " | ||
- | chmod +x cmd_file.sh | ||
- | cmd=$(pwd)/ | ||
- | |||
- | # 4th, start the taskfarm: | ||
- | staskfarm $cmd *_R1.fastq | ||
- | |||
- | # finally, we need to clean up our mess: | ||
- | rm *fastq cmd_file.sh | ||
- | </ | ||
- | |||
- | ===== Taskfarm: Screening one application many parameters ===== | ||
- | |||
- | <WRAP center round alert 80%> | ||
- | As stated, the most sensible use case for the taskfarm are application with +/- equal run times for the inputs. When screening parameters in a simulation, is is likely that run times greatly depend on the parameters. Therefore, it might be better to consider using GNU parallel or other parallelisation schemes. | ||
- | </ | ||
- | |||
- | <file bash taskfarm_file2> | ||
- | #!/bin/bash | ||
- | |||
- | #SBATCH -J taskfarm_example | ||
- | #SABTCH -o taskfarm_example_%j.out | ||
- | #SBATCH -N1 # in this example we take <1 node | ||
- | #SBATCH -n 40 | ||
- | #SBATCH --cpus-per-task=2 | ||
- | #SBATCH -A <your account> | ||
- | #SBATCH --time 10 | ||
- | #SBATCH -p short | ||
- | |||
- | # will load the most recent module version of the taskfarm | ||
- | module load tools/ | ||
- | |||
- | # - suppose we have a program which requires 2 intputs: | ||
- | # all natural numbers for arg1 between 1 and 5 and for arg2 just 0 and 1 | ||
- | # - assume further we want to run a permutation test | ||
- | |||
- | # 1st we " | ||
- | permutations=$(echo {1..5}, | ||
- | |||
- | # 3rd, we specify our input command. | ||
- | # Instead of a ' | ||
- | echo '# | ||
- | echo 'echo " | ||
- | chmod +x cmd_file.sh | ||
- | cmd=$(pwd)/ | ||
- | |||
- | # 4th, start the taskfarm: | ||
- | # NOTE: We start each application with 2 threads (-t 2) | ||
- | # the ' | ||
- | staskfarm -t 2 --parameters $cmd $permutations | ||
- | |||
- | # finally, we need to clean up our mess: | ||
- | rm cmd_file.sh | ||
- | </ |