User Tools

Site Tools


node_local_scheduling

Differences

This shows you the differences between two versions of the page.

Link to this comparison view

Both sides previous revision Previous revision
node_local_scheduling [2018/08/01 14:23]
meesters [An example showing the use of functions, variables and redirection]
node_local_scheduling [2018/11/28 09:54] (current)
baumg001
Line 1: Line 1:
 ====== Node-local scheduling ====== ====== Node-local scheduling ======
  
-There are some use cases, where you would want to simply request a **full cluster node** from the LSF batch system ​and then run **many** //(e.g. much more than 64)// **small** //(e.g. only a fragment of the total job runtime)// tasks on this full node. Then of course you will need some **local scheduling** on this node to ensure proper utilization of all cores.+There are some use cases, where you would want to simply request a **full cluster node** from slurm and then run **many** //(e.g. much more than 64)// **small** //(e.g. only a fragment of the total job runtime)// tasks on this full node. Then of course you will need some **local scheduling** on this node to ensure proper utilization of all cores.
  
 To accomplish this, we suggest you use the [[http://​www.gnu.org/​software/​parallel/​|GNU Parallel]] program. The program is installed to ''/​cluster/​bin'',​ but you can also simply load the [[modules|modulefile]] ''​software/​gnu_parallel''​ so that you can also access its man page. To accomplish this, we suggest you use the [[http://​www.gnu.org/​software/​parallel/​|GNU Parallel]] program. The program is installed to ''/​cluster/​bin'',​ but you can also simply load the [[modules|modulefile]] ''​software/​gnu_parallel''​ so that you can also access its man page.
Line 23: Line 23:
 </​file>​ </​file>​
  
-Now of course we could submit 150 jobs using LSF or we could use one job which processes the files one after another, but the most elegant way would be to submit one job for 64 cores (e.g. a whole node on Mogon I) and process the files in parallel. This is especially convenient, since we can then use the ''​nodeshort''​ queue which has better scheduling characteristics than ''​short''​ (while both show better scheduling compared to there ''​long''​ counterparts:​+Now of course we could submit 150 jobs using slurm or we could use one job which processes the files one after another, but the most elegant way would be to submit one job for 64 cores (e.g. a whole node on Mogon I) and process the files in parallel. This is especially convenient, since we can then use the ''​nodeshort''​ queue which has better scheduling characteristics than ''​short''​ (while both show better scheduling compared to their ''​long''​ counterparts:​
  
 <file bash parallel_job>​ <file bash parallel_job>​
node_local_scheduling.txt · Last modified: 2018/11/28 09:54 by baumg001