User Tools

Site Tools


job_arrays

Differences

This shows you the differences between two versions of the page.

Link to this comparison view

Both sides previous revision Previous revision
Next revision
Previous revision
job_arrays [2017/10/09 08:38]
meesters [Multiprog for Uneven Arrays]
job_arrays [2019/03/27 11:55]
meesters [Purpose]
Line 7: Line 7:
 We strongly recommend, using parallel processing in addition and to use job arrays as a convenience feature, without neglecting performance optimization. We strongly recommend, using parallel processing in addition and to use job arrays as a convenience feature, without neglecting performance optimization.
  
 +===== Maximal Job Array Size =====
 +
 +You can look up the maximal job array size using the following command:
 +<code bash>
 +$ scontrol show config| grep MaxArraySize
 +</​code>​
 +
 +The reason we do not document this value is, that it is subject to change.
 +
 +<WRAP center round info>
 +**Why is there an array size limit at all?**
 +
 +If unlimited or if the limit is huge, some users will see this as an invitation to submit a large number of otherwise unoptimized jobs. The idea behind job arrays is to ease workflows, particularly the submission of a bigger number of jobs. However, the motivations to [[node_local_scheduling|pool jobs]] and to optimize still apply.
 +</​WRAP>​
 ===== Job Arrays in Slurm ===== ===== Job Arrays in Slurm =====
  
Line 55: Line 69:
 The ''​%A_%a''​ construct in the output and error file names is used to generate unique output and error files based on the master job ID (''​%A''​) and the array-tasks ID (''​%a''​). In this fashion, each array-tasks will be able to write to its own output and error file. The ''​%A_%a''​ construct in the output and error file names is used to generate unique output and error files based on the master job ID (''​%A''​) and the array-tasks ID (''​%a''​). In this fashion, each array-tasks will be able to write to its own output and error file.
  
 +==== Limiting the number of concurrent jobs of an array ====
 +
 +It is possible to limit the number of concurrently executed jobs of an array, e.g. to minimize I/O overhead within one approach, with this syntax:
 +
 +<code bash>
 +#SBATCH --array=1-1000%50
 +</​code>​
 +
 +where a limit of 50 concurrent jobs would be in place.
 ===== Multiprog for "​Uneven"​ Arrays ===== ===== Multiprog for "​Uneven"​ Arrays =====
  
 The ''​%%--multi-prog%%''​ option in ''​srun''​ allows you to assign each parallel task in your job with a different option. More information can be found at [[node_local_scheduling|our wiki page on node-local scheduling]]. The ''​%%--multi-prog%%''​ option in ''​srun''​ allows you to assign each parallel task in your job with a different option. More information can be found at [[node_local_scheduling|our wiki page on node-local scheduling]].
job_arrays.txt · Last modified: 2019/03/27 11:55 by meesters