When using etapfs you should acess these filesystem only within your job and not indicate paths on the command line with SLURM commands (e.g. sbatch). Especially stdout should not be directed to /etapfs as this can prevent a job to be ended (even if the wall clock time limit is hit), if one such filesystem “hangs”.

This means that the -o, -e and -D/--workdir=<path> parameters of sbatch should not point to etapfs. However, copying within your script to etapfs is ok, e.g.:

  # bash code + statements
  <application call + options>
} > <outputfile on etapfs>

Likewise, pointing the output of subshells to etapfs is an alternate solution. This helps the scheduler to clean up jobs.

Also, you should not submit jobs when your actual working directory at the time of submit is on an etapfs. Instead,
#SBATCH --workdir=<path outside etapfs>

can intentionally be used to avoid the mentioned problems, whilst submitting from a working directory in etapfs.

I/O bandwidth reservation

The GPFS fileserver of MOGON provides a file system which is exclusively reserved for ATLAS users. The maximum total I/O bandwidth of this file system is about 8000 MB/s. Until the GPFS file system has been optimized for typical ATLAS ROOT jobs this value is set to a lower value in order to prevent oversubscription of the provided bandwidth, which would result in an unnecessarily large Wall time of the job. When submitting a job, the user must specify the expected bandwidth.

If the user needs an I/O bandwidth of e.g. 10MB/s, the SBATCH statements must provide the following parameter:

#SBATCH -L atlasio:10

Jobs will only start running, if there is enough I/O bandwidth available. The amount of available bandwidth can be checked via:

$ scontrol show license

Here, the first number gives the total bandwidth (in MB/s), which is available. The second gives the currently reserved bandwidth. Hence, jobs requesting more bandwidth than currently available will have to wait. (The result also mentions the hosts managing this bandwidth in the following columns.)

1. If the rusage parameter is omitted by a user, slurm will automatically assume a bandwith of 10 MB/s.

2. For jobs that will not do a reasonable amount (sum of all files < 100MByte) of I/O, the user should specify the parameter with 0 MB/s, otherwise - like mentioned before - 10MB/s will be assumed.

  • start/working_on_mogon/io_odds_and_ends/special_fs.txt
  • Last modified: 2020/04/16 11:49
  • by jrutte02