This is an old revision of the document!

GPU Queues

The titan-Queues (titanshort/long) currently include hosts i0001-i0009, while the gpu-Queues (gpushort/long) include the hosts g0001-g0009. The titan-hosts carry 4 GeForce GTX TITAN, hence a usage request up to cuda=4 can be selected (see below). In contrast the GeForce GTX 480 is installed on the gpu-hosts (for the gpushort/long queues). Finally, for the tesla-Queues (teslashort/long) 4 Tesla K20m cards are installed.

The following link gives an overview on the Compute Nodes. To associate hosts with queues, type

$ bqueues -l <queuename> | grep HOSTS

pick the resulting short name from the output and with

$ bhosts <nodename>

you will get the relevant hosts.

The max. runtime is analogous to the other short/long-Queues. Note however, that wall clock times are given to the reference hosts of a queue, if this host's CPU is faster, than the submission host and you still want to give a time estimate without calculating you first have to infer the reference host of the queue you are using. This is the host given in

$ bqueues -l <queuename> | grep '.0 min'


bsub -q <queuename> -W 30/<reference_host>

will result in 30 min wall clock time (real time).

GPU Usage

To use a GPU you have to explicitly reserve it as a resource in the bsub call:

$ bsub -n 1 -R 'rusage[cuda=1]' -q gpulong ./my_program

The code or application to be carried out needs to

  1. be an executable script or program.
  2. carry a shebang.

While this is true for LSF in general, it is imposed for the GPU-resource requests.

In the previous example 1 CPU is requested (-n 1) and 1 GPU (cuda=1). The actual number of GPUs available depends on the queue you are using.

If supported by the queue, you can request multiple GPUs like

$ bsub -n 1 -R 'rusage[cuda=4]' -q titanshort ./my_program

Be sure to add a sufficient time estimate with -W. Also, multiple CPUs can be requested with the usual ptile option.

In order to use multiples nodes, you have to request entire nodes and entire GPU sets, e.g.

$ bsub -q titanshort -n 2 -R 'span[ptile=1]' -R 'affinity[core(16)]' -R 'rusage[cuda=4]

In this example 2 entire titannodes will be used (also the CPU set).

Your job script / job command has to export the environment of your job. mpirun implementations do have an option for this (see your mpirun man page).


Multiple GPU nodes require to take entire nodes. The entire GPU set has to be claimed and the entire CPU set - either by setting affinity(core or ptile.

  • lsf_gpu.1449159248.txt.gz
  • Last modified: 2015/12/03 17:14
  • by meesters