Eulag

EULAG is a numerical solver for all-scale geophysical flows. The underlying anelastic equations are either solved in an EULerian (flux form), or a LAGrangian (advective form) framework.(taken from http://www.mmm.ucar.edu/eulag/, more details there)

General #

Currently, only the multiprocessor version without NCARG is adapted to MOGON. And only NETCDF 0 and NETCDF 2 are supported. If there’s the need for other options drop a mail to HPC Group.

Make sure that you choose as symmetric values for nprocx and nprocy as possible. Execution times for eulag seem to be 1/3 for nprocx=nprocy8 as compared to nprocx=64 nprocy1.

Make sure your memory reservation is big enough for your calculation. Probably you will request full nodes. In this case and if you don’t know yet how much memory you need, you can choose -app Reserve1800M to ask for the complete memory. The LSF report will tell you the rough maximum memory needed by your job and you can adjust the memory reservation for the subsequent jobs. Remember that memory reservation is made per core.

Usage #

Users need to load the following modules for use without netcdf (NETCDF 0) :

module load pgi/13.10
module load mpi/mvapich2/2.1a/pgi13.10

If you want netcdf-files you need to load the following module, too (NETCDF 2):

module load software/netcdf/parallel_1.5.0-pgi13.10

Adapting your eulag-script to MOGON #

In the very beginning of the script you have to add our cluster in the area “Determine hostname/user on NCAR system” (or similar, depending on the version of your script)

if($hname == 'lo') setenv machname 'mogon'

und

if($hname == 'lo') setenv MACHINE LNX

Im Bereich “Set batch / queue options” müssen die entsprechenden queue-, project- und wall-clock-time-Parameter für Mogon eingetragen werden:

if( $machname == 'mogon') then
 setenv PROJECT <your project>
 setenv QUEUE nodeshort
 setenv NTIME 300 # minutes for nodeshort (max 300minutes)
# setenv QUEUE nodelong
# setenv NTIME 600 # minutes (currently max 7200 minutes)
# you can check wallclocktimes with bqueues -l nodelong  
endif

❗ Keep in mind that your job is going to be killed if it takes longer than your wall-clock-time or longer than the max wall-clock-time of the queue.

Then you need a big part about the compilation. Search for “L I N U X C L U S T E R C O M P I L A T I O N”. Within the following block of if ( $MACHINE ###### LNX ) then add somewhere

######################################################
if ($machname == 'mogon') then  # this is MOGON's PGF setup
######################################################

setenv NETCDFCMP ''
if ($NETCDF == 1) then
setenv NETCDFINC ${NETCDF_HOME}/include
setenv NETCDFLIB ${NETCDF_HOME}/lib
setenv NETCDFCMP '-L'$NETCDFLIB' -I'$NETCDFINC' -lnetcdf -lnetcdff -L${HDF5_HOME}/lib -lm -lz'
cp -frp ${NETCDFINC}/netcdf.inc .
endif
if ($NETCDF == 2) then
setenv NETCDFINC ${PNETCDF_HOME}/include
setenv NETCDFLIB ${PNETCDF_HOME}/lib
setenv NETCDFCMP '-L'$NETCDFLIB' -I'$NETCDFINC' -lpnetcdf '$NETCDFLIB'/libpnetcdf.a'
cp ${NETCDFINC}/pnetcdf.inc .
endif

echo $NETCDFCMP

set MPI_LIB = $MPI_HOME/lib
set PGI_LIB = $PGI/lib

## case processor=1 isn't handled #####
if ($NPE == 1) then
set PGF90 = 'pgf90 -fpic -tpbulldozer'
if ( $NCARG == 1 ) then  # COMPILE WITH NCAR GRAPHICS
   echo 'NCAR GRAPHICS not available on Mogon'
else   # COMPILE WITHOUT NCAR GRAPHICS
   echo 'single processor not available in this versionon Mogon'
endif
else
####################
#### here ist the multiprocessor part that we need (NPE>1)
#################
set PGF90 = 'mpif90 -O2 -fastsse -Msmart -tpbulldozer' ### optimize with vectorization and AMD specific $
   if ( $WORD == 4 ) then
      echo "${PGF90} -c -Mcray=pointer -Mnoframe -Kieee src.F"
      ${PGF90} -c -Mcray=pointer -Mnoframe -Kieee src.F
      echo "${PGF90} src.o $OBJECTS $NETCDFCMP -L${PGI_LIB} -L${MPI_LIB} -lfmpich -lmpich"
      ${PGF90} src.o $OBJECTS $NETCDFCMP -L${PGI_LIB} -L${MPI_LIB} -lfmpich -lmpich
   endif
   if ( $WORD == 8 ) then
      echo "CURRENTLY NOT AVAILABLE FOR DOUBLE PRECISION IMPLEMETATION ON LINUXCLUSTERS"
      echo "${PGF90} -c -r8 -Mcray=pointer -Mnoframe -Kieee src.F"
      ${PGF90} -c -r8 -Mcray=pointer -Mnoframe -Kieee src.F
      echo "${PGF90} src.o $OBJECTS $NETCDFCMP -r8 -L${PGI_LIB} -L${MPI_LIB} -lfmpich -lmpich"
      ${PGF90} src.o $OBJECTS $NETCDFCMP -r8 -L${PGI_LIB} -L${MPI_LIB} -lfmpich -lmpich
   endif
endif
endif
######################################################
endif         ## mogon
######################################################

Somewhere later inside the same block ($MACHINE = LNX) you have to add the following lines for the job submission:

######################################################
if( $machname == 'mogon') then
######################################################
## Only handled MPI-case since this is for parllel run on MOGON
###################

if ($MESSG == MPI) then
setenv OBJECT_MODE 64
rm -f run_paral
echo 'saving submission script'
cat > run_paral << '<br/>eof'
#!/bin/ksh
#
# LSF batch script to run a parallel code
#
#BSUB -W NTIME                         # cpu time limit
#######################
##BSUB -x                               # exclusive use of node (not_shared)
##BSUB -a poe                           # select poe
#BSUB -R 'span[ptile=64]'              # run a max of 64 tasks per node
#BSUB -n NPE                           # number of tasks
#BSUB -app Reserve500M                 # memory reservation
#BSUB -J SUFFIX.NPE                    # job name
#BSUB -e OUTPUTDIR/out.SUFFIX.NPE      # output filename
#BSUB -o OUTPUTDIR/out.SUFFIX.NPE      # input filename
#BSUB -q QUEUE                         # queue
#BSUB -P PROJECT
#######################

set -ex
echo $PATH
cd PATH
pwd
date
export OBJECT_MODE=OBJECTMODE

echo 'setting MPI'
#######################
# MPI runtime settings
#######################
# not sure yet about whether XLSMPOPTS are used
export MP_PROCS=NPE
export MP_SHARED_MEMORY=yes
export MP_WAIT_MODE=poll
export XLSMPOPTS="parthds1:stack=50000000 : spins500000 : yields=50000"
export MP_COREFILE_FORMAT=STDERR
export MPI_BUFS_PER_PROC=256
export MPI_NAP=yes
#export MPI_INFOLEVEL=2
export MP_EUILIB=us
export MP_LABELIO=yes
export MP_EAGER_LIMIT=64000
export XLFRTEOPTS=err_recoveryno:buffering=disable_preconn
#######################
ulimit -a
echo 'System load: $(uptime)'

echo 'sending out job'
mpirun.zdv ./a.out
date
exit

'<br/>eof'
 sed -e "s#OUTPUTDIR#$OUTPUTDIR#g" -e "s#PATH#$DIR#g" run_paral > tmp
 mv tmp run_paral
 sed -e "s#OBJECTMODE#$OBJECT_MODE#g" run_paral > tmp
 mv tmp run_paral
 sed -e "s#SUFFIX#$JOBNAME#g" run_paral > tmp
 mv tmp run_paral
 sed -e "s#PROJECT#$PROJECT#g" run_paral > tmp
 mv tmp run_paral
 sed -e "s#QUEUE#$QUEUE#g" run_paral > tmp
 mv tmp run_paral
 sed -e "s#NTIME#$NTIME#g" run_paral > tmp
 mv tmp run_paral
 sed -e "s#NPE#$NPE#g" run_paral > tmp
 mv tmp run_paral

#---------------
endif
#---------------

######################################################
endif         ## mogon
######################################################

Now save your file, set up your simulation parameters and call csh ./<my_eulag_script> as usual.