start:software:topical:physics:eulag

Differences

This shows you the differences between two versions of the page.

Link to this comparison view

start:software:topical:physics:eulag [2020/04/16 15:40] (current)
jrutte02 created
Line 1: Line 1:
 +====== EULAG  ====== ​
  
 +EULAG is a numerical solver for all-scale geophysical flows. The underlying anelastic equations are either solved in an EULerian (flux form), or a LAGrangian (advective form) framework.(taken from http://​www.mmm.ucar.edu/​eulag/,​ more details there) ​
 +
 +<callout type="​danger"​ title="​Outdated"​ icon="​true">​
 +This page is outdated and under revision!
 +</​callout>​
 +
 +
 +
 +^  EULAG  ^^
 +|**Version:​**|?​|
 +|**Lizenz:​**|-|
 +|**Developer**:​|EULAG was developed at UCAR|
 +
 +===== General =====
 +
 +Currently, only the multiprocessor version **without NCARG** is adapted to MOGON. And only NETCDF 0 and NETCDF 2 are supported. If there'​s the need for other options drop a mail to [[hpc@uni-mainz.de|HPC Group]].
 +
 +Make sure that you choose as symmetric values for ''​nprocx''​ and ''​nprocy''​ as possible. Execution times for eulag seem to be 1/3 for ''​nprocx=nprocy=8''​ as compared to ''​nprocx=64 nprocy=1''​. ​
 +
 +Make sure your memory reservation is big enough for your calculation. Probably you will request full nodes. ​
 +In this case and if you don't know yet how much memory you need, you can choose ''​-app Reserve1800M''​ to ask for the complete memory. The LSF report will tell you the rough maximum memory needed by your job and you can adjust the memory reservation for the subsequent jobs. Remember that memory reservation is made per core. 
 +
 +===== Usage =====
 +
 +Users need to load the following modules for use without netcdf (NETCDF 0) :
 +<code bash>
 +module load pgi/13.10
 +module load mpi/​mvapich2/​2.1a/​pgi13.10
 +</​code>​
 +
 +If you want netcdf-files you need to load the following module, too (NETCDF 2): 
 +<code bash>
 +module load software/​netcdf/​parallel_1.5.0-pgi13.10 ​
 +</​code> ​
 +
 +===== Adapting your eulag-script to MOGON =====
 +
 +In the very beginning of the script you have to add our cluster in the area "​Determine hostname/​user on NCAR system"​ (or similar, depending on the version of your script) ​
 +<code csh>
 +if($hname == '​lo'​) setenv machname '​mogon' ​
 +</​code>​
 +und 
 +<code csh>
 +if($hname == '​lo'​) setenv MACHINE LNX
 +</​code>​
 +
 +Im Bereich "Set batch / queue options"​ müssen die entsprechenden queue-, project- und wall-clock-time-Parameter für Mogon eingetragen werden: ​
 +
 +<code csh>
 +if( $machname == '​mogon'​) then
 + ​setenv PROJECT <your project>
 + ​setenv QUEUE nodeshort
 + ​setenv NTIME 300 # minutes for nodeshort (max 300minutes)
 +# setenv QUEUE nodelong
 +# setenv NTIME 600 # minutes (currently max 7200 minutes)
 +# you can check wallclocktimes with bqueues -l nodelong  ​
 +endif
 +</​code>​
 +
 +:!: Keep in mind that your job is going to be killed if it takes longer than your wall-clock-time or longer than the max wall-clock-time of the queue. ​
 +
 +Then you need a big part about the compilation. Search for "L I N U X C L U S T E R    C O M P I L A T I O N". Within the following block of ''​if ( $MACHINE == LNX ) then''​ add somewhere
 +
 +<file csh Compilationpart> ​
 +######################################################​
 +if ($machname == '​mogon'​) then  # this is MOGON'​s PGF setup
 +######################################################​
 +
 +setenv NETCDFCMP ''​
 +if ($NETCDF == 1) then
 +setenv NETCDFINC ${NETCDF_HOME}/​include
 +setenv NETCDFLIB ${NETCDF_HOME}/​lib
 +setenv NETCDFCMP '​-L'​$NETCDFLIB'​ -I'​$NETCDFINC'​ -lnetcdf -lnetcdff -L${HDF5_HOME}/​lib -lm -lz'
 +cp -frp ${NETCDFINC}/​netcdf.inc .
 +endif
 +if ($NETCDF == 2) then
 +setenv NETCDFINC ${PNETCDF_HOME}/​include
 +setenv NETCDFLIB ${PNETCDF_HOME}/​lib
 +setenv NETCDFCMP '​-L'​$NETCDFLIB'​ -I'​$NETCDFINC'​ -lpnetcdf '​$NETCDFLIB'/​libpnetcdf.a'​
 +cp ${NETCDFINC}/​pnetcdf.inc .
 +endif
 +
 +echo $NETCDFCMP
 +
 +set MPI_LIB = $MPI_HOME/​lib
 +set PGI_LIB = $PGI/lib
 +
 +## case processor=1 isn't handled #####
 +if ($NPE == 1) then
 +set PGF90 = 'pgf90 -fpic -tp=bulldozer'​
 +if ( $NCARG == 1 ) then  # COMPILE WITH NCAR GRAPHICS
 +   echo 'NCAR GRAPHICS not available on Mogon'
 +else   # COMPILE WITHOUT NCAR GRAPHICS
 +   echo '​single processor not available in this versionon Mogon'
 +endif
 +else
 +####################​
 +#### here ist the multiprocessor part that we need (NPE>1)
 +#################​
 +set PGF90 = '​mpif90 -O2 -fastsse -Msmart -tp=bulldozer'​ ### optimize with vectorization and AMD specific $
 +   if ( $WORD == 4 ) then
 +      echo "​${PGF90} -c -Mcray=pointer -Mnoframe -Kieee src.F"
 +      ${PGF90} -c -Mcray=pointer -Mnoframe -Kieee src.F
 +      echo "​${PGF90} src.o $OBJECTS $NETCDFCMP -L${PGI_LIB} -L${MPI_LIB} -lfmpich -lmpich"​
 +      ${PGF90} src.o $OBJECTS $NETCDFCMP -L${PGI_LIB} -L${MPI_LIB} -lfmpich -lmpich
 +   endif
 +   if ( $WORD == 8 ) then
 +      echo "​CURRENTLY NOT AVAILABLE FOR DOUBLE PRECISION IMPLEMETATION ON LINUXCLUSTERS"​
 +      echo "​${PGF90} -c -r8 -Mcray=pointer -Mnoframe -Kieee src.F"
 +      ${PGF90} -c -r8 -Mcray=pointer -Mnoframe -Kieee src.F
 +      echo "​${PGF90} src.o $OBJECTS $NETCDFCMP -r8 -L${PGI_LIB} -L${MPI_LIB} -lfmpich -lmpich"​
 +      ${PGF90} src.o $OBJECTS $NETCDFCMP -r8 -L${PGI_LIB} -L${MPI_LIB} -lfmpich -lmpich
 +   endif
 +endif
 +endif
 +######################################################​
 +endif         ## mogon
 +######################################################​
 +</​file>​
 +
 +Somewhere later inside the same block ($MACHINE = LNX) you have to add the following lines for the job submission: ​
 +
 +<file csh Submissionpart>​
 +######################################################​
 +if( $machname == '​mogon'​) then
 +######################################################​
 +## Only handled MPI-case since this is for parllel run on MOGON
 +###################​
 +
 +if ($MESSG == MPI) then
 +setenv OBJECT_MODE 64
 +rm -f run_paral
 +echo '​saving submission script'​
 +cat > run_paral << '​\eof'​
 +#!/bin/ksh
 +#
 +# LSF batch script to run a parallel code
 +#
 +#BSUB -W NTIME                         # cpu time limit
 +#######################​
 +##BSUB -x                               # exclusive use of node (not_shared)
 +##BSUB -a poe                           # select poe
 +#BSUB -R '​span[ptile=64]' ​             # run a max of 64 tasks per node
 +#BSUB -n NPE                           # number of tasks
 +#BSUB -app Reserve500M ​                # memory reservation
 +#BSUB -J SUFFIX.NPE ​                   # job name
 +#BSUB -e OUTPUTDIR/​out.SUFFIX.NPE ​     # output filename
 +#BSUB -o OUTPUTDIR/​out.SUFFIX.NPE ​     # input filename
 +#BSUB -q QUEUE                         # queue
 +#BSUB -P PROJECT
 +#######################​
 +
 +set -ex
 +echo $PATH
 +cd PATH
 +pwd
 +date
 +export OBJECT_MODE=OBJECTMODE
 +
 +echo '​setting MPI'
 +#######################​
 +# MPI runtime settings
 +#######################​
 +# not sure yet about whether XLSMPOPTS are used
 +export MP_PROCS=NPE
 +export MP_SHARED_MEMORY=yes
 +export MP_WAIT_MODE=poll
 +export XLSMPOPTS="​parthds=1:​stack=50000000 : spins=500000 : yields=50000"​
 +export MP_COREFILE_FORMAT=STDERR
 +export MPI_BUFS_PER_PROC=256
 +export MPI_NAP=yes
 +#export MPI_INFOLEVEL=2
 +export MP_EUILIB=us
 +export MP_LABELIO=yes
 +export MP_EAGER_LIMIT=64000
 +export XLFRTEOPTS=err_recovery=no:​buffering=disable_preconn
 +#######################​
 +ulimit -a
 +echo '​System load: $(uptime)'​
 +
 +echo '​sending out job'
 +mpirun.zdv ./a.out
 +date
 +exit
 +
 +'​\eof'​
 + sed -e "​s#​OUTPUTDIR#​$OUTPUTDIR#​g"​ -e "​s#​PATH#​$DIR#​g"​ run_paral > tmp
 + mv tmp run_paral
 + sed -e "​s#​OBJECTMODE#​$OBJECT_MODE#​g"​ run_paral > tmp
 + mv tmp run_paral
 + sed -e "​s#​SUFFIX#​$JOBNAME#​g"​ run_paral > tmp
 + mv tmp run_paral
 + sed -e "​s#​PROJECT#​$PROJECT#​g"​ run_paral > tmp
 + mv tmp run_paral
 + sed -e "​s#​QUEUE#​$QUEUE#​g"​ run_paral > tmp
 + mv tmp run_paral
 + sed -e "​s#​NTIME#​$NTIME#​g"​ run_paral > tmp
 + mv tmp run_paral
 + sed -e "​s#​NPE#​$NPE#​g"​ run_paral > tmp
 + mv tmp run_paral
 +
 +#​---------------
 +endif
 +#​---------------
 +
 +######################################################​
 +endif         ## mogon
 +######################################################​
 +</​file>​
 +
 +
 +Now save your file, set up your simulation parameters and call ''​csh ./<​my_eulag_script>''​ as usual. ​
  • start/software/topical/physics/eulag.txt
  • Last modified: 2020/04/16 15:40
  • by jrutte02