EULAG
EULAG is a numerical solver for all-scale geophysical flows. The underlying anelastic equations are either solved in an EULerian (flux form), or a LAGrangian (advective form) framework.(taken from http://www.mmm.ucar.edu/eulag/, more details there)
Outdated
This page is outdated and under revision!EULAG | |
---|---|
Version: | ? |
Lizenz: | - |
Developer: | EULAG was developed at UCAR |
General
Currently, only the multiprocessor version without NCARG is adapted to MOGON. And only NETCDF 0 and NETCDF 2 are supported. If there's the need for other options drop a mail to HPC Group.
Make sure that you choose as symmetric values for nprocx
and nprocy
as possible. Execution times for eulag seem to be 1/3 for nprocx=nprocy=8
as compared to nprocx=64 nprocy=1
.
Make sure your memory reservation is big enough for your calculation. Probably you will request full nodes.
In this case and if you don't know yet how much memory you need, you can choose -app Reserve1800M
to ask for the complete memory. The LSF report will tell you the rough maximum memory needed by your job and you can adjust the memory reservation for the subsequent jobs. Remember that memory reservation is made per core.
Usage
Users need to load the following modules for use without netcdf (NETCDF 0) :
module load pgi/13.10 module load mpi/mvapich2/2.1a/pgi13.10
If you want netcdf-files you need to load the following module, too (NETCDF 2):
module load software/netcdf/parallel_1.5.0-pgi13.10
Adapting your eulag-script to MOGON
In the very beginning of the script you have to add our cluster in the area “Determine hostname/user on NCAR system” (or similar, depending on the version of your script)
if($hname == 'lo') setenv machname 'mogon'
und
if($hname == 'lo') setenv MACHINE LNX
Im Bereich “Set batch / queue options” müssen die entsprechenden queue-, project- und wall-clock-time-Parameter für Mogon eingetragen werden:
if( $machname == 'mogon') then setenv PROJECT <your project> setenv QUEUE nodeshort setenv NTIME 300 # minutes for nodeshort (max 300minutes) # setenv QUEUE nodelong # setenv NTIME 600 # minutes (currently max 7200 minutes) # you can check wallclocktimes with bqueues -l nodelong endif
Keep in mind that your job is going to be killed if it takes longer than your wall-clock-time or longer than the max wall-clock-time of the queue.
Then you need a big part about the compilation. Search for “L I N U X C L U S T E R C O M P I L A T I O N”. Within the following block of if ( $MACHINE == LNX ) then
add somewhere
- Compilationpart
###################################################### if ($machname == 'mogon') then # this is MOGON's PGF setup ###################################################### setenv NETCDFCMP '' if ($NETCDF == 1) then setenv NETCDFINC ${NETCDF_HOME}/include setenv NETCDFLIB ${NETCDF_HOME}/lib setenv NETCDFCMP '-L'$NETCDFLIB' -I'$NETCDFINC' -lnetcdf -lnetcdff -L${HDF5_HOME}/lib -lm -lz' cp -frp ${NETCDFINC}/netcdf.inc . endif if ($NETCDF == 2) then setenv NETCDFINC ${PNETCDF_HOME}/include setenv NETCDFLIB ${PNETCDF_HOME}/lib setenv NETCDFCMP '-L'$NETCDFLIB' -I'$NETCDFINC' -lpnetcdf '$NETCDFLIB'/libpnetcdf.a' cp ${NETCDFINC}/pnetcdf.inc . endif echo $NETCDFCMP set MPI_LIB = $MPI_HOME/lib set PGI_LIB = $PGI/lib ## case processor=1 isn't handled ##### if ($NPE == 1) then set PGF90 = 'pgf90 -fpic -tp=bulldozer' if ( $NCARG == 1 ) then # COMPILE WITH NCAR GRAPHICS echo 'NCAR GRAPHICS not available on Mogon' else # COMPILE WITHOUT NCAR GRAPHICS echo 'single processor not available in this versionon Mogon' endif else #################### #### here ist the multiprocessor part that we need (NPE>1) ################# set PGF90 = 'mpif90 -O2 -fastsse -Msmart -tp=bulldozer' ### optimize with vectorization and AMD specific $ if ( $WORD == 4 ) then echo "${PGF90} -c -Mcray=pointer -Mnoframe -Kieee src.F" ${PGF90} -c -Mcray=pointer -Mnoframe -Kieee src.F echo "${PGF90} src.o $OBJECTS $NETCDFCMP -L${PGI_LIB} -L${MPI_LIB} -lfmpich -lmpich" ${PGF90} src.o $OBJECTS $NETCDFCMP -L${PGI_LIB} -L${MPI_LIB} -lfmpich -lmpich endif if ( $WORD == 8 ) then echo "CURRENTLY NOT AVAILABLE FOR DOUBLE PRECISION IMPLEMETATION ON LINUXCLUSTERS" echo "${PGF90} -c -r8 -Mcray=pointer -Mnoframe -Kieee src.F" ${PGF90} -c -r8 -Mcray=pointer -Mnoframe -Kieee src.F echo "${PGF90} src.o $OBJECTS $NETCDFCMP -r8 -L${PGI_LIB} -L${MPI_LIB} -lfmpich -lmpich" ${PGF90} src.o $OBJECTS $NETCDFCMP -r8 -L${PGI_LIB} -L${MPI_LIB} -lfmpich -lmpich endif endif endif ###################################################### endif ## mogon ######################################################
Somewhere later inside the same block ($MACHINE = LNX) you have to add the following lines for the job submission:
- Submissionpart
###################################################### if( $machname == 'mogon') then ###################################################### ## Only handled MPI-case since this is for parllel run on MOGON ################### if ($MESSG == MPI) then setenv OBJECT_MODE 64 rm -f run_paral echo 'saving submission script' cat > run_paral << '\eof' #!/bin/ksh # # LSF batch script to run a parallel code # #BSUB -W NTIME # cpu time limit ####################### ##BSUB -x # exclusive use of node (not_shared) ##BSUB -a poe # select poe #BSUB -R 'span[ptile=64]' # run a max of 64 tasks per node #BSUB -n NPE # number of tasks #BSUB -app Reserve500M # memory reservation #BSUB -J SUFFIX.NPE # job name #BSUB -e OUTPUTDIR/out.SUFFIX.NPE # output filename #BSUB -o OUTPUTDIR/out.SUFFIX.NPE # input filename #BSUB -q QUEUE # queue #BSUB -P PROJECT ####################### set -ex echo $PATH cd PATH pwd date export OBJECT_MODE=OBJECTMODE echo 'setting MPI' ####################### # MPI runtime settings ####################### # not sure yet about whether XLSMPOPTS are used export MP_PROCS=NPE export MP_SHARED_MEMORY=yes export MP_WAIT_MODE=poll export XLSMPOPTS="parthds=1:stack=50000000 : spins=500000 : yields=50000" export MP_COREFILE_FORMAT=STDERR export MPI_BUFS_PER_PROC=256 export MPI_NAP=yes #export MPI_INFOLEVEL=2 export MP_EUILIB=us export MP_LABELIO=yes export MP_EAGER_LIMIT=64000 export XLFRTEOPTS=err_recovery=no:buffering=disable_preconn ####################### ulimit -a echo 'System load: $(uptime)' echo 'sending out job' mpirun.zdv ./a.out date exit '\eof' sed -e "s#OUTPUTDIR#$OUTPUTDIR#g" -e "s#PATH#$DIR#g" run_paral > tmp mv tmp run_paral sed -e "s#OBJECTMODE#$OBJECT_MODE#g" run_paral > tmp mv tmp run_paral sed -e "s#SUFFIX#$JOBNAME#g" run_paral > tmp mv tmp run_paral sed -e "s#PROJECT#$PROJECT#g" run_paral > tmp mv tmp run_paral sed -e "s#QUEUE#$QUEUE#g" run_paral > tmp mv tmp run_paral sed -e "s#NTIME#$NTIME#g" run_paral > tmp mv tmp run_paral sed -e "s#NPE#$NPE#g" run_paral > tmp mv tmp run_paral #--------------- endif #--------------- ###################################################### endif ## mogon ######################################################
Now save your file, set up your simulation parameters and call csh ./<my_eulag_script>
as usual.