start:development:scripting_languages:julia

Differences

This shows you the differences between two versions of the page.

Link to this comparison view

Both sides previous revision Previous revision
Next revision
Previous revision
start:development:scripting_languages:julia [2021/04/19 18:20]
jrutte02 [Submitting a Julia GPU Job]
start:development:scripting_languages:julia [2021/12/08 19:58] (current)
jrutte02 [Julia Modules]
Line 34: Line 34:
 JGU HPC Modules JGU HPC Modules
 ------------------- /cluster/easybuild/broadwell/modules/all ------------------- ------------------- /cluster/easybuild/broadwell/modules/all -------------------
-   lang/Julia/1.5.3-linux-x86_64    lang/Julia/1.6.0-linux-x86_64 (D)+   lang/Julia/1.5.3-linux-x86_64    lang/Julia/1.6.3-linux-x86_64 
 +   lang/Julia/1.6.0-linux-x86_64    lang/Julia/1.7.0-linux-x86_64 (D)
 </code> </code>
  
-One can use a certain version with the following command:+One can use a specific version with the following command:
  
 <code bash> <code bash>
Line 61: Line 62:
 </code> </code>
  
-For package installation, we will use the REPL (read-eval-print loop) that comes built-in to the ''julia'' executable. Start Julia using:+For package installation, we will use the REPL (read-eval-print loop) that comes built-in to the ''julia'' executable. Start Julia by using:
  
 <code bash> <code bash>
Line 81: Line 82:
 you should get an output similar to: you should get an output similar to:
  
-<code bash>+<code julia>
 (v1.6) pkg> add Dates (v1.6) pkg> add Dates
    Resolving package versions...    Resolving package versions...
Line 91: Line 92:
 Let's check the successful installation: First we display the status of the packages in our standard project: Let's check the successful installation: First we display the status of the packages in our standard project:
  
-<code bash>+<code julia>
 (v1.6) pkg> status (v1.6) pkg> status
 </code> </code>
Line 97: Line 98:
 Depending on which packages you have already installed, you should get an output similar to the following: Depending on which packages you have already installed, you should get an output similar to the following:
  
-<code bash>+<code julia>
 (v1.6) pkg> status (v1.6) pkg> status
       Status `/gpfs/fs1/home/<username>/.julia/environments/v1.6/Project.toml`       Status `/gpfs/fs1/home/<username>/.julia/environments/v1.6/Project.toml`
Line 113: Line 114:
 This indicates that ''Dates'' has been successfully installed, but let's check and confirm this with ''Pkg'' This indicates that ''Dates'' has been successfully installed, but let's check and confirm this with ''Pkg''
  
-<code bash>+<code julia>
 (v1.6) pkg> test Dates (v1.6) pkg> test Dates
      Testing Dates      Testing Dates
Line 141: Line 142:
 The installation was sucessful and ''Dates'' can now be included in ''.jl'' file via the following line: The installation was sucessful and ''Dates'' can now be included in ''.jl'' file via the following line:
  
-<code bash>+<code julia>
 using Dates using Dates
 </code> </code>
Line 186: Line 187:
 and then change to the Pkg REPL with '']'', the command line prompt should now look like: and then change to the Pkg REPL with '']'', the command line prompt should now look like:
  
-<code bash>+<code julia>
 (v1.6) pkg> (v1.6) pkg>
 </code> </code>
Line 192: Line 193:
 We are now ready to add CUDA via We are now ready to add CUDA via
  
-<code bash>+<code julia>
 (v1.6) pkg> add CUDA (v1.6) pkg> add CUDA
 </code> </code>
Line 209: Line 210:
 </code> </code>
  
-Open Julia after the modules have been successfully loaded+Open Julia by executing the following command after the modules have been successfully loaded
  
 <code bash> <code bash>
Line 216: Line 217:
  
 now enter the Pkg REPL by pressing '']'', the command line prompt should look like: now enter the Pkg REPL by pressing '']'', the command line prompt should look like:
-<code bash>+<code julia>
 (v1.6) pkg> (v1.6) pkg>
 </code> </code>
 First, the actual packages are added and then the backend is configured. Install ''Plots.jl'' with: First, the actual packages are added and then the backend is configured. Install ''Plots.jl'' with:
  
-<code bash>+<code julia>
 (v1.6) pkg> add Plots (v1.6) pkg> add Plots
 </code> </code>
Line 228: Line 229:
  
  
-<code bash>+<code julia>
 (v1.6) pkg> add PyPlot (v1.6) pkg> add PyPlot
 </code> </code>
 Afterwards, test the successful installation with: Afterwards, test the successful installation with:
-<code bash>+<code julia>
 (v1.6) pkg> test Plots (v1.6) pkg> test Plots
 </code> </code>
Line 238: Line 239:
 and and
  
-<code bash>+<code julia>
 (v1.6) pkg> test PyPlot (v1.6) pkg> test PyPlot
 </code> </code>
Line 248: Line 249:
 ^ Package ^ Comment ^ ^ Package ^ Comment ^
 | Random | |  | Random | | 
 +| CUDA | |
 | LinearAlgebra | | | LinearAlgebra | |
 | Printf| | | Printf| |
 | MPI | | | MPI | |
 | BenchmarkTools|| | BenchmarkTools||
 +| FFTW | |
  
  
 ====== Submitting a Serial Julia Job ====== ====== Submitting a Serial Julia Job ======
  
-<file bash hello_mogon.jl>+<file julia hello_mogon.jl>
 println("Hello MOGON!") println("Hello MOGON!")
 </file> </file>
Line 304: Line 307:
 But let's explore the basics of Julia's multi-threading capabilities with an example: But let's explore the basics of Julia's multi-threading capabilities with an example:
  
-<file bash hello_mogon_smp.jl>+<file julia hello_mogon_smp.jl>
  
 Threads.@threads for i=1:20 Threads.@threads for i=1:20
Line 369: Line 372:
  
  
-<file bash distributed_julia_example.jl>+<file julia distributed_julia_example.jl>
 @everywhere begin @everywhere begin
     using LinearAlgebra     using LinearAlgebra
Line 427: Line 430:
  
 ===== Using MPI with Julia ===== ===== Using MPI with Julia =====
 + Of course, you can also use MPI with Julia on MOGON. This requires you to first carry out the following setup for Julia and the MPI interface ''MPI.jl''
 +
 +
 === Julia MPI Setup === === Julia MPI Setup ===
  
Line 433: Line 439:
  
 <code bash> <code bash>
-module load mpi/OpenMPI/4.0.5-GCC-10.2.0+module load mpi/OpenMPI/4.0.3-GCC-9.3.0
 module load lang/Julia module load lang/Julia
 </code> </code>
Line 444: Line 450:
 The output should should be similar to the following if installation and build was successful: The output should should be similar to the following if installation and build was successful:
  
-<code bash>+<code julia>
 [ ... ] [ ... ]
 [ Info: using system MPI                              0/1 [ Info: using system MPI                              0/1
Line 450: Line 456:
 │   libmpi = "libmpi" │   libmpi = "libmpi"
 │   mpiexec_cmd = `mpiexec` │   mpiexec_cmd = `mpiexec`
-└   MPI_LIBRARY_VERSION_STRING = "Open MPI v4.0.5, package: Open MPI henkela@login22.mogon Distribution, ident: 4.0.5, repo rev: v4.0.5Aug 26, 2020\0"+└   MPI_LIBRARY_VERSION_STRING = "Open MPI v4.0.3, package: Open MPI henkela@z0590.mogon Distribution, ident: 4.0.3, repo rev: v4.0.3Mar 03, 2020\0"
 ┌ Info: MPI implementation detected ┌ Info: MPI implementation detected
 │   impl = OpenMPI::MPIImpl = 2 │   impl = OpenMPI::MPIImpl = 2
-│   version = v"4.0.5"+│   version = v"4.0.3"
 └   abi = "OpenMPI" └   abi = "OpenMPI"
 +
 +1 dependency successfully precompiled in 3 seconds (140 already precompiled, 1 skipped during auto due to previous errors)
 </code> </code>
  
Line 460: Line 468:
 Now that MPI and Julia have been set up correctly, we can proceed to the example. Now that MPI and Julia have been set up correctly, we can proceed to the example.
  
-<file bash hello_mogon_mpi.jl>+<file julia hello_mogon_mpi.jl>
 using MPI using MPI
  
Line 520: Line 528:
 ====== Submitting a Julia GPU Job ====== ====== Submitting a Julia GPU Job ======
  
 +Before you start parallelising with Julia on MOGON GPUs, you need to prepare your Julia environemnt for the usage pf GPUs, as we explaind earlier in the Article about [[https://mogonwiki.zdv.uni-mainz.de/dokuwiki/start:development:scripting_languages:julia#pkg_-_cuda|CUDA.jl]]. After successfully setting up ''CUDA.jl'', you can directly start utilising the advantages of GPUs. We have given some examples below to make it easier for you to start using Julia on MOGON GPUs and to somewhat reflect the advantages of GPUs.
  
  
 === CPU/GPU Bandwidth and Read+Write Speed  === === CPU/GPU Bandwidth and Read+Write Speed  ===
-<file bash gpu_rw_perf.jl>+ 
 +The test estimates how fast data can be sent to and read from the GPU. Since the GPU is plugged into a PCI bus, this largely depends on the speed of the PCI bus as well as many other factors. However, there is also some overhead included in the measurements, in particular the overhead for function calls and array allocation time. However, since these are present in any "real" use of the GPU, it is reasonable to include them. Memory is allocated and data is sent to the GPU using Julia's ''CUDA.jl'' package. Memory is allocated and data is transferred back to CPU memory using Julia's native ''Array()'' function. 
 + 
 +The theoretical bandwidth per lane for **PCIe 3.0** is $0.985 GB/s$. For the GTX 1080Ti (**PCIe3 x16**) used in our MOGON GPU nodes the 16-lane slot could theoretical give $15.754 GB/s$.(( This example was taken from the [[https://de.mathworks.com/help/parallel-computing/measuring-gpu-performance.html|MATLAB Help Center]] and adapted.)) 
 + 
 +<file julia gpu_rw_perf.jl>
 using LinearAlgebra using LinearAlgebra
 using Plots using Plots
Line 610: Line 624:
 savefig(p3, "gpu_rw_perf_result.png") savefig(p3, "gpu_rw_perf_result.png")
 </file> </file>
 +
 +The job script is pretty ordinary. In this example, we use only one GPU and start Julia with four threads. To do this, we request one process with four cpus for multithreading:
  
 <file bash julia_gpu_rw_job.slurm> <file bash julia_gpu_rw_job.slurm>
Line 633: Line 649:
 </file> </file>
  
 +The job is submitted with the following command
 +
 +<code bash>
 +sbatch julia_gpu_rw_job.slurm
 +</code>
 +The job will be finished after a few minutes, you can view the output as follows:
 <code bash> <code bash>
 cat gpu_rw*.out cat gpu_rw*.out
 </code> </code>
 +The output should be similar to the following lines:
 <code bash> <code bash>
 Achieved peak send speed of 8.9 GB/s Achieved peak send speed of 8.9 GB/s
Line 643: Line 665:
 Achieved peak read+write speed on the CPU: 5.7 GB/s Achieved peak read+write speed on the CPU: 5.7 GB/s
 </code> </code>
 + The Julia script also generates a plot, which we would like to show here:
  
 {{ :start:development:scripting_languages:gpu_rw_result.png?direct&600 |}} {{ :start:development:scripting_languages:gpu_rw_result.png?direct&600 |}}
Line 649: Line 671:
 === Memory Intensive Operations === === Memory Intensive Operations ===
  
-<file bash gpu_perf.jl>+You might be familiar with this example if you sumbled upon our [[https://mogonwiki.zdv.uni-mainz.de/dokuwiki/start:development:scripting_languages:matlab#submitting_a_matlab_job_on_gpus|MATLAB article]] or read it on purpose. At this point we would simply like to restate what we originally took from the [[https://de.mathworks.com/help/parallel-computing/measuring-gpu-performance.html|MATLAB Help Center]]: 
 + 
 +<blockquote> 
 +For operations where the number of floating-point computations performed per element read from or written to memory is high, the memory speed is much less important. In this case the number and speed of the floating-point units is the limiting factor. These operations are said to have high "computational density"
 + 
 +A good test of computational performance is a matrix-matrix multiply. For multiplying two $N \times N$ matrices, the total number of floating-point calculations is 
 +$$ FLOPS(N) = 2N^3 - N^2 $$ 
 + 
 +Two input matrices are read and one resulting matrix is written, for a total of $3N^2$ elements read or written. This gives a computational density of $(2N - 1)/3$ FLOP/element. Contrast this with plus as used above, which has a computational density of $1/2$ FLOP/element. 
 +<cite>MATLAB Help Center, [[https://de.mathworks.com/help/parallel-computing/measuring-gpu-performance.html#d123e33031|Measuring GPU Performance]]</cite> 
 +</blockquote> 
 + 
 +The difference to our MATLAB article is of course the adaptation to native Julia code. But even so, we have made a few alterations due to the use of the Julia language. When defining vectors or arrays, we have purposely chosen ''Float32'', since GPUs are faster when working with ''Float32'' than with ''Float64''. In addition ''CuArrys'' are by default ''Float32'', as well as functions like ''CUDA.rand'' or ''CUDA.zeros''
 + 
 +<file julia gpu_perf.jl>
 using LinearAlgebra using LinearAlgebra
 using Plots using Plots
Line 720: Line 756:
 savefig("gpu_perf_result.png") savefig("gpu_perf_result.png")
 </file> </file>
 +
 +The job script is quite ordinary. In this example, we only use one GPU and start Julia with four threads. For this we request one process with four cpus for multithreading.
  
 <file bash julia_gpu_perf_job.slurm> <file bash julia_gpu_perf_job.slurm>
 #!/bin/bash #!/bin/bash
-#SBATCH --account=m2_zdvhpc+#SBATCH --account=<YourHPC-Account>
 #SBATCH --job-name=gpu_perf #SBATCH --job-name=gpu_perf
 #SBATCH --output=%x_%j.out #SBATCH --output=%x_%j.out
Line 730: Line 768:
 #SBATCH --gres=gpu:1 #SBATCH --gres=gpu:1
 #SBATCH --time=0-00:10:00 #SBATCH --time=0-00:10:00
-#SBATCH --mem=11550+#SBATCH --mem=8192
 #SBATCH --ntasks=1 #SBATCH --ntasks=1
 #SBATCH --cpus-per-task=4 #SBATCH --cpus-per-task=4
Line 742: Line 780:
 julia --threads 4 gpu_perf.jl julia --threads 4 gpu_perf.jl
 </file> </file>
 +
 +You can submit the job by simply executing:
 +
 +<code bash>
 +sbatch julia_gpu_perf_job.slurm
 +</code>
 +
 +The job will be completed after acouple of minutes and you can view the output with:
 +
  
 <code bash> <code bash>
 cat gpu_perf*.out cat gpu_perf*.out
 </code> </code>
 +
 +The Output should resemble the following lines:
  
 <code bash> <code bash>
 Achieved peak calculation rates of 140.4 GFLOPS on CPU, 10134.1 GFLOPS on GPU Achieved peak calculation rates of 140.4 GFLOPS on CPU, 10134.1 GFLOPS on GPU
 </code> </code>
 +
 +The graphic generated in the script is shown below:
  
 {{ :start:development:scripting_languages:gpu_perf_result_julia.png?direct&600 |}} {{ :start:development:scripting_languages:gpu_perf_result_julia.png?direct&600 |}}
  
  • start/development/scripting_languages/julia.1618849215.txt.gz
  • Last modified: 2021/04/19 18:20
  • by jrutte02