Differences
This shows you the differences between two versions of the page.
Next revision | Previous revision | ||
spark [2018/11/28 14:20] kbob01 created |
— (current) | ||
---|---|---|---|
Line 1: | Line 1: | ||
- | ====== Apache Spark on Mogon ====== | ||
- | Apache Spark is an open source framework for big data applications using Java Virtual Machines. | ||
- | |||
- | When using it on Mogon, one typically occupies a full node, as Spark requires a lot of resources. | ||
- | |||
- | After having occupied a node there are two possible use cases. | ||
- | === Job based usage === | ||
- | If you have an already packaged application that shall be submitted as a job, the the following script could be used to start a scala application (packed to myJar.jar) with an the entry point in the class Main in the package main. | ||
- | |||
- | <code bash> | ||
- | #!/bin/bash | ||
- | |||
- | # load module | ||
- | module load devel/ | ||
- | |||
- | # start application | ||
- | spark-submit --driver-memory 8G --master local[*] --class main.Main myJar.jar | ||
- | |||
- | </ | ||
- | |||
- | |||
- | The option '' | ||
- | The option '' | ||
- | |||
- | === Interactive usage === | ||
- | If you want to use it in a more explorative manner, then using the spark shell by '' |