In order to use Spark, environment has to bee set up according to Using Spark in UFAL Environment.

When Spark computation starts, it uses environment variable MASTER to determine the mode of computation. The following values are possible:

• local: Run locally using single thread.
• local[N] (e.g., local[2] or local[4]): Run locally using N threads.
• local[*] (default if MASTER variable does not exist): Run locally using as many threads as there are processor cores.
• spark://master_address:master_port: Run in a distributed fashion using specified master.

Spark computations can be started both on desktop machines and on cluster machines, either by specifying MASTER to one of local modes, or by not specifying MASTER at all (local[*] is used then).

Note that when you use qrsh or qsub, your job is usually expected to use one core, so you should specify MASTER=local. If you do not, Spark will use all cores on the machine, even though SGE gave you only one.

Spark cluster can be started using SGE. The cluster is user-specific, but it can be used for multiple Spark computations.

The Spark cluster can be started using one of the following two commands:

• spark-qsub: start Spark cluster and perform qsub

  [SGE_OPTS=additional_SGE_args] spark-qsub workers memory command [arguments]

• spark-qrsh: start Spark cluster and perform qrsh

  [SGE_OPTS=additional_SGE_args] spark-qrsh workers memory [command arguments]

Both spark-qsub and spark-qrsh commands start Spark cluster with specified number of workers, each with given number of memory. Then they set MASTER and SPARK_ADDRESS to address of the Spark master and SPARK_WEBUI to http address of the master web interface. Both these values are also written on standard output and added to SGE job metadata. Lastly, specified command is started either as using qsub or qrsh. Note that when spark-qrsh is used, no command may be specified, in which case an interactive shell is opened.

Start Spark cluster with 10 machines 1GB RAM each and then run interactive shell. The cluster stops after the shell is exited.

spark-qrsh 10 1G

Start Spark cluster with 20 machines 512MB RAM each. The cluster has to be stopped manually using qdel.

spark-qsub 20 512m sleep infinity

Memory specification used for master and worker heap size (and for mem_free SGE constraint) must be specified. The memory can be specified either in bytes, or using kK/mM/gG suffix. A reasonable default value is 512M or 1G.

Additional qrsh or qsub options can be specified in SGE_OPTS environmental variable (not as spark-qsub or spark-qrsh arguments), as in the following example which schedules the Spark master and workers to machines different from hyperion* and pandora*:

SGE_OPTS='-q *@!(hyperion*|pandora*)' spark-qrsh 10 1G