Differences
This shows you the differences between two versions of the page.
Both sides previous revision
Previous revision
Next revision
|
Previous revision
Next revision
Both sides next revision
|
spark:running-spark-on-single-machine-or-on-cluster [2014/11/10 17:25] straka |
spark:running-spark-on-single-machine-or-on-cluster [2014/11/11 09:19] straka |
* ''spark-qrsh'': start Spark cluster and perform qrsh <file>[SGE_OPTS=additional_SGE_args] spark-qrsh workers memory [command arguments]</file> | * ''spark-qrsh'': start Spark cluster and perform qrsh <file>[SGE_OPTS=additional_SGE_args] spark-qrsh workers memory [command arguments]</file> |
| |
Both ''spark-qsub'' and ''spark-qrsh'' commands start Spark cluster with specified number of workers, each with given number of memory. Then they set ''MASTER'' and ''SPARK_ADDRESS'' to address of the Spark master and ''SPARK_WEBUI'' to http address of the master web interface. Both these values are also written on standard output and added to SGE job metadata. Lastly, specified command is started either as using ''qsub'' or ''qrsh''. Note that when ''spark-qrsh'' is used, no command may be specified, in which case an interactive shell is opened. | Both ''spark-qsub'' and ''spark-qrsh'' commands start Spark cluster with specified number of workers, each with given number of memory. Then they set ''MASTER'' and ''SPARK_ADDRESS'' to address of the Spark master and ''SPARK_WEBUI'' to http address of the master web interface. Both these values are also written on standard output and added to SGE job metadata. Lastly, specified command is started either as using ''qsub'' or ''qrsh''. Note that when ''spark-qrsh'' is used, the command may be empty, in which case an interactive shell is opened. |
| |
==== Memory Specification ==== | ==== Memory Specification ==== |
Start Spark cluster with 20 machines 512MB RAM each. The cluster has to be stopped manually using ''qdel''. | Start Spark cluster with 20 machines 512MB RAM each. The cluster has to be stopped manually using ''qdel''. |
<file>spark-qsub 20 512m sleep infinity</file> | <file>spark-qsub 20 512m sleep infinity</file> |
| |
Note that a running Spark cluster can currently be used only from our cluster machines (connections to a running SGE Spark cluster from my workstation time-out). | Note that a running Spark cluster can currently be used only from our cluster machines (connections to a running SGE Spark cluster from my workstation time-out). |
| |