Differences
This shows you the differences between two versions of the page.
Next revision | Previous revision Next revision Both sides next revision | ||
spark:running-spark-on-single-machine-or-on-cluster [2014/11/07 16:09] straka created |
spark:running-spark-on-single-machine-or-on-cluster [2022/12/14 12:54] straka [Memory Specification] |
||
---|---|---|---|
Line 11: | Line 11: | ||
===== Running Spark on Single Machine ===== | ===== Running Spark on Single Machine ===== | ||
- | Spark computations can be started both on desktop machines and on cluster machines, either by specifying '' | + | Spark computations can be started both on desktop machines and on cluster machines, either by specifying '' |
- | Note that when you use '' | + | Note that when you use '' |
===== Starting Spark Cluster | ===== Starting Spark Cluster | ||
+ | |||
+ | Spark cluster can be started using Slurm. The cluster is user-specific, | ||
+ | |||
+ | The Spark cluster can be started using one of the following two commands: | ||
+ | * '' | ||
+ | * '' | ||
+ | |||
+ | Both '' | ||
+ | |||
+ | ==== Memory Specification ==== | ||
+ | |||
+ | TL;DR: Good default is '' | ||
+ | |||
+ | The memory for each worker is specified using the following format: < | ||
+ | |||
+ | The Spark memory limits the Java heap, and half of it is reserved for memory storage of cached RDDs. The second value sets a memory limit of every Python process and is by default set to '' | ||
+ | |||
+ | ==== Examples ==== | ||
+ | |||
+ | Start Spark cluster with 10 machines 1GB RAM each and then run interactive shell. The cluster stops after the shell is exited. | ||
+ | < | ||
+ | |||
+ | Start Spark cluster with 20 machines 512MB RAM each. The cluster has to be stopped manually using '' | ||
+ | < | ||
+ | |||
+ | Note that a running Spark cluster can currently be used only from other cluster machines (connections to a running SGE Spark cluster from my workstation ends with timeout). | ||
+ | |||
+ | ==== Additional SGE Options ==== | ||
+ | |||
+ | Additional '' | ||
+ | < | ||