[ Skip to the content ]

Institute of Formal and Applied Linguistics Wiki

[ Back to the navigation ]

You are here: start » spark » running-spark-on-single-machine-or-on-cluster

Differences

This shows you the differences between two versions of the page.

Link to this comparison view

Both sides previous revision Previous revision
Next revision 		Previous revision
Next revision Both sides next revision
spark:running-spark-on-single-machine-or-on-cluster [2022/12/14 12:53]
straka [Memory Specification] 		spark:running-spark-on-single-machine-or-on-cluster [2022/12/14 12:54]
straka [Memory Specification]
Line 29: Line 29:
 TL;DR: Good default is ''2G'. TL;DR: Good default is ''2G'.
  
-The memory for each worker is specified using the following format: <file>spark_memory_per_workerG[:memory_per_Python_processG]</file>+The memory for each worker is specified using the following format: 
 +<file>spark_memory_per_workerG[:memory_per_Python_processG]</file> 
 The Spark memory limits the Java heap, and half of it is reserved for memory storage of cached RDDs. The second value sets a memory limit of every Python process and is by default set to ''2G''. The Spark memory limits the Java heap, and half of it is reserved for memory storage of cached RDDs. The second value sets a memory limit of every Python process and is by default set to ''2G''.
 +
 ==== Examples ==== ==== Examples ====
  

[ Back to the navigation ] [ Back to the content ]