[ Skip to the content ]

Institute of Formal and Applied Linguistics Wiki

[ Back to the navigation ]

You are here: start » spark » using-python

Differences

This shows you the differences between two versions of the page.

Link to this comparison view

Both sides previous revision Previous revision
Next revision 		Previous revision
Last revision Both sides next revision
spark:using-python [2022/12/14 13:23]
straka 		spark:using-python [2022/12/14 13:25]
straka [Usage Examples]
Line 28: Line 28:
   * run interactive shell with local Spark cluster using one thread:   * run interactive shell with local Spark cluster using one thread:
   <file>MASTER=local PYSPARK_DRIVER_PYTHON=ipython3 pyspark</file>   <file>MASTER=local PYSPARK_DRIVER_PYTHON=ipython3 pyspark</file>
-  * start Spark cluster (10 machines, 1GB RAM each) on SGE and run interactive shell: +  * start Spark cluster (10 machines, 2GB RAM each) on Slurm and run interactive shell: 
-  <file>PYSPARK_DRIVER_PYTHON=ipython3 spark-qrsh 10 1G pyspark</file>+  <file>PYSPARK_DRIVER_PYTHON=ipython3 spark-srun 10 2G pyspark</file>
  
 Note that ''PYSPARK_DRIVER_PYTHON'' variable can be left out or specified in ''.bashrc'' (or other configuration files). Note that ''PYSPARK_DRIVER_PYTHON'' variable can be left out or specified in ''.bashrc'' (or other configuration files).

[ Back to the navigation ] [ Back to the content ]