[ Skip to the content ]

Institute of Formal and Applied Linguistics Wiki


[ Back to the navigation ]

Differences

This shows you the differences between two versions of the page.

Link to this comparison view

Both sides previous revision Previous revision
Next revision Both sides next revision
spark:using-python [2017/10/16 20:56]
ufal [Starting Interactive Shell]
spark:using-python [2017/10/16 20:58]
ufal [Using Python]
Line 25: Line 25:
  
   * run interactive shell using existing Spark cluster (i.e., inside ''spark-qrsh''), or start local Spark cluster using as many threads as there are cores if there is none:   * run interactive shell using existing Spark cluster (i.e., inside ''spark-qrsh''), or start local Spark cluster using as many threads as there are cores if there is none:
-  <file>IPYTHON=pyspark</file>+  <file>PYSPARK_DRIVER_PYTHON=ipython pyspark</file>
   * run interactive shell with local Spark cluster using one thread:   * run interactive shell with local Spark cluster using one thread:
-  <file>MASTER=local IPYTHON=pyspark</file>+  <file>MASTER=local PYSPARK_DRIVER_PYTHON=ipython pyspark</file>
   * start Spark cluster (10 machines, 1GB RAM each) on SGE and run interactive shell:   * start Spark cluster (10 machines, 1GB RAM each) on SGE and run interactive shell:
-  <file>IPYTHON=spark-qrsh 10 1G pyspark</file>+  <file>PYSPARK_DRIVER_PYTHON=ipython spark-qrsh 10 1G pyspark</file>
  
-Note that ''IPYTHON'' variable can be left out or specified in ''.bashrc'' (or similar).+Note that ''PYSPARK_DRIVER_PYTHON'' variable can be left out or specified in ''.bashrc'' (or similar).
  
  

[ Back to the navigation ] [ Back to the content ]