[ Skip to the content ]

Institute of Formal and Applied Linguistics Wiki


[ Back to the navigation ]

Differences

This shows you the differences between two versions of the page.

Link to this comparison view

Both sides previous revision Previous revision
spark:using-python [2017/10/16 20:56]
ufal [Starting Interactive Shell]
spark:using-python [2017/10/16 20:58] (current)
ufal [Using Python]
Line 25: Line 25:
  
   * run interactive shell using existing Spark cluster (i.e., inside ''​spark-qrsh''​),​ or start local Spark cluster using as many threads as there are cores if there is none:   * run interactive shell using existing Spark cluster (i.e., inside ''​spark-qrsh''​),​ or start local Spark cluster using as many threads as there are cores if there is none:
-  <​file>​IPYTHON=pyspark</​file>​+  <​file>​PYSPARK_DRIVER_PYTHON=ipython ​pyspark</​file>​
   * run interactive shell with local Spark cluster using one thread:   * run interactive shell with local Spark cluster using one thread:
-  <​file>​MASTER=local ​IPYTHON=pyspark</​file>​+  <​file>​MASTER=local ​PYSPARK_DRIVER_PYTHON=ipython ​pyspark</​file>​
   * start Spark cluster (10 machines, 1GB RAM each) on SGE and run interactive shell:   * start Spark cluster (10 machines, 1GB RAM each) on SGE and run interactive shell:
-  <​file>​IPYTHON=spark-qrsh 10 1G pyspark</​file>​+  <​file>​PYSPARK_DRIVER_PYTHON=ipython ​spark-qrsh 10 1G pyspark</​file>​
  
-Note that ''​IPYTHON''​ variable can be left out or specified in ''​.bashrc''​ (or similar).+Note that ''​PYSPARK_DRIVER_PYTHON''​ variable can be left out or specified in ''​.bashrc''​ (or similar).
  
  

[ Back to the navigation ] [ Back to the content ]