[ Skip to the content ]

Institute of Formal and Applied Linguistics Wiki

[ Back to the navigation ]

You are here: start » spark » using-python

Differences

This shows you the differences between two versions of the page.

Link to this comparison view

Both sides previous revision Previous revision
spark:using-python [2022/12/14 13:25]
straka [Usage Examples] 		spark:using-python [2026/10/14 19:56] (current)
straka Drop the ipython3 override.
Line 8: Line 8:
 <file>pyspark</file> <file>pyspark</file>
  
-Better interactive shell with code completion using ''ipython3'' (do `pip3 install --user ipython` if you do not have it) can be started using: 
-<file>PYSPARK_DRIVER_PYTHON=ipython3 pyspark</file> 
  
 As described in [[running-spark-on-single-machine-or-on-cluster|Running Spark on Single Machine or on Cluster]], environmental variable ''MASTER'' specifies which Spark master to use (or whether to start a local one). As described in [[running-spark-on-single-machine-or-on-cluster|Running Spark on Single Machine or on Cluster]], environmental variable ''MASTER'' specifies which Spark master to use (or whether to start a local one).
Line 25: Line 23:
  
   * run interactive shell using existing Spark cluster (i.e., inside ''spark-srun''), or start local Spark cluster using as many threads as there are cores if there is none:   * run interactive shell using existing Spark cluster (i.e., inside ''spark-srun''), or start local Spark cluster using as many threads as there are cores if there is none:
-  <file>PYSPARK_DRIVER_PYTHON=ipython3 pyspark</file>+  <file>pyspark</file>
   * run interactive shell with local Spark cluster using one thread:   * run interactive shell with local Spark cluster using one thread:
-  <file>MASTER=local PYSPARK_DRIVER_PYTHON=ipython3 pyspark</file>+  <file>MASTER=local pyspark</file>
   * start Spark cluster (10 machines, 2GB RAM each) on Slurm and run interactive shell:   * start Spark cluster (10 machines, 2GB RAM each) on Slurm and run interactive shell:
-  <file>PYSPARK_DRIVER_PYTHON=ipython3 spark-srun 10 2G pyspark</file> +  <file>spark-srun 10 2G pyspark</file>
- +
-Note that ''PYSPARK_DRIVER_PYTHON'' variable can be left out or specified in ''.bashrc'' (or other configuration files).+
  
  

[ Back to the navigation ] [ Back to the content ]