[ Skip to the content ]

Institute of Formal and Applied Linguistics Wiki

[ Back to the navigation ]

You are here: start » spark » using-python

Differences

This shows you the differences between two versions of the page.

Link to this comparison view

Both sides previous revision Previous revision
Next revision 		Previous revision
spark:using-python [2022/12/14 13:24]
straka [Usage Examples] 		spark:using-python [2022/12/14 13:25] (current)
straka [Usage Examples]
Line 28: Line 28:
   * run interactive shell with local Spark cluster using one thread:   * run interactive shell with local Spark cluster using one thread:
   <file>MASTER=local PYSPARK_DRIVER_PYTHON=ipython3 pyspark</file>   <file>MASTER=local PYSPARK_DRIVER_PYTHON=ipython3 pyspark</file>
-  * start Spark cluster (10 machines, @GB RAM each) on Slurm and run interactive shell:+  * start Spark cluster (10 machines, 2GB RAM each) on Slurm and run interactive shell:
   <file>PYSPARK_DRIVER_PYTHON=ipython3 spark-srun 10 2G pyspark</file>   <file>PYSPARK_DRIVER_PYTHON=ipython3 spark-srun 10 2G pyspark</file>
  
Line 69: Line 69:
   * run ''word_count.py'' script with local Spark cluster using one thread:   * run ''word_count.py'' script with local Spark cluster using one thread:
   <file>MASTER=local spark-submit word_count.py /net/projects/spark-example-data/wiki-cs outdir</file>   <file>MASTER=local spark-submit word_count.py /net/projects/spark-example-data/wiki-cs outdir</file>
-  * start Spark cluster (10 machines, @GB RAM each) using Slurm SGE and run ''word_count.py'' script:+  * start Spark cluster (10 machines, @GB RAM each) using Slurm and run ''word_count.py'' script:
   <file>spark-sbatch 10 2G spark-submit word_count.py /net/projects/spark-example-data/wiki-cs outdir</file>   <file>spark-sbatch 10 2G spark-submit word_count.py /net/projects/spark-example-data/wiki-cs outdir</file>
  

[ Back to the navigation ] [ Back to the content ]