[ Skip to the content ]

Institute of Formal and Applied Linguistics Wiki


[ Back to the navigation ]

Differences

This shows you the differences between two versions of the page.

Link to this comparison view

Both sides previous revision Previous revision
Next revision Both sides next revision
spark:using-python [2022/12/14 13:00]
straka [Usage Examples]
spark:using-python [2022/12/14 13:01]
straka [Usage Examples]
Line 65: Line 65:
 </file> </file>
  
-  * run ''word_count.py'' script inside existing Spark cluster (i.e., inside ''spark-qsub'' or ''spark-qrsh''), or start local Spark cluster using as many threads as there are cores if there is none: +  * run ''word_count.py'' script inside existing Spark cluster (i.e., inside ''spark-sbatch'' or ''spark-srun''), or start local Spark cluster using as many threads as there are cores if there is none: 
-  <file>spark-submit word_count.py /net/projects/spark-example-data/wiki-cs outdir</file>+  <file>spark-submit word_count.py /lnet/troja/data/npfl118/wiki/cs/wiki.txt outdir</file>
   * run ''word_count.py'' script with local Spark cluster using one thread:   * run ''word_count.py'' script with local Spark cluster using one thread:
-  <file>MASTER=local spark-submit word_count.py /net/projects/spark-example-data/wiki-cs outdir</file> +  <file>MASTER=local spark-submit word_count.py /lnet/troja/data/npfl118/wiki/cs/wiki.txt outdir</file> 
-  * start Spark cluster (10 machines, 1GB RAM each) on SGE and run ''word_count.py'' script: +  * start Spark cluster (10 machines, @GB RAM each) using Slurm SGE and run ''word_count.py'' script: 
-  <file>spark-qsub 10 1G spark-submit word_count.py /net/projects/spark-example-data/wiki-cs outdir</file>+  <file>spark-sbatch 10 2G spark-submit word_count.py /lnet/troja/data/npfl118/wiki/cs/wiki.txt outdir</file>
  
 ===== Using Virtual Environments ===== ===== Using Virtual Environments =====

[ Back to the navigation ] [ Back to the content ]