[ Skip to the content ]

Institute of Formal and Applied Linguistics Wiki

[ Back to the navigation ]

You are here: start » spark » using-python

Differences

This shows you the differences between two versions of the page.

Link to this comparison view

Both sides previous revision Previous revision
Next revision 		Previous revision
Next revision Both sides next revision
spark:using-python [2022/12/14 13:00]
straka [Usage Examples] 		spark:using-python [2022/12/14 13:01]
straka [Usage Examples]
Line 59: Line 59:
    .flatMap(lambda line: line.split())    .flatMap(lambda line: line.split())
    .map(lambda token: (token, 1))    .map(lambda token: (token, 1))
-   .reduceByKey(lambda x,y: x + y)+   .reduceByKey(lambda x, y: x + y)
    .sortBy(lambda word_count: word_count[1], ascending=False)    .sortBy(lambda word_count: word_count[1], ascending=False)
    .saveAsTextFile(output))    .saveAsTextFile(output))
Line 65: Line 65:
 </file> </file>
  
-  * run ''word_count.py'' script inside existing Spark cluster (i.e., inside ''spark-qsub'' or ''spark-qrsh''), or start local Spark cluster using as many threads as there are cores if there is none: +  * run ''word_count.py'' script inside existing Spark cluster (i.e., inside ''spark-sbatch'' or ''spark-srun''), or start local Spark cluster using as many threads as there are cores if there is none: 
-  <file>spark-submit word_count.py /net/projects/spark-example-data/wiki-cs outdir</file>+  <file>spark-submit word_count.py /lnet/troja/data/npfl118/wiki/cs/wiki.txt outdir</file>
   * run ''word_count.py'' script with local Spark cluster using one thread:   * run ''word_count.py'' script with local Spark cluster using one thread:
-  <file>MASTER=local spark-submit word_count.py /net/projects/spark-example-data/wiki-cs outdir</file> +  <file>MASTER=local spark-submit word_count.py /lnet/troja/data/npfl118/wiki/cs/wiki.txt outdir</file> 
-  * start Spark cluster (10 machines, 1GB RAM each) on SGE and run ''word_count.py'' script: +  * start Spark cluster (10 machines, @GB RAM each) using Slurm SGE and run ''word_count.py'' script: 
-  <file>spark-qsub 10 1G spark-submit word_count.py /net/projects/spark-example-data/wiki-cs outdir</file>+  <file>spark-sbatch 10 2G spark-submit word_count.py /lnet/troja/data/npfl118/wiki/cs/wiki.txt outdir</file>
  
 ===== Using Virtual Environments ===== ===== Using Virtual Environments =====

[ Back to the navigation ] [ Back to the content ]