[ Skip to the content ]

Institute of Formal and Applied Linguistics Wiki

• ÚFAL Wiki Home
• ÚFAL Home
• Sitemap
• RSS
• ÚFAL calendar
• Recent changes
• Login
• Hledat:

[ Back to the navigation ]

Differences

This shows you the differences between two versions of the page.

View differences: Side by SideInline Go

Link to this comparison view

Both sides previous revision Previous revision 2022/12/14 13:25 straka [Usage Examples] 2022/12/14 13:25 straka [Usage Examples] 2022/12/14 13:24 straka [Usage Examples] 2022/12/14 13:23 straka 2022/12/14 13:01 straka [Usage Examples] 2022/12/14 13:00 straka [Usage Examples] 2022/12/14 13:00 straka [Usage Examples] 2022/12/14 12:59 straka [Usage Examples] 2019/11/12 13:37 straka 2019/11/12 13:19 straka 2017/10/16 20:58 ufal [Using Python] 2017/10/16 20:56 ufal [Starting Interactive Shell] 2015/10/23 14:24 ufal 2014/11/11 09:31 straka 2014/11/11 09:28 straka 2014/11/10 15:42 straka 2014/11/10 15:36 straka 2014/11/10 15:36 straka 2014/11/10 15:35 straka 2014/11/10 15:32 straka 2014/11/10 15:17 straka 2014/11/10 15:15 straka created Go Next revision		Previous revision 2022/12/14 13:25 straka [Usage Examples] 2022/12/14 13:25 straka [Usage Examples] 2022/12/14 13:24 straka [Usage Examples] 2022/12/14 13:23 straka 2022/12/14 13:01 straka [Usage Examples] 2022/12/14 13:00 straka [Usage Examples] 2022/12/14 13:00 straka [Usage Examples] 2022/12/14 12:59 straka [Usage Examples] 2019/11/12 13:37 straka 2019/11/12 13:19 straka 2017/10/16 20:58 ufal [Using Python] 2017/10/16 20:56 ufal [Starting Interactive Shell] 2015/10/23 14:24 ufal 2014/11/11 09:31 straka 2014/11/11 09:28 straka 2014/11/10 15:42 straka 2014/11/10 15:36 straka 2014/11/10 15:36 straka 2014/11/10 15:35 straka 2014/11/10 15:32 straka 2014/11/10 15:17 straka 2014/11/10 15:15 straka created Go Next revision Both sides next revision
spark:using-python [2019/11/12 13:37] straka		spark:using-python [2022/12/14 13:01] straka [Usage Examples]
Line 16:		Line 16:
	Consider the following simple script computing 10 most frequent words of Czech Wikipedia:		Consider the following simple script computing 10 most frequent words of Czech Wikipedia:
	<file python>		<file python>
-	(sc.textFile("/net/projects/spark-example-data/wiki-cs", 3*sc.defaultParallelism)	+	(sc.textFile("/lnet/troja/data/npfl118/wiki/cs/wiki.txt", 3*sc.defaultParallelism)
	.flatMap(lambda line: line.split())		.flatMap(lambda line: line.split())
	.map(lambda word: (word, 1))		.map(lambda word: (word, 1))
-	.reduceByKey(lambda c1,c2: c1+c2)	+	.reduceByKey(lambda c1, c2: c1+c2)
	.sortBy(lambda word_count: word_count[1], ascending=False)		.sortBy(lambda word_count: word_count[1], ascending=False)
	.take(10))		.take(10))
	</file>		</file>
-	* run interactive shell using existing Spark cluster (i.e., inside ''spark-qrsh''), or start local Spark cluster using as many threads as there are cores if there is none:	+	* run interactive shell using existing Spark cluster (i.e., inside ''spark-srun''), or start local Spark cluster using as many threads as there are cores if there is none:
	<file>PYSPARK_DRIVER_PYTHON=ipython3 pyspark</file>		<file>PYSPARK_DRIVER_PYTHON=ipython3 pyspark</file>
	* run interactive shell with local Spark cluster using one thread:		* run interactive shell with local Spark cluster using one thread:
Line 31:		Line 31:
	<file>PYSPARK_DRIVER_PYTHON=ipython3 spark-qrsh 10 1G pyspark</file>		<file>PYSPARK_DRIVER_PYTHON=ipython3 spark-qrsh 10 1G pyspark</file>
-	Note that ''PYSPARK_DRIVER_PYTHON'' variable can be left out or specified in ''.bashrc'' (or similar).	+	Note that ''PYSPARK_DRIVER_PYTHON'' variable can be left out or specified in ''.bashrc'' (or other configuration files).
Line 59:		Line 59:
	.flatMap(lambda line: line.split())		.flatMap(lambda line: line.split())
	.map(lambda token: (token, 1))		.map(lambda token: (token, 1))
-	.reduceByKey(lambda x,y: x + y)	+	.reduceByKey(lambda x, y: x + y)
	.sortBy(lambda word_count: word_count[1], ascending=False)		.sortBy(lambda word_count: word_count[1], ascending=False)
	.saveAsTextFile(output))		.saveAsTextFile(output))
Line 65:		Line 65:
	</file>		</file>
-	* run ''word_count.py'' script inside existing Spark cluster (i.e., inside ''spark-qsub'' or ''spark-qrsh''), or start local Spark cluster using as many threads as there are cores if there is none:	+	* run ''word_count.py'' script inside existing Spark cluster (i.e., inside ''spark-sbatch'' or ''spark-srun''), or start local Spark cluster using as many threads as there are cores if there is none:
-	<file>spark-submit word_count.py /net/projects/spark-example-data/wiki-cs outdir</file>	+	<file>spark-submit word_count.py /lnet/troja/data/npfl118/wiki/cs/wiki.txt outdir</file>
	* run ''word_count.py'' script with local Spark cluster using one thread:		* run ''word_count.py'' script with local Spark cluster using one thread:
-	<file>MASTER=local spark-submit word_count.py /net/projects/spark-example-data/wiki-cs outdir</file>	+	<file>MASTER=local spark-submit word_count.py /lnet/troja/data/npfl118/wiki/cs/wiki.txt outdir</file>
-	* start Spark cluster (10 machines, 1GB RAM each) on SGE and run ''word_count.py'' script:	+	* start Spark cluster (10 machines, @GB RAM each) using Slurm SGE and run ''word_count.py'' script:
-	<file>spark-qsub 10 1G spark-submit word_count.py /net/projects/spark-example-data/wiki-cs outdir</file>	+	<file>spark-sbatch 10 2G spark-submit word_count.py /lnet/troja/data/npfl118/wiki/cs/wiki.txt outdir</file>
	===== Using Virtual Environments =====		===== Using Virtual Environments =====

---

[ Back to the navigation ] [ Back to the content ]