[ Skip to the content ]

Institute of Formal and Applied Linguistics Wiki

• ÚFAL Wiki Home
• ÚFAL Home
• Sitemap
• RSS
• ÚFAL calendar
• Recent changes
• Login
• Hledat:

[ Back to the navigation ]

Differences

This shows you the differences between two versions of the page.

View differences: Side by SideInline Go

Link to this comparison view

Both sides previous revision Previous revision 2022/12/14 13:25 straka [Usage Examples] 2022/12/14 13:25 straka [Usage Examples] 2022/12/14 13:24 straka [Usage Examples] 2022/12/14 13:23 straka 2022/12/14 13:01 straka [Usage Examples] 2022/12/14 13:00 straka [Usage Examples] 2022/12/14 13:00 straka [Usage Examples] 2022/12/14 12:59 straka [Usage Examples] 2019/11/12 13:37 straka 2019/11/12 13:19 straka 2017/10/16 20:58 ufal [Using Python] 2017/10/16 20:56 ufal [Starting Interactive Shell] 2015/10/23 14:24 ufal 2014/11/11 09:31 straka 2014/11/11 09:28 straka 2014/11/10 15:42 straka 2014/11/10 15:36 straka 2014/11/10 15:36 straka 2014/11/10 15:35 straka 2014/11/10 15:32 straka 2014/11/10 15:17 straka 2014/11/10 15:15 straka created Go Next revision		Previous revision 2022/12/14 13:25 straka [Usage Examples] 2022/12/14 13:25 straka [Usage Examples] 2022/12/14 13:24 straka [Usage Examples] 2022/12/14 13:23 straka 2022/12/14 13:01 straka [Usage Examples] 2022/12/14 13:00 straka [Usage Examples] 2022/12/14 13:00 straka [Usage Examples] 2022/12/14 12:59 straka [Usage Examples] 2019/11/12 13:37 straka 2019/11/12 13:19 straka 2017/10/16 20:58 ufal [Using Python] 2017/10/16 20:56 ufal [Starting Interactive Shell] 2015/10/23 14:24 ufal 2014/11/11 09:31 straka 2014/11/11 09:28 straka 2014/11/10 15:42 straka 2014/11/10 15:36 straka 2014/11/10 15:36 straka 2014/11/10 15:35 straka 2014/11/10 15:32 straka 2014/11/10 15:17 straka 2014/11/10 15:15 straka created Go
spark:using-python [2014/11/11 09:31] straka		spark:using-python [2022/12/14 13:25] (current) straka [Usage Examples]
Line 8:		Line 8:
	<file>pyspark</file>		<file>pyspark</file>
-	Better interactive shell with code completion using ''ipython'' (installed everywhere on cluster; ask our IT if you want to have it installed on your workstations too) can be started using:	+	Better interactive shell with code completion using ''ipython3'' (do `pip3 install --user ipython` if you do not have it) can be started using:
-	<file>IPYTHON=1 pyspark</file>	+	<file>PYSPARK_DRIVER_PYTHON=ipython3 pyspark</file>
	As described in [[running-spark-on-single-machine-or-on-cluster|Running Spark on Single Machine or on Cluster]], environmental variable ''MASTER'' specifies which Spark master to use (or whether to start a local one).		As described in [[running-spark-on-single-machine-or-on-cluster|Running Spark on Single Machine or on Cluster]], environmental variable ''MASTER'' specifies which Spark master to use (or whether to start a local one).
Line 19:		Line 19:
	.flatMap(lambda line: line.split())		.flatMap(lambda line: line.split())
	.map(lambda word: (word, 1))		.map(lambda word: (word, 1))
-	.reduceByKey(lambda c1,c2: c1+c2)	+	.reduceByKey(lambda c1, c2: c1+c2)
-	.sortBy(lambda (word,count): count, ascending=False)	+	.sortBy(lambda word_count: word_count[1], ascending=False)
	.take(10))		.take(10))
	</file>		</file>
-	* run interactive shell using existing Spark cluster (i.e., inside ''spark-qrsh''), or start local Spark cluster using as many threads as there are cores if there is none:	+	* run interactive shell using existing Spark cluster (i.e., inside ''spark-srun''), or start local Spark cluster using as many threads as there are cores if there is none:
-	<file>IPYTHON=1 pyspark</file>	+	<file>PYSPARK_DRIVER_PYTHON=ipython3 pyspark</file>
	* run interactive shell with local Spark cluster using one thread:		* run interactive shell with local Spark cluster using one thread:
-	<file>MASTER=local IPYTHON=1 pyspark</file>	+	<file>MASTER=local PYSPARK_DRIVER_PYTHON=ipython3 pyspark</file>
-	* start Spark cluster (10 machines, 1GB RAM each) on SGE and run interactive shell:	+	* start Spark cluster (10 machines, 2GB RAM each) on Slurm and run interactive shell:
-	<file>IPYTHON=1 spark-qrsh 10 1G pyspark</file>	+	<file>PYSPARK_DRIVER_PYTHON=ipython3 spark-srun 10 2G pyspark</file>
-	Note that ''IPYTHON'' variable can be left out or specified in ''.bashrc'' (or similar).	+	Note that ''PYSPARK_DRIVER_PYTHON'' variable can be left out or specified in ''.bashrc'' (or other configuration files).
Line 59:		Line 59:
	.flatMap(lambda line: line.split())		.flatMap(lambda line: line.split())
	.map(lambda token: (token, 1))		.map(lambda token: (token, 1))
-	.reduceByKey(lambda x,y: x + y)	+	.reduceByKey(lambda x, y: x + y)
-	.sortBy(lambda (word,count): count, ascending=False)	+	.sortBy(lambda word_count: word_count[1], ascending=False)
	.saveAsTextFile(output))		.saveAsTextFile(output))
	sc.stop()		sc.stop()
	</file>		</file>
-	* run ''word_count.py'' script inside existing Spark cluster (i.e., inside ''spark-qsub'' or ''spark-qrsh''), or start local Spark cluster using as many threads as there are cores if there is none:	+	* run ''word_count.py'' script inside existing Spark cluster (i.e., inside ''spark-sbatch'' or ''spark-srun''), or start local Spark cluster using as many threads as there are cores if there is none:
	<file>spark-submit word_count.py /net/projects/spark-example-data/wiki-cs outdir</file>		<file>spark-submit word_count.py /net/projects/spark-example-data/wiki-cs outdir</file>
	* run ''word_count.py'' script with local Spark cluster using one thread:		* run ''word_count.py'' script with local Spark cluster using one thread:
	<file>MASTER=local spark-submit word_count.py /net/projects/spark-example-data/wiki-cs outdir</file>		<file>MASTER=local spark-submit word_count.py /net/projects/spark-example-data/wiki-cs outdir</file>
-	* start Spark cluster (10 machines, 1GB RAM each) on SGE and run ''word_count.py'' script:	+	* start Spark cluster (10 machines, @GB RAM each) using Slurm and run ''word_count.py'' script:
-	<file>spark-qsub 10 1G spark-submit word_count.py /net/projects/spark-example-data/wiki-cs outdir</file>	+	<file>spark-sbatch 10 2G spark-submit word_count.py /net/projects/spark-example-data/wiki-cs outdir</file>
		+
		+	===== Using Virtual Environments =====
		+
		+	If you want to use specific virtual environment in your Spark job, use
		+	<file>PYSPARK_PYTHON=path_to_python_in_venv [pyspark|spark-submit]</file>

---

[ Back to the navigation ] [ Back to the content ]