[ Skip to the content ]

Institute of Formal and Applied Linguistics Wiki

• ÚFAL Wiki Home
• ÚFAL Home
• Sitemap
• RSS
• ÚFAL calendar
• Recent changes
• Login
• Hledat:

[ Back to the navigation ]

Differences

This shows you the differences between two versions of the page.

View differences: Side by SideInline Go

Link to this comparison view

Both sides previous revision Previous revision 2022/12/14 13:25 straka [Usage Examples] 2022/12/14 13:25 straka [Usage Examples] 2022/12/14 13:24 straka [Usage Examples] 2022/12/14 13:23 straka 2022/12/14 13:01 straka [Usage Examples] 2022/12/14 13:00 straka [Usage Examples] 2022/12/14 13:00 straka [Usage Examples] 2022/12/14 12:59 straka [Usage Examples] 2019/11/12 13:37 straka 2019/11/12 13:19 straka 2017/10/16 20:58 ufal [Using Python] 2017/10/16 20:56 ufal [Starting Interactive Shell] 2015/10/23 14:24 ufal 2014/11/11 09:31 straka 2014/11/11 09:28 straka 2014/11/10 15:42 straka 2014/11/10 15:36 straka 2014/11/10 15:36 straka 2014/11/10 15:35 straka 2014/11/10 15:32 straka 2014/11/10 15:17 straka 2014/11/10 15:15 straka created Go Next revision		Previous revision 2022/12/14 13:25 straka [Usage Examples] 2022/12/14 13:25 straka [Usage Examples] 2022/12/14 13:24 straka [Usage Examples] 2022/12/14 13:23 straka 2022/12/14 13:01 straka [Usage Examples] 2022/12/14 13:00 straka [Usage Examples] 2022/12/14 13:00 straka [Usage Examples] 2022/12/14 12:59 straka [Usage Examples] 2019/11/12 13:37 straka 2019/11/12 13:19 straka 2017/10/16 20:58 ufal [Using Python] 2017/10/16 20:56 ufal [Starting Interactive Shell] 2015/10/23 14:24 ufal 2014/11/11 09:31 straka 2014/11/11 09:28 straka 2014/11/10 15:42 straka 2014/11/10 15:36 straka 2014/11/10 15:36 straka 2014/11/10 15:35 straka 2014/11/10 15:32 straka 2014/11/10 15:17 straka 2014/11/10 15:15 straka created Go
spark:using-python [2022/12/14 13:23] straka		spark:using-python [2022/12/14 13:25] (current) straka [Usage Examples]
Line 28:		Line 28:
	* run interactive shell with local Spark cluster using one thread:		* run interactive shell with local Spark cluster using one thread:
	<file>MASTER=local PYSPARK_DRIVER_PYTHON=ipython3 pyspark</file>		<file>MASTER=local PYSPARK_DRIVER_PYTHON=ipython3 pyspark</file>
-	* start Spark cluster (10 machines, 1GB RAM each) on SGE and run interactive shell:	+	* start Spark cluster (10 machines, 2GB RAM each) on Slurm and run interactive shell:
-	<file>PYSPARK_DRIVER_PYTHON=ipython3 spark-qrsh 10 1G pyspark</file>	+	<file>PYSPARK_DRIVER_PYTHON=ipython3 spark-srun 10 2G pyspark</file>
	Note that ''PYSPARK_DRIVER_PYTHON'' variable can be left out or specified in ''.bashrc'' (or other configuration files).		Note that ''PYSPARK_DRIVER_PYTHON'' variable can be left out or specified in ''.bashrc'' (or other configuration files).
Line 69:		Line 69:
	* run ''word_count.py'' script with local Spark cluster using one thread:		* run ''word_count.py'' script with local Spark cluster using one thread:
	<file>MASTER=local spark-submit word_count.py /net/projects/spark-example-data/wiki-cs outdir</file>		<file>MASTER=local spark-submit word_count.py /net/projects/spark-example-data/wiki-cs outdir</file>
-	* start Spark cluster (10 machines, @GB RAM each) using Slurm SGE and run ''word_count.py'' script:	+	* start Spark cluster (10 machines, @GB RAM each) using Slurm and run ''word_count.py'' script:
	<file>spark-sbatch 10 2G spark-submit word_count.py /net/projects/spark-example-data/wiki-cs outdir</file>		<file>spark-sbatch 10 2G spark-submit word_count.py /net/projects/spark-example-data/wiki-cs outdir</file>

---

[ Back to the navigation ] [ Back to the content ]