Differences
This shows you the differences between two versions of the page.
| Both sides previous revision
Previous revision
|
Next revision
Both sides next revision
|
spark:using-scala [2015/07/27 18:34]
straka |
spark:using-scala [2015/07/27 18:46]
straka |
| |
| * run ''word_count'' application inside existing Spark cluster (i.e., inside ''spark-qsub'' or ''spark-qrsh''), or start local Spark cluster using as many threads as there are cores if there is none: | * run ''word_count'' application inside existing Spark cluster (i.e., inside ''spark-qsub'' or ''spark-qrsh''), or start local Spark cluster using as many threads as there are cores if there is none: |
| <file>spark-submit --class Main target/scala-2.10/word_count_2.10-1.0.jar /net/projects/spark-example-data/wiki-cs outdir</file> | <file>spark-submit target/scala-2.10/word_count_2.10-1.0.jar /net/projects/spark-example-data/wiki-cs outdir</file> |
| * run ''word_count'' application with local Spark cluster using one thread: | * run ''word_count'' application with local Spark cluster using one thread: |
| <file>MASTER=local spark-submit --class Main target/scala-2.10/word_count_2.10-1.0.jar /net/projects/spark-example-data/wiki-cs outdir</file> | <file>MASTER=local spark-submit target/scala-2.10/word_count_2.10-1.0.jar /net/projects/spark-example-data/wiki-cs outdir</file> |
| * start Spark cluster (10 machines, 1GB RAM each) on SGE and run ''word_count'' application: | * start Spark cluster (10 machines, 1GB RAM each) on SGE and run ''word_count'' application: |
| <file>spark-qsub 10 1G spark-submit --class Main target/scala-2.10/word_count_2.10-1.0.jar /net/projects/spark-example-data/wiki-cs outdir</file> | <file>spark-qsub 10 1G spark-submit target/scala-2.10/word_count_2.10-1.0.jar /net/projects/spark-example-data/wiki-cs outdir</file> |
| | |
| Note that the ''-''''-class Main'' arguments are needed only because of [[https://issues.apache.org/jira/browse/SPARK-4298|Spark bug 4298]] and will not be needed when fixed. | |
| |
