Both sides previous revision
Previous revision
Next revision
|
Previous revision
Next revision
Both sides next revision
|
spark:using-scala [2014/11/10 17:40] straka |
spark:using-scala [2014/11/11 09:31] straka |
</file> | </file> |
| |
* run interactive shell inside ''spark-qrsh'', or start local Spark cluster using as many threads as there are cores: | * run interactive shell using existing Spark cluster (i.e., inside ''spark-qrsh''), or start local Spark cluster using as many threads as there are cores if there is none: |
<file>spark-shell</file> | <file>spark-shell</file> |
* run interactive shell with local Spark cluster using one thread: | * run interactive shell with local Spark cluster using one thread: |
- copy ''/net/projects/spark/sbt/spark-template.sbt'' to your project directory and rename it to your project name (i.e., ''my-best-project.sbt'') | - copy ''/net/projects/spark/sbt/spark-template.sbt'' to your project directory and rename it to your project name (i.e., ''my-best-project.sbt'') |
- replace the ''spark-template'' by your project name in the first line (i.e., ''name := "my-best-project"'') | - replace the ''spark-template'' by your project name in the first line (i.e., ''name := "my-best-project"'') |
- run ''sbt package'' (note that first run of ''sbt'' will take several minutes) | - run ''sbt package'' to create JAR (note that first run of ''sbt'' will take several minutes) |
The resulting JAR can be found in ''target/scala-2.10'' subdirectory, named after your project. | The resulting JAR can be found in ''target/scala-2.10'' subdirectory, named after your project. |
| |
<file>sbt package</file> | <file>sbt package</file> |
| |
* run ''word_count'' application inside ''spark-qsub'', ''spark-qrsh'', or start local Spark cluster using as many threads as there are cores: | * run ''word_count'' application inside existing Spark cluster (i.e., inside ''spark-qsub'' or ''spark-qrsh''), or start local Spark cluster using as many threads as there are cores if there is none: |
<file>spark-submit --class Main target/scala-2.10/word_count_2.10-1.0.jar input output</file> | <file>spark-submit --class Main target/scala-2.10/word_count_2.10-1.0.jar input output</file> |
* run ''word_count'' application with local Spark cluster using one thread: | * run ''word_count'' application with local Spark cluster using one thread: |