[ Skip to the content ]

Institute of Formal and Applied Linguistics Wiki

[ Back to the navigation ]

You are here: start » spark » using-scala

Differences

This shows you the differences between two versions of the page.

Link to this comparison view

Both sides previous revision Previous revision
Next revision 		Previous revision
Next revision Both sides next revision
spark:using-scala [2015/07/27 18:46]
straka 		spark:using-scala [2017/10/16 21:24]
ufal [Using Scala]
Line 42: Line 42:
   - replace the ''spark-template'' by your project name in the first line (i.e., ''name := "my-best-project"'')   - replace the ''spark-template'' by your project name in the first line (i.e., ''name := "my-best-project"'')
   - run ''sbt package'' to create JAR (note that first run of ''sbt'' will take several minutes)   - run ''sbt package'' to create JAR (note that first run of ''sbt'' will take several minutes)
-The resulting JAR can be found in ''target/scala-2.10'' subdirectory, named after your project.+The resulting JAR can be found in ''target/scala-2.11'' subdirectory, named after your project.
  
 ==== Usage Examples ==== ==== Usage Examples ====
Line 72: Line 72:
 version := "1.0" version := "1.0"
  
-scalaVersion := "2.10.4"+scalaVersion := "2.11.8"
  
-libraryDependencies += "org.apache.spark" %% "spark-core" % "1.4.1"+libraryDependencies += "org.apache.spark" %% "spark-core" % "2.1.1"
 </file> </file>
  
Line 81: Line 81:
  
   * run ''word_count'' application inside existing Spark cluster (i.e., inside ''spark-qsub'' or ''spark-qrsh''), or start local Spark cluster using as many threads as there are cores if there is none:   * run ''word_count'' application inside existing Spark cluster (i.e., inside ''spark-qsub'' or ''spark-qrsh''), or start local Spark cluster using as many threads as there are cores if there is none:
-  <file>spark-submit target/scala-2.10/word_count_2.10-1.0.jar /net/projects/spark-example-data/wiki-cs outdir</file>+  <file>spark-submit target/scala-2.11/word_count_2.11-1.0.jar /net/projects/spark-example-data/wiki-cs outdir</file>
   * run ''word_count'' application with local Spark cluster using one thread:   * run ''word_count'' application with local Spark cluster using one thread:
-  <file>MASTER=local spark-submit target/scala-2.10/word_count_2.10-1.0.jar /net/projects/spark-example-data/wiki-cs outdir</file>+  <file>MASTER=local spark-submit target/scala-2.11/word_count_2.11-1.0.jar /net/projects/spark-example-data/wiki-cs outdir</file>
   * start Spark cluster (10 machines, 1GB RAM each) on SGE and run ''word_count'' application:   * start Spark cluster (10 machines, 1GB RAM each) on SGE and run ''word_count'' application:
-  <file>spark-qsub 10 1G spark-submit target/scala-2.10/word_count_2.10-1.0.jar /net/projects/spark-example-data/wiki-cs outdir</file>+  <file>spark-qsub 10 1G spark-submit target/scala-2.11/word_count_2.11-1.0.jar /net/projects/spark-example-data/wiki-cs outdir</file>
  

[ Back to the navigation ] [ Back to the content ]