[ Skip to the content ]

Institute of Formal and Applied Linguistics Wiki

[ Back to the navigation ]

You are here: start » spark » using-scala

Differences

This shows you the differences between two versions of the page.

Link to this comparison view

Both sides previous revision Previous revision
spark:using-scala [2026/09/16 12:57]
straka 		spark:using-scala [2026/09/16 13:00] (current)
straka
Line 42: Line 42:
   - replace the ''spark-template'' by your project name in the first line (i.e., ''name := "my-best-project"'')   - replace the ''spark-template'' by your project name in the first line (i.e., ''name := "my-best-project"'')
   - run ''sbt --server package'' to create JAR   - run ''sbt --server package'' to create JAR
-The resulting JAR can be found in ''target'' subdirectory, named after your project.+The resulting JAR can be found in ''target/out'' subdirectory, named after your project.
  
 ==== Usage Examples ==== ==== Usage Examples ====
Line 73: Line 73:
  
 scalaVersion := "2.13.18" scalaVersion := "2.13.18"
-  + 
-libraryDependencies += "org.apache.spark" %% "spark-core" % "4.2.0"+libraryDependencies += "org.apache.spark" %% "spark-core" % "4.2.0" % "provided" 
 + 
 +Compile / packageBin / artifactPath := { 
 +  val jarFile = baseDirectory.value / "target" / "out" / s"${name.value}-${version.value}.jar" 
 +  fileConverter.value.toVirtualFile(jarFile.toPath) 
 +}
 </file> </file>
  
Line 81: Line 86:
  
   * run ''word_count'' application inside existing Spark cluster (i.e., inside ''spark-sbatch'' or ''spark-srun''), or start local Spark cluster using as many threads as there are cores if there is none:   * run ''word_count'' application inside existing Spark cluster (i.e., inside ''spark-sbatch'' or ''spark-srun''), or start local Spark cluster using as many threads as there are cores if there is none:
-  <file>spark-submit target/scala-2.13/word_count_2.13-1.0.jar /net/projects/spark-example-data/wiki-cs outdir</file>+  <file>spark-submit target/out/word_count-1.0.jar /net/projects/spark-example-data/wiki-cs outdir</file>
   * run ''word_count'' application with local Spark cluster using one thread:   * run ''word_count'' application with local Spark cluster using one thread:
-  <file>MASTER=local spark-submit target/scala-2.13/word_count_2.13-1.0.jar /net/projects/spark-example-data/wiki-cs outdir</file>+  <file>MASTER=local spark-submit target/out/word_count-1.0.jar /net/projects/spark-example-data/wiki-cs outdir</file>
   * start Spark cluster (10 machines, 2GB RAM each) on Slurm and run ''word_count'' application:   * start Spark cluster (10 machines, 2GB RAM each) on Slurm and run ''word_count'' application:
-  <file>spark-sbatch 10 2G spark-submit target/scala-2.13/word_count_2.13-1.0.jar /net/projects/spark-example-data/wiki-cs outdir</file>+  <file>spark-sbatch 10 2G spark-submit target/out/word_count-1.0.jar /net/projects/spark-example-data/wiki-cs outdir</file>
  

[ Back to the navigation ] [ Back to the content ]