Both sides previous revision
Previous revision
Next revision
|
Previous revision
Next revision
Both sides next revision
|
spark [2022/12/14 12:26] straka [Basic Information] |
spark [2022/12/14 12:43] straka [Using Spark in UFAL Environment] |
* Official [[http://spark.apache.org/docs/latest/programming-guide.html|Spark Programming Guide]] | * Official [[http://spark.apache.org/docs/latest/programming-guide.html|Spark Programming Guide]] |
* Official [[http://spark.apache.org/docs/latest/mllib-guide.html|MLlib Programming Guide]] (Spark’s scalable machine learning library consisting of common learning algorithms and utilities, including classification, regression, clustering, collaborative filtering, dimensionality reduction, as well as underlying optimization primitives) | * Official [[http://spark.apache.org/docs/latest/mllib-guide.html|MLlib Programming Guide]] (Spark’s scalable machine learning library consisting of common learning algorithms and utilities, including classification, regression, clustering, collaborative filtering, dimensionality reduction, as well as underlying optimization primitives) |
* Official [[http://spark.apache.org/docs/latest/api/python/index.html|Python API Reference]]/[[http://spark.apache.org/docs/latest/api/scala/index.html#org.apache.spark.package|Scala API Reference]] | * Official [[http://spark.apache.org/docs/latest/api/python/index.html|Python API Reference]]/[[https://spark.apache.org/docs/latest/api/scala/org/apache/spark/index.html|Scala API Reference]] |
| |
===== Using Spark in UFAL Environment ===== | ===== Using Spark in UFAL Environment ===== |
| |
Latest supported version of Spark is available in ''/net/projects/spark''. To use it, add | Latest supported version of Spark is available in ''/net/projects/spark''. To use it, add |
export PATH="/net/projects/spark/bin:/net/projects/spark/sge:$PATH" | export PATH="/net/projects/spark/bin:/net/projects/spark/slurm:$PATH" |
to your ''.bashrc'' (or to your favourite shell config file). If you want to use Scala and do not have ''sbt'' already installed (or you do not know what ''sbt'' is), add also | to your ''.bashrc'' (or to your favourite shell config file). If you want to use Scala and do not have ''sbt'' already installed (or you do not know what ''sbt'' is), add also |
export PATH="/net/projects/spark/sbt/bin:$PATH" | export PATH="/net/projects/spark/sbt/bin:$PATH" |