Differences
This shows you the differences between two versions of the page.
Both sides previous revision Previous revision | Next revision Both sides next revision | ||
spark:spark-introduction [2022/12/14 12:34] straka [Word Count Example] |
spark:spark-introduction [2022/12/14 12:36] straka [Word Count Example] |
||
---|---|---|---|
Line 43: | Line 43: | ||
The output of ' | The output of ' | ||
- | Note that 'map' and ' | + | Note that '' |
The Scala versions is quite similar: | The Scala versions is quite similar: | ||
<file scala> | <file scala> | ||
- | val wiki = sc.textFile("/ | + | val wiki = sc.textFile("/ |
val words = wiki.flatMap(line => line.split(" | val words = wiki.flatMap(line => line.split(" | ||
- | val counts = words.map(word => (word, | + | val counts = words.map(word => (word, 1)).reduceByKey((c1, |
val sorted = counts.sortBy({case (word, count) => count}, ascending=false) | val sorted = counts.sortBy({case (word, count) => count}, ascending=false) | ||
sorted.saveAsTextFile(" | sorted.saveAsTextFile(" | ||
// Alternatively without variables and using placeholders in lambda parameters: | // Alternatively without variables and using placeholders in lambda parameters: | ||
- | (sc.textFile("/ | + | (sc.textFile("/ |
| | ||
| |