Differences
This shows you the differences between two versions of the page.
Both sides previous revision Previous revision Next revision | Previous revision Next revision Both sides next revision | ||
spark:spark-introduction [2014/11/11 08:55] straka |
spark:spark-introduction [2014/11/11 09:06] straka |
||
---|---|---|---|
Line 5: | Line 5: | ||
===== Running Spark Shell in Python ===== | ===== Running Spark Shell in Python ===== | ||
- | To run interactive Python shell in local Spark mode, run (on your local workstation or on cluster) | + | To run interactive Python shell in local Spark mode, run (on your local workstation or on cluster |
IPYTHON=1 pyspark | IPYTHON=1 pyspark | ||
- | The IPYTHON=1 parameter instructs Spark to use '' | + | The IPYTHON=1 parameter instructs Spark to use '' |
After a local Spark executor is started, the Python shell starts. Severel lines above | After a local Spark executor is started, the Python shell starts. Severel lines above | ||
Line 63: | Line 63: | ||
===== K-Means Example ===== | ===== K-Means Example ===== | ||
- | To show an example | + | An example |
<file python> | <file python> | ||
import numpy as np | import numpy as np | ||
Line 71: | Line 71: | ||
lines = sc.textFile("/ | lines = sc.textFile("/ | ||
- | data = lines.map(lambda line: np.array([float(x) for x in line.split()])).cache() | + | data = lines.map(lambda line: map(float, line.split())).cache() |
K = 50 | K = 50 |