Differences
This shows you the differences between two versions of the page.
Both sides previous revision Previous revision Next revision | Previous revision Next revision Both sides next revision | ||
spark:spark-introduction [2014/11/11 08:57] straka |
spark:spark-introduction [2014/11/11 09:06] straka |
||
---|---|---|---|
Line 5: | Line 5: | ||
===== Running Spark Shell in Python ===== | ===== Running Spark Shell in Python ===== | ||
- | To run interactive Python shell in local Spark mode, run (on your local workstation or on cluster) | + | To run interactive Python shell in local Spark mode, run (on your local workstation or on cluster |
IPYTHON=1 pyspark | IPYTHON=1 pyspark | ||
- | The IPYTHON=1 parameter instructs Spark to use '' | + | The IPYTHON=1 parameter instructs Spark to use '' |
After a local Spark executor is started, the Python shell starts. Severel lines above | After a local Spark executor is started, the Python shell starts. Severel lines above | ||
Line 71: | Line 71: | ||
lines = sc.textFile("/ | lines = sc.textFile("/ | ||
- | data = lines.map(lambda line: np.array([float(x) for x in line.split()])).cache() | + | data = lines.map(lambda line: map(float, line.split())).cache() |
K = 50 | K = 50 |