In order to use Spark in Python, environment has to bee set up according to Using Spark in UFAL Environment.

Interactive shell can be started using:

pyspark

Better interactive shell with code completion using ipython (installed everywhere on cluster; ask our IT if you want to have it installed on your workstations too) can be started using:

IPYTHON=1 pyspark

As described in Running Spark on Single Machine or on Cluster, environmental variable MASTER specifies which Spark master to use (or whether to start a local one).

• run interactive shell with local Spark cluster using as many threads as there are cores:

IPYTHON=1 pyspark #if MASTER not defined
MASTER="local[*]" IPYTHON=1 pyspark #if MASTER is defined differently and must be overwritten

• run interactive shell with local Spark cluster using one thread:

MASTER=local IPYTHON=1 pyspark

• start Spark cluster (10 machines, 1GB RAM each) on SGE and run interactive shell:

IPYTHON=1 spark-qrsh 10 1G pyspark

Note that IPYTHON variable can be left out or specified in .bashrc (or similar).