[ Skip to the content ]

Institute of Formal and Applied Linguistics Wiki

[ Back to the navigation ]

You are here: start » spark » spark-introduction

This is an old revision of the document!

Table of Contents

Spark Introduction

Spark Introduction in Python

To run interactive Python shell in local Spark mode, run (on your local workstation or on cluster) 

IPYSPARK=1 pyspark

The IPYSPARK=1 parameter instructs Spark to use ipython instead of python (the ipython is an enhanced interactive shell than Python). If you do not want ipython or you do not have it installed (it is installed everywhere on the cluster, but maybe not on your local workstations – ask Milan if you want it), leave out the IPYSPARK=1.

After a local Spark executor is started, the Python shell starts. 

14/10/03 10:54:35 INFO SparkUI: Started SparkUI at http://tauri4.ufal.hide.ms.mff.cuni.cz:4040

Spark Introduction in Scala

[ Back to the navigation ] [ Back to the content ]