• Part 1: Monday January 30, 14:00-17:00, lab SU2
• Part 2: Tuesday January 31, 14:00-17:00, lab SU2

• Introduction

Today we will be using the Perl API (there is no need to study it now, the tutorial will explain it).

• Step 1: Setting the environment.

• Step 2: Input and output format, testing data.
• Step 3: Basic mapper.
• Step 4: Counters.
• Step 5: Basic reducer.

• Step 6: Running on cluster.
• Step 7: Dynamic Hadoop cluster for several computations.

From now on, it is best to run MR jobs using a one-machine cluster – create a one-machine cluster using hadoop-cluster for 3h (10800s) and run jobs using -jt cluster_master. Running the scripts locally without any cluster has several disadvantages, most notably having only one reducer per job.

• Step 8: Multiple mappers, reducers and partitioning.
• Step 9: Hadoop properties.
• Step 10: Combiners.
• Step 11: Initialization and cleanup of MR tasks, performance of combiners.
• Step 12: Additional output from mappers and reducers.

Exercises in this section can be made in any order, but it is recommended to try solving all of them. The Perl API reference may come handy.

• Step 13: Sorting.
• Step 14: N-gram language model.
• Step 15: K-means clustering.

Today we will be using the Java API.

• Step 21: Preparing the environment.
• Step 22: Optional – Setting Eclipse.

• Step 23: Predefined formats and types.
• Step 24: Mappers, running Java Hadoop jobs, counters.
• Step 25: Reducers, combiners and partitioners.
• Step 26: Compression and job configuration.

• Step 27: Custom data types.
• Step 28: Running multiple Hadoop jobs in one class.
• Step 29: Custom sorting and grouping comparators.
• Step 30: Custom input formats.

• Step 31: Implementing iterative MapReduce jobs faster using All-Reduce.

• Further information