MapReduce Tutorial

• Part 1: Monday January 30, 14:00-17:00, lab SU2
• Part 2: Tuesday January 31, 14:00-17:00, lab SU2

• Introduction

• Hadoop job overview
• Managing a Hadoop cluster
• Running jobs
• Perl API, Java API
• Making your job configurable
• If things go wrong

Today we will be using the Perl API (there is no need to study it now, the tutorial will explain it).

• Step 1: Setting the environment.

• Step 2: Input and output format, testing data.
• Step 3: Basic mapper.
• Step 4: Counters.
• Step 5: Basic reducer.

• Step 6: Running on cluster.
• Step 7: Dynamic Hadoop cluster for several computations.

• Step 8: Multiple mappers, reducers and partitioning.
• Step 9: Hadoop properties.
• Step 10: Combiners.
• Step 11: Initialization and cleanup of MR tasks, performance of combiners.
• Step 12: Additional output from mappers and reducers.

Exercises in this section can be made in any order, but it is recommended to try solving all of them. The Perl API reference may come handy.

• Step 13: Sorting.
• Step 14: N-gram language model.
• Step 15: K-means clustering.

• Step 16: Implementing iterative MapReduce jobs faster using All-Reduce.

Today we will be using the Java API.

• Step 21: Preparing the environment.
• Step 22: Optional – Setting Eclipse.

• Step 23: Predefined formats and types.
• Step 24: Mappers, running Java Hadoop jobs, counters.
• Step 25: Reducers, combiners and partitioners.
• Step 26: Compression and job configuration.
• Step 27: Running multiple Hadoop jobs in one source file.

• Step 28: Custom data types.
• Step 29: Custom sorting and grouping comparators.
• Step 30: Custom input formats.

• Step 31: Implementing iterative MapReduce jobs faster using All-Reduce.

• Further information