Differences
This shows you the differences between two versions of the page.
Both sides previous revision Previous revision Next revision | Previous revision | ||
courses:mapreduce-tutorial:step-26 [2012/01/28 20:22] straka |
courses:mapreduce-tutorial:step-26 [2012/01/31 14:41] (current) straka |
||
---|---|---|---|
Line 1: | Line 1: | ||
- | ====== MapReduce Tutorial : Counters, compression | + | ====== MapReduce Tutorial : Compression |
- | ===== Counters ===== | ||
- | As in the Perl API, a mapper or a reducer can increment various counters by using '' | ||
- | <code java> | ||
- | public void map(Text key, Text value, Context context) throws IOException, | ||
- | ... | ||
- | context.getCounter(" | ||
- | ... | ||
- | } | ||
- | </ | ||
- | The '' | ||
- | <code java> | ||
- | public void reduce(Text key, Iterable< | ||
- | ... | ||
- | Counter values = context.getCounter(" | ||
- | for (IntWritable value : values) { | ||
- | ... | ||
- | values.increment(1); | ||
- | } | ||
- | } | ||
- | </ | ||
===== Compression ===== | ===== Compression ===== | ||
Line 30: | Line 10: | ||
</ | </ | ||
| | ||
+ | The default compression format is '' | ||
+ | <code java> | ||
+ | import org.apache.hadoop.io.compress.*; | ||
+ | |||
+ | ... | ||
+ | FileOutputFormat.setOutputCompressorClass(GzipCodec.class); | ||
+ | FileOutputFormat.setOutputCompressorClass(BZip2Codec.class); | ||
+ | </ | ||
+ | |||
+ | Of course, any of these formats is decompressed transparently when the file is being read. | ||
===== Job configuration ===== | ===== Job configuration ===== | ||
Line 45: | Line 35: | ||
Apart from already mentioned [[.: | Apart from already mentioned [[.: | ||
* **'' | * **'' | ||
+ | |||
+ | |||
+ | ---- | ||
+ | |||
+ | < | ||
+ | <table style=" | ||
+ | <tr> | ||
+ | <td style=" | ||
+ | <td style=" | ||
+ | <td style=" | ||
+ | </tr> | ||
+ | </ | ||
+ | </ |