Differences
This shows you the differences between two versions of the page.
Both sides previous revision Previous revision Next revision | Previous revision Next revision Both sides next revision | ||
courses:mapreduce-tutorial:step-26 [2012/01/28 15:41] straka |
courses:mapreduce-tutorial:step-26 [2012/01/31 11:28] straka |
||
---|---|---|---|
Line 1: | Line 1: | ||
- | ====== MapReduce Tutorial : Counters and job configuration ====== | + | ====== MapReduce Tutorial : Counters, compression |
- | ===== Counters ===== | ||
- | As in the Perl API, a mapper or a reducer | + | |
+ | ===== Compression ===== | ||
+ | |||
+ | The output files can be compressed | ||
<code java> | <code java> | ||
- | public void map(Text key, Text value, Context context) throws IOException, | + | FileOutputFormat.setCompressOutput(job, true); |
- | | + | |
- | context.getCounter(" | + | |
- | ... | + | |
- | } | + | |
</ | </ | ||
- | The '' | + | |
+ | The default compression format is '' | ||
<code java> | <code java> | ||
- | public void reduce(Text key, Iterable< | + | import org.apache.hadoop.io.compress.*; |
... | ... | ||
- | | + | |
- | | + | |
- | | + | |
- | values.increment(1); | + | |
- | } | + | |
- | } | + | |
</ | </ | ||
+ | |||
+ | Of course, any of these formats is decompressed transparently when the file is being read. | ||
===== Job configuration ===== | ===== Job configuration ===== | ||
Line 36: | Line 34: | ||
Apart from already mentioned [[.: | Apart from already mentioned [[.: | ||
+ | * **'' | ||
+ | |||
+ | |||
+ | ---- | ||
+ | < | ||
+ | <table style=" | ||
+ | <tr> | ||
+ | <td style=" | ||
+ | <td style=" | ||
+ | <td style=" | ||
+ | </tr> | ||
+ | </ | ||
+ | </ |