Both sides previous revision
Previous revision
Next revision
|
Previous revision
Next revision
Both sides next revision
|
courses:mapreduce-tutorial:step-26 [2012/01/28 20:22] straka |
courses:mapreduce-tutorial:step-26 [2012/01/30 15:41] majlis |
</code> | </code> |
| |
| The default compression format is ''deflate'' -- raw Zlib compression. Several other compression formats can be selected: |
| <code java> |
| import org.apache.hadoop.io.compress.*; |
| |
| ... |
| FileOutputFormat.setOutputCompressorClass(GzipCodec.class); //.gz |
| FileOutputFormat.setOutputCompressorClass(BZip2Codec.class); //.bz2 |
| </code> |
| |
| Of course, any of these formats is decompressed transparently when the file is being read. |
| |
===== Job configuration ===== | ===== Job configuration ===== |
Apart from already mentioned [[.:step-9#a-brief-list-of-hadoop-options|brief list of Hadoop properties]], there is one important Java-specific property: | Apart from already mentioned [[.:step-9#a-brief-list-of-hadoop-options|brief list of Hadoop properties]], there is one important Java-specific property: |
* **''mapred.child.java.opts''** with default value **''-Xmx200m''**. This property sets the Java options used for every task attempt (mapper, reducer, combiner, partitioner). The default value ''-Xmx200m'' specifies the maximum size of memory allocation pool. If your mappers and reducers need 1GB memory, use ''-Xmx1024m''. Other Java options can be found in ''man java''. | * **''mapred.child.java.opts''** with default value **''-Xmx200m''**. This property sets the Java options used for every task attempt (mapper, reducer, combiner, partitioner). The default value ''-Xmx200m'' specifies the maximum size of memory allocation pool. If your mappers and reducers need 1GB memory, use ''-Xmx1024m''. Other Java options can be found in ''man java''. |
| |
| |
| ---- |
| |
| <html> |
| <table style="width:100%"> |
| <tr> |
| <td style="text-align:left; width: 33%; "></html>[[step-25|Step 25]]: Reducers, combiners and partitioners.<html></td> |
| <td style="text-align:center; width: 33%; "></html>[[.|Overview]]<html></td> |
| <td style="text-align:right; width: 33%; "></html>[[step-27|Step 27]]: Custom data types.<html></td> |
| </tr> |
| </table> |
| </html> |