This is an old revision of the document!
Table of Contents
MapReduce tutorial : Input and output format, testing data.
The MapReduce framework is frequently using (key, value) pairs. These
pairs can be read from a file and written to a file and there are several formats available.
Input formats
TextInputFormat– values are lines of plain text files, keys are the positions of their first character in the fileKeyValueTextInputFormat– every line of plain text file is split using first TAB character, forming key and value. If there is no TAB character, value is emptySequenceFileInputFormat– binary format
The input format can be compressed and will be decompressed transparently by the MR framework.
Output formats
TextOutputFormat– (key, value) pair is printed on one line separated by a TAB character. If key or value is empty, no TAB character is used.SequenceFileOutputFormat– binary format
The output format can be compressed on demand.
Input data
Testing data are available in several formats and sizes:
/home/straka/wiki/cs-seq– compressed SequenceFile of Czech Wikipedia, 85MB./home/straka/wiki/cs-seq-medium– compressed SequenceFile of Czech Wikipedia, 8MB./home/straka/wiki/cs-seq-small– compressed SequenceFile of Czech Wikipedia, 35kB./home/straka/wiki/cs-text– uncompressed plain text files of Czech Wikipedia, 200MB./home/straka/wiki/cs-text-medium– uncompressed plain text files of Czech Wikipedia, 16MB./home/straka/wiki/cs-text-small– uncompressed plain text files of Czech Wikipedia, 70kB./home/straka/wiki/en-seq– compressed SequenceFile of English Wikipedia, 1.9GB.