[ Skip to the content ]

Institute of Formal and Applied Linguistics Wiki

[ Back to the navigation ]

You are here: start » courses » mapreduce-tutorial » step-6

Differences

This shows you the differences between two versions of the page.

Link to this comparison view

Both sides previous revision Previous revision
Next revision Both sides next revision
courses:mapreduce-tutorial:step-6 [2012/01/25 00:03]
straka 		courses:mapreduce-tutorial:step-6 [2012/01/25 00:11]
straka
Line 8: Line 8:
  
 When a distributed MR computations is executed, it submits a job to SGE cluster, with the name of the Perl script. The SGE cluster creates 3 files in current directory: When a distributed MR computations is executed, it submits a job to SGE cluster, with the name of the Perl script. The SGE cluster creates 3 files in current directory:
-  * ''script.pl.c$SGE_JOBID'' -- high-level status of computation. First line contains the name of cluster master.+  * ''script.pl.c$SGE_JOBID'' -- high-level status of computation
   * ''script.pl.o$SGE_JOBID'' -- contains stdout and stderr of the MR job   * ''script.pl.o$SGE_JOBID'' -- contains stdout and stderr of the MR job
   * ''script.pl.po$SGE_JOBID'' -- contains stdout and stderr of the MR cluster   * ''script.pl.po$SGE_JOBID'' -- contains stdout and stderr of the MR cluster
 When the computation ends and is waiting because of the ''-w'' parameter, removing the file ''script.pl.c$SGE_JOBID'' stops the cluster. The cluster can be also stopped by removing its SGE job. When the computation ends and is waiting because of the ''-w'' parameter, removing the file ''script.pl.c$SGE_JOBID'' stops the cluster. The cluster can be also stopped by removing its SGE job.
 +
 +===== Web interface =====
 +
 +The cluster master provides a web interface on port 50030 (the port may change in the future). The cluster master address can be found at the first line of ''script.pl.c$SGE_JOBID'', or using ''qstat -j $SGE_JOBID'' (context variable ''hdfs_jobtracker_admin'').
 +
 +The web interface provides a lot of useful informations:
 +  * running, failed and successfully completed jobs
 +  * for running job, current progress and counters of the whole job and also of each mapper and reducer is available
 +  * for any job, the counters and outputs of all mappers and reducers are stored
 +

[ Back to the navigation ] [ Back to the content ]