[ Skip to the content ]

Institute of Formal and Applied Linguistics Wiki

• ÚFAL Wiki Home
• ÚFAL Home
• Sitemap
• RSS
• ÚFAL calendar
• Recent changes
• Login
• Hledat:

[ Back to the navigation ]

Differences

This shows you the differences between two versions of the page.

View differences: Side by SideInline Go

Link to this comparison view

Both sides previous revision Previous revision 2012/02/06 14:52 dusek 2012/02/06 13:26 straka 2012/02/06 13:25 straka 2012/02/06 13:24 straka 2012/02/06 13:24 straka 2012/02/06 13:24 straka 2012/02/06 13:23 straka 2012/02/06 13:11 straka 2012/02/06 13:10 straka 2012/02/06 13:10 straka 2012/02/06 13:10 straka 2012/02/06 13:10 straka 2012/02/06 09:22 straka 2012/02/06 09:01 straka 2012/02/06 08:55 straka 2012/02/06 08:50 straka 2012/02/06 08:41 straka 2012/02/06 08:40 straka 2012/02/06 08:28 straka 2012/02/06 08:26 straka 2012/02/06 08:21 straka 2012/02/06 08:21 straka 2012/02/06 08:20 straka 2012/02/06 08:20 straka 2012/02/06 08:18 straka 2012/02/06 08:13 straka 2012/02/06 08:10 straka 2012/02/06 08:09 straka 2012/02/06 08:03 straka 2012/02/06 08:03 straka 2012/02/06 08:02 straka 2012/02/06 07:59 straka vytvořeno Go Next revision		Previous revision 2012/02/06 14:52 dusek 2012/02/06 13:26 straka 2012/02/06 13:25 straka 2012/02/06 13:24 straka 2012/02/06 13:24 straka 2012/02/06 13:24 straka 2012/02/06 13:23 straka 2012/02/06 13:11 straka 2012/02/06 13:10 straka 2012/02/06 13:10 straka 2012/02/06 13:10 straka 2012/02/06 13:10 straka 2012/02/06 09:22 straka 2012/02/06 09:01 straka 2012/02/06 08:55 straka 2012/02/06 08:50 straka 2012/02/06 08:41 straka 2012/02/06 08:40 straka 2012/02/06 08:28 straka 2012/02/06 08:26 straka 2012/02/06 08:21 straka 2012/02/06 08:21 straka 2012/02/06 08:20 straka 2012/02/06 08:20 straka 2012/02/06 08:18 straka 2012/02/06 08:13 straka 2012/02/06 08:10 straka 2012/02/06 08:09 straka 2012/02/06 08:03 straka 2012/02/06 08:03 straka 2012/02/06 08:02 straka 2012/02/06 07:59 straka vytvořeno Go Next revision Both sides next revision
courses:mapreduce-tutorial:step-31 [2012/02/06 08:40] straka		courses:mapreduce-tutorial:step-31 [2012/02/06 13:10] straka
Line 32:		Line 32:
	It is crucial that all the mappers run simultaneously. This can be achieved using the ''/net/projects/hadoop/bin/compute-splitsize'' script: for given Hadoop input and requested number of mappers, it computes the appropriate splitsize.		It is crucial that all the mappers run simultaneously. This can be achieved using the ''/net/projects/hadoop/bin/compute-splitsize'' script: for given Hadoop input and requested number of mappers, it computes the appropriate splitsize.
-	When the computation finishes, only one of the mappers should print the results, as all of them have the same results. For simplicity, the ''cooperate'' method has ''boolean shouldWrite'' argument, which is set in exactly one mapper.	+	When the computation finishes, only one of the mappers should print the results, as all of them have the same results. For simplicity, the ''cooperate'' method has ''boolean writeResults'' argument, which is set in exactly one mapper.
	===== Example =====		===== Example =====
Line 102:		Line 102:
	You can run the example locally using:		You can run the example locally using:
	wget --no-check-certificate 'https://wiki.ufal.ms.mff.cuni.cz/_export/code/courses:mapreduce-tutorial:step-31?codeblock=0' -O Sum.java		wget --no-check-certificate 'https://wiki.ufal.ms.mff.cuni.cz/_export/code/courses:mapreduce-tutorial:step-31?codeblock=0' -O Sum.java
-	make -f /net/projects/hadoop/java/Makefile Sum.java	+	make -f /net/projects/hadoop/java/Makefile Sum.jar
	rm -rf step-31-out; /net/projects/hadoop/bin/hadoop Sum.jar /net/projects/hadoop/examples/inputs/numbers-small step-31-out		rm -rf step-31-out; /net/projects/hadoop/bin/hadoop Sum.jar /net/projects/hadoop/examples/inputs/numbers-small step-31-out
	less step-31-out/part-*		less step-31-out/part-*
Line 122:		Line 122:
	# NOW VIEW THE FILE		# NOW VIEW THE FILE
	# $EDITOR Statistics.java		# $EDITOR Statistics.java
-	make -f /net/projects/hadoop/java/Makefile Statistics.java	+	make -f /net/projects/hadoop/java/Makefile Statistics.jar
	rm -rf step-31-out; /net/projects/hadoop/bin/hadoop Statistics.jar -c C `/net/projects/hadoop/bin/compute-splitsize /net/projects/hadoop/examples/inputs/numbers-small C` /net/projects/hadoop/examples/inputs/numbers-small step-31-out		rm -rf step-31-out; /net/projects/hadoop/bin/hadoop Statistics.jar -c C `/net/projects/hadoop/bin/compute-splitsize /net/projects/hadoop/examples/inputs/numbers-small C` /net/projects/hadoop/examples/inputs/numbers-small step-31-out
	less step-31-out/part-*		less step-31-out/part-*
Line 141:		Line 141:
	# NOW VIEW THE FILE		# NOW VIEW THE FILE
	# $EDITOR Median.java		# $EDITOR Median.java
-	make -f /net/projects/hadoop/java/Makefile Median.java	+	make -f /net/projects/hadoop/java/Makefile Median.jar
	rm -rf step-31-out; /net/projects/hadoop/bin/hadoop Median.jar -c C `/net/projects/hadoop/bin/compute-splitsize /net/projects/hadoop/examples/inputs/numbers-small C` /net/projects/hadoop/examples/inputs/numbers-small step-31-out		rm -rf step-31-out; /net/projects/hadoop/bin/hadoop Median.jar -c C `/net/projects/hadoop/bin/compute-splitsize /net/projects/hadoop/examples/inputs/numbers-small C` /net/projects/hadoop/examples/inputs/numbers-small step-31-out
	less step-31-out/part-*		less step-31-out/part-*
Line 149:		Line 149:
	===== Exercise 3 =====		===== Exercise 3 =====
-	Implement an AllReduce job on ''/net/projects/hadoop/examples/inputs/numbers-small'', which computes	+	Implement an AllReduce job on ''/net/projects/hadoop/examples/inputs/points-small'', which implements the [[http://en.wikipedia.org/wiki/K-means_clustering#Standard_algorithm|K-means clustering algorithm]]. See [[.:step-15|K-means clustering exercise]] for description of input data.
-	You can download the template {{:courses:mapreduce-tutorial:step-31-exercise3.txt|Median.java}} and execute it using:	+	You can download the template {{:courses:mapreduce-tutorial:step-31-exercise3.txt|KMeans.java}}. This template uses two Hadoop properties:
-	wget --no-check-certificate 'https://wiki.ufal.ms.mff.cuni.cz/_media/courses:mapreduce-tutorial:step-31-exercise3.txt' -O KMeans.java.java	+	* ''clusters.num'' -- number of clusters
		+	* ''clusters.file'' -- file where to read the initial clusters from
		+	You can download and compile it using:
		+	wget --no-check-certificate 'https://wiki.ufal.ms.mff.cuni.cz/_media/courses:mapreduce-tutorial:step-31-exercise3.txt' -O KMeans.java
	# NOW VIEW THE FILE		# NOW VIEW THE FILE
-	# $EDITOR KMeans.java.java	+	# $EDITOR KMeans.java
-	make -f /net/projects/hadoop/java/Makefile KMeans.java.java	+	make -f /net/projects/hadoop/java/Makefile KMeans.java
-	rm -rf step-31-out; /net/projects/hadoop/bin/hadoop KMeans.java.jar -c C `/net/projects/hadoop/bin/compute-splitsize /net/projects/hadoop/examples/inputs/numbers-small C` /net/projects/hadoop/examples/inputs/numbers-small step-31-out	+	You can run it using specified number of machines on the following input data:
-	less step-31-out/part-*	+	* ''/net/projects/hadoop/examples/inputs/points-small'':
		+	<code>M=machines; K=50; INPUT=/net/projects/hadoop/examples/inputs/points-small/points.txt
		+	rm -rf step-31-out; /net/projects/hadoop/bin/hadoop KMeans.jar -Dclusters.num=$K -Dclusters.file=$INPUT [-jt jobtracker | -c $M] `/net/projects/hadoop/bin/compute-splitsize $INPUT $M` $INPUT step-31-out</code>
		+	* ''/net/projects/hadoop/examples/inputs/points-medium'':
		+	<code>M=machines; K=100; INPUT=/net/projects/hadoop/examples/inputs/points-medium/points.txt
		+	rm -rf step-31-out; /net/projects/hadoop/bin/hadoop KMeans.jar -Dclusters.num=$K -Dclusters.file=$INPUT [-jt jobtracker | -c $M] `/net/projects/hadoop/bin/compute-splitsize $INPUT $M` $INPUT step-31-out</code>
		+	* ''/net/projects/hadoop/examples/inputs/points-large'':
		+	<code>M=machines; K=200; INPUT=/net/projects/hadoop/examples/inputs/points-large/points.txt
		+	rm -rf step-31-out; /net/projects/hadoop/bin/hadoop KMeans.jar -Dclusters.num=$K -Dclusters.file=$INPUT [-jt jobtracker | -c $M] `/net/projects/hadoop/bin/compute-splitsize $INPUT $M` $INPUT step-31-out</code>
	Solution: {{:courses:mapreduce-tutorial:step-31-solution3.txt|KMeans.java}}.		Solution: {{:courses:mapreduce-tutorial:step-31-solution3.txt|KMeans.java}}.

---

[ Back to the navigation ] [ Back to the content ]