Differences

This shows you the differences between two versions of the page.

--- user:zeman:joshua [2009/07/13 14:47]
zeman Odkaz na dokumentaci Chrise Callisona-Burche.
+++ user:zeman:joshua [2010/08/31 15:55] (current)
zeman Překlep.
@@ Line 95: / Line 95: @@
   * Tokenizovaný a segmentovaný text v cílovém jazyce (hi).
   * Párování vyrobíme [[Giza++|Gizou++]]. Správný soubor s párováním má stejný počet řádků jako zdrojový a cílový text (co řádek, to věta), ale místo slov má na každém řádku posloupnost dvojic čísel (např. "2-0 2-1 2-2 2-3 1-4"). Čísla vyjadřují, kolikáté slovo zdrojové věty je spárované s kolikátým slovem cílové věty. Ten správný soubor s takovýmto obsahem z Gizy++ obvykle vypadne pod jménem ''model/aligned.grow-diag-final-and'' nebo nějakým podobným.
+==== Zastaralý návod ====
+**Pozor, následující poznámky pocházejí z&nbsp;doby, kdy jsem se učil pracovat s&nbsp;první verzí Joshuy, a jsou zastaralé! Níže je kopie mailu od Lana Schwartze, kde je vysvětlen postup pro Joshuu 1.3.**
 A takhle pustíme Joshuu, aby z trénovacích dat extrahoval gramatiku. Joshua z nějakého důvodu vyžaduje také testovací soubor se zdrojovým jazykem. Soudě podle příkladu, který dodali, stačí zkopírovat první větu ze zdrojových trénovacích dat. Gramatiku je pak ještě třeba seřadit, vyházet duplicitní pravidla a zagzipovat.
@@ Line 142: / Line 146: @@
     --output=model/en-hi.grammar.unsorted \
     --maxPhraseLength=5</code>
+==== Nový návod pro Joshuu 1.3 ====
+Následuje výtah z&nbsp;mailu od Lana Schwartze, který vysvětluje, jak zadávat parametry při extrakci gramatiky s&nbsp;Joshuou 1.3. V&nbsp;dokumentaci u Joshuy jsem to nenašel.
+The recommended way to extract a grammar is to configure an ant XML file for ExtractRules. All available parameters can be configured using that technique. The main method is meant now to just be a simple version for use if you don't need any custom configuration.
+The current version of ExtractRules and its parameters are documented in my and Chris's paper at the most recent MT Marathon:
+http://www.mtmarathon2010.info/web/Program_files/art-schwartz-callison-burch.pdf
+The ant file should look something like this:
+extract.xml:
+<code xml><project name="JoshuaTasks">
+	<!-- Define the path to Joshua class files -->
+	<property name="classes.dir"
+		value="/path/to/joshua/bin"/>
+	<!-- Define the ant task to compile a corpus into binary memory-mappable files -->
+	<taskdef name="compileJosh"
+		classname="joshua.corpus.suffix_array.Compile"
+		classpath="${classes.dir}"/>
+	<!-- Define the ant task to extract rules -->
+	<taskdef name="extractRules"
+		classname="joshua.prefix_tree.ExtractRules"
+		classpath="${classes.dir}"/>
+	<!-- Declare a target to compile a corpus -->
+	<target name="compile_de-en"
+                      description="Compile josh dir for German-English">
+		<compileJosh
+			sourceCorpus="/path/to/train.de"
+			targetCorpus="/path/to/train.en"
+			alignments="/path/to/train.de-en.aln"
+			outputDir="/path/to/output/de-en.josh"
+		/>
+	</target>
+	<!-- Declare a target to extract a grammar -->
+	<target name="extract_de-en"
+                      description="Extract grammar for German-English">
+		<extractRules
+			joshDir="/path/to/output/de-en.josh"
+			outputFile="/path/to/output/de-en.grammar"
+			testFile="/path/to/test.de"
+		/>
+	</target>
+	<!-- Declare a target to extract a grammar with other parameters-->
+	<target name="extract_de-en-custom"
+                      description="Extract grammar for German-English">
+		<extractRules
+			joshDir="/path/to/output/de-en.josh"
+			outputFile="/path/to/output/de-en.grammar"
+			testFile="/path/to/test.de"
+			maxPhraseSpan="5"
+			maxPhraseLength="5"
+			requireTightSpans="true"
+			edgeXViolates="false"
+			sentenceInitialX="false"
+			sentenceFinalX="false"
+			ruleSampleSize="200"
+			maxNonterminals="2"
+		/>
+	</target>
+</project></code>
+You can call this, with any of the targets that you define in extract.xml, using ant:
+<code bash># Compile the corpus
+ant -f extract.xml compile_de-en
+# Extract rules using defaults
+ant -f extract.xml extract_de-en
+# Extract rules using custom settings
+ant -f extract.xml extract_de-en-custom</code>
 ===== Decoding =====
@@ Line 305: / Line 393: @@
     $HINDI/mert/zmert-config.txt \
     > $HINDI/mert/zmert.out</code>
+===== Troubleshooter =====
+==== Grammar extraction: Negative array size ====
+If you encounter this exception during corpus binarization or (in older releases of Joshua) during grammar extraction, check your alignment file whether it matches your source and target corpus. Did you switch translation direction accidentially? The alignment file must have the same number of lines as your source and target corpus, one line per sentence (segment) pair. The "tokens" on each line are pairs of numbers, such as "0-0 1-2 2-2 3-5". The first number in each pair is the index to the source sentence (first token has index 0) and the second number is index to the target sentence. By switching the source and the target, you are likely to cause some indices to point out of the sentence, and you are in trouble.
+Another source of this error could be sentences with 100 or more words. This is not a strict limit, often I was able to extract grammars for corpora unchecked for such sentences, but according to Lance Schwartz, long sentences can cause problems. (For me, filtering out such sentences helped with es-en WMT08 training data.) And after all, they are suspicious anyway, and their contribution to the learnt model is doubtful.
+==== ZMERT: corrupted temp file ====
+Hi all,
+does the following ZMERT exception look familiar to anyone? My only idea was that the nbest output from the decoder is corrupted somehow. However, I cannot find anything strange in it, such as sequence of more then three "|||" etc.
+Thanks,
+Dan
+zmert.out:
+-----
+<code>tmpDirPrefix: /ha/work/people/zeman/wmt/experiments/obo-max/mert/ZMERT.
+Processed the following args array:
+ -dir /ha/work/people/zeman/wmt/experiments/obo-max/mert -s src.txt -r ref.txt -rps 1 -p params.txt -m BLEU 4 closest -maxIt 5 -ipi 20 -cmd ./decoder.pl -decOut nbest.txt -dcfg decoder-config.txt -N 300 -v 1 -seed 12341234
+----------------------------------------------------
+Initializing...
+----------------------------------------------------
+Random number generator initialized using seed: 12341234
+Number of sentences: 2051
+Number of documents: 1
+Optimizing BLEU
+docSubsetInfo: {0, 1, 1, 1, 1, 0, 0}
+Number of features: 5
+Feature names: {"lm","phrasemodel pt 0","phrasemodel pt 1","phrasemodel pt 2","wordpenalty"}
+c    Default value    Optimizable?    Crit. val. range    Rand. val. range
+     1.0000         Yes         [0.1,Infinity]         [0.5,1.5]
+     1.0669         Yes         [-Infinity,Infinity]         [-1.0,1.0]
+     0.7522         Yes         [-Infinity,Infinity]         [-1.0,1.0]
+     0.5898         Yes         [-Infinity,Infinity]         [-1.0,1.0]
+     -2.8448         Yes         [-Infinity,Infinity]         [-5.0,0.0]
+Weight vector normalization method: weights will be scaled so that the "lm" weight has an absolute value of 1.0.
+----------------------------------------------------
+----------------------------------------------------
+Z-MERT run started @ Sat Mar 06 23:52:57 CET 2010
+----------------------------------------------------
+Initial lambda[]: {1.0, 1.066893, 0.752247, 0.589793, -2.844814}
+--- Starting Z-MERT iteration #1 @ Sat Mar 06 23:52:57 CET 2010 ---
+Decoding using initial weight vector {1.0, 1.066893, 0.752247, 0.589793, -2.844814}
+Running external decoder...
+...finished decoding @ Sun Mar 07 00:02:33 CET 2010
+Reading candidate translations from iterations 1-1
+(and computing BLEU sufficient statistics for previously unseen candidates)
+ Progress:
+Exception in thread "main" java.lang.NumberFormatException: For input string: "||||||"
+   at java.lang.NumberFormatException.forInputString(NumberFormatException.java:48)
+   at java.lang.Integer.parseInt(Integer.java:449)
+   at java.lang.Integer.parseInt(Integer.java:499)
+   at joshua.zmert.MertCore.run_single_iteration(MertCore.java:1071)
+   at joshua.zmert.MertCore.main(MertCore.java:3129)
+Z-MERT exiting prematurely (MertCore returned 1)...</code>
+-----
+Omar's response:
+Hi Dan,
+The "||||||" sequence is in a temp file, not the decoder's output.  If
+if there are any *temp* (or *tmp*) files in the folder from earlier
+runs, make sure you delete them first, then try launching Z-MERT
+again.  Such files are left over from runs that crash.  Z-MERT does
+not delete them because they can be used to restart Z-MERT from the
+point where it crashed.  But that assumes the crash is due to power
+loss or an interrupted job, etc.  In your case, I think what happened
+is that a prior run crashed because of an external problem in the
+setup itself, which you fixed and tried to restart Z-MERT.  For that
+reason, Z-MERT should not be using those temp files in the first
+place, but when it sees them there, it assumes it can use them because
+the user did not delete them.
+Let me know if that's not the case.
+O.Z.

[ Back to the navigation ] [ Back to the content ]

Institute of Formal and Applied Linguistics Wiki

Differences