Institute of Formal and Applied Linguistics Wiki

Note: I am going to move around some stuff, especially that in my home folder.

• $PARSINGROOT - working copy of the parsers and related scripts. See Parsing on how to create your own.
• /fs/clip-corpora/ptb/processed - Penn Treebank (referred to as $PTB)
• /fs/clip-corpora/north_american_news - North American News Text Corpus, including everything I made of it

• Small data - treebank.
• Big data - raw. (But for the language adaptation task we must be able to tag it using the same tagset as the small data.)
• A parser able to do N-best parsing. Charniak's or Stanford. We must be able to retrain the parser.
• A reranker (Johnson's). We don't need to retrain it as far as our small data is the Penn Treebank.

We have the parser and reranker (both together = reranking parser), both trained (probably) on Penn Treebank Wall Street Journal sections 2-21 (training done already at Brown).

• Charniak parser, parser, first-stage parser or P (possibly with index) denote Charniak parser without reranker. It can return N best parses for various N, including N=1.
• Reranker, second stage or R (typically without index) denote the reranker. It could return reordered N-best list, but usually we only want one best parse.
• Brown parser, reranking parser or PR (possibly with index) denote the sequence of (P, R). It returns one best parse.
• The models pretrained at Brown and delivered with the reranking parser are indexed with 0: P₀, R₀ (or R) and PR₀.

• Install the parser-reranker suite.
  • Build it for the right architecture, using the right optimizations.
  • Investigate parallelization possibilities.
  • Write invocation scripts.
• Get the baseline. Test PR₀ on PTB WSJ 22. We will use section 22 for development and section 23 for “final testing”. The distinction is not very important since we only repeate someone's experiment we are not going to publish, but just in case we need to look more closely on the data, we will follow the usual procedure.
• Get the big raw corpus (North American News Text, NANT).
• Separate the Los Angeles Times part (from now on, this will be what we mean by NANT).
• Convert SGML to plain text.
• Tokenize it ($HIEROROOT/preprocess/tokenizeE.pl - - < plainText > tokenizedText).
• Find sentence boundaries.
• Clean the data, discard bad and long sentences.
• Parse NANT using the pretrained reranking parser (PR₀).
• Retrain the first-stage parser on WSJ + the parsed NANT ⇒ get P₁.
• New reranking parser PR₁ consists of the new first-stage parser P₁ and the old second-stage reranker R. Test the new reranking parser on WSJ.

We tested the pretrained reranking parser (PR₀) on sections 22 (development) 23 (final evaluation test). All evaluations take into account only sentences of 40 or fewer words.

$PARSINGROOT/evalb/evalb -p $PARSINGROOT/evalb/charniak.prm $PTB/ptbwsj22.penn $PTB/ptbwsj22.charniak
$PARSINGROOT/evalb/evalb -p $PARSINGROOT/evalb/charniak.prm $PTB/ptbwsj22.penn $PTB/ptbwsj22.brown
$PARSINGROOT/evalb/evalb -p $PARSINGROOT/evalb/charniak.prm $PTB/ptbwsj23.penn $PTB/ptbwsj23.charniak
$PARSINGROOT/evalb/evalb -p $PARSINGROOT/evalb/charniak.prm $PTB/ptbwsj23.penn $PTB/ptbwsj23.brown

See North American News Text Corpus for more information on the data and its preparation.

See here for more information on the Brown Reranking Parser. We parsed the LATWP part of NANTC on the C cluster using the following command:

cd /fs/clip-corpora/north_american_news
$PARSINGROOT/brown-reranking-parser/scripts/cluster-parse.pl -l en -g wsj.tgz < latwp.04.clean.txt \
    -o latwp.05a.brown.penn -w workdir05 -k

Parsing takes about 80 CPU-hours.

The following command trains the Charniak parser on 5 copies of the sections 02-21 of the Penn Treebank Wall Street Journal, and 1 copy of the parsed part of NANTC (<span style=“background:yellow”>3,143,433</span> sentences).

$PARSINGROOT/charniak-parser/scripts/train.pl ptbwsj02-21.5times.penn latwp.05a.brown.penn > ptb+latwp3000.tgz

The new non-reranking parser will be called P₁. The reranking parser P₁+R will be called PR₁.

$PARSINGROOT/charniak-parser/scripts/parse.pl -g ptb+latwp3000.tgz \
    < $PTB/ptbwsj22.txt \
    > ptbwsj22.ec.ptb+latwp3000.penn
$PARSINGROOT/evalb/evalb -p $PARSINGROOT/evalb/charniak.prm $PTB/ptbwsj22.penn ptbwsj22.ec.ptb+latwp3000.penn

$PARSINGROOT/charniak-parser/scripts/parse.pl -g ptb+latwp3000.tgz \
    < $PTB/ptbwsj23.txt \
    > ptbwsj23.ec.ptb+latwp3000.penn
$PARSINGROOT/evalb/evalb -p $PARSINGROOT/evalb/charniak.prm $PTB/ptbwsj23.penn ptbwsj23.ec.ptb+latwp3000.penn

First we combine the new parser with the old reranker.

$PARSINGROOT/brown-reranking-parser/scripts/combine_brown_models.pl ptb+latwp3000.tgz wsj.tgz \
    > ptb+latwp3000.brown.tgz

Then we use the combined model to parse the test data.

$PARSINGROOT/brown-reranking-parser/scripts/cluster-parse.pl -g ptb+latwp3000.brown.tgz \
    < $PTB/ptbwsj22.txt \
    -o ptbwsj22.br.ptb+latwp3000.penn
$PARSINGROOT/evalb/evalb -p $PARSINGROOT/evalb/charniak.prm $PTB/ptbwsj22.penn ptbwsj22.br.ptb+latwp3000.penn

$PARSINGROOT/brown-reranking-parser/scripts/cluster-parse.pl -g ptb+latwp3000.brown.tgz \
    < $PTB/ptbwsj23.txt \
    -o ptbwsj23.br.ptb+latwp3000.penn
$PARSINGROOT/evalb/evalb -p $PARSINGROOT/evalb/charniak.prm $PTB/ptbwsj23.penn ptbwsj23.br.ptb+latwp3000.penn

McClosky et al. do not use all 3 million sentences. They found that the best results over their development data (section 22) were obtained by mixing 5 copies of the Penn Treebank Wall Street Journal sections 02-21 and (first?) 1,750,000 sentences from NANTC LATWP.

head -1750000 latwp.05a.brown.penn > latwp.1750k.brown.penn

Train Charniak on this mix. It takes more than 4 hours.

$PARSINGROOT/charniak-parser/scripts/train.pl ptbwsj02-21.5times.penn latwp.1750k.brown.penn > ptb+latwp1750.tgz

Create new Brown model: combine the new parser with the old reranker.

$PARSINGROOT/brown-reranking-parser/scripts/combine_brown_models.pl ptb+latwp1750.tgz wsj.tgz \
    > ptb+latwp1750.brown.tgz

Parse the test sections and evaluate the results.

$PARSINGROOT/charniak-parser/scripts/cluster-parse.pl -g ptb+latwp1750.tgz \
    < $PTB/ptbwsj22.txt \
    -o ptbwsj22.ec.ptb+latwp1750.penn
$PARSINGROOT/evalb/evalb -p $PARSINGROOT/evalb/charniak.prm $PTB/ptbwsj22.penn ptbwsj22.ec.ptb+latwp1750.penn

$PARSINGROOT/charniak-parser/scripts/cluster-parse.pl -g ptb+latwp1750.tgz \
    < $PTB/ptbwsj23.txt \
    -o ptbwsj23.ec.ptb+latwp1750.penn
$PARSINGROOT/evalb/evalb -p $PARSINGROOT/evalb/charniak.prm $PTB/ptbwsj23.penn ptbwsj23.ec.ptb+latwp1750.penn

$PARSINGROOT/brown-reranking-parser/scripts/cluster-parse.pl -g ptb+latwp1750.brown.tgz \
    < $PTB/ptbwsj22.txt \
    -o ptbwsj22.br.ptb+latwp1750.penn
$PARSINGROOT/evalb/evalb -p $PARSINGROOT/evalb/charniak.prm $PTB/ptbwsj22.penn ptbwsj22.br.ptb+latwp1750.penn

$PARSINGROOT/brown-reranking-parser/scripts/cluster-parse.pl -g ptb+latwp1750.brown.tgz \
    < $PTB/ptbwsj23.txt \
    -o ptbwsj23.br.ptb+latwp1750.penn
$PARSINGROOT/evalb/evalb -p $PARSINGROOT/evalb/charniak.prm $PTB/ptbwsj23.penn ptbwsj23.br.ptb+latwp1750.penn

Train Charniak on 5 copies of PTB WSJ 02-21, without any trees from NANTC.

$PARSINGROOT/charniak-parser/scripts/train.pl < ptbwsj02-21.5times.penn > 5ptb.tgz

Parse sections 22 and 23 using the just trained model.

$PARSINGROOT/charniak-parser/scripts/cluster-parse.pl -g 5ptb.tgz \
    < $PTB/ptbwsj22.txt \
    -o ptbwsj22.ec.5ptb.penn
$PARSINGROOT/evalb/evalb -p $PARSINGROOT/evalb/charniak.prm $PTB/ptbwsj22.penn ptbwsj22.ec.5ptb.penn

$PARSINGROOT/charniak-parser/scripts/cluster-parse.pl -g 5ptb.tgz \
    < $PTB/ptbwsj23.txt \
    -o ptbwsj23.ec.5ptb.penn
$PARSINGROOT/evalb/evalb -p $PARSINGROOT/evalb/charniak.prm $PTB/ptbwsj23.penn ptbwsj23.ec.5ptb.penn

The following table combines tables from previous sections and from the McClosky et al. (2006) paper. I am not sure why I do not get the same baseline numbers as McClosky et al. Possibly they evaluate all sentences rather than just those of 40 or fewer words.

Remember that Charniak parser means without reranker, Brown parser means with reranker. PTB WSJ (or WSJ) in training means sections 02-21. 50k NANTC means 50,000 sentences of NANTC LATWP.

Experiments
English
Parsing