| Both sides previous revision
Previous revision
Next revision
|
Previous revision
|
user:zeman:interset:how-to-use [2008/03/10 14:23]
zeman Cesta k nástrojům. |
user:zeman:interset:how-to-use [2017/01/16 13:06] (current)
zeman |
| ===== Manual ===== | ====== DZ Interset Manual ====== |
| |
| ==== Installation ==== | ===== Installation ===== |
| |
| If you exist on the ÚFAL network, you can use directly Dan's version here. Otherwise, you need to [[mailto:zeman@ufal.mff.cuni.cz|ask Dan]] for a zipped package of the currently existing drivers. (I intend to maintain it here for download some time later.) Unzip it to a convenient place; below, we assume it is in ''/home/zeman/lib/perl''. | If you exist on the ÚFAL network and use Perl from PerlBrew, you probably already have ''Lingua::Interset'' available (depending on which Perl version you take from PerlBrew). Or you can point your ''PERL5LIB'' directly to Dan's version in ''/home/zeman/projekty/interset/lib''. Otherwise you can install ''Lingua::Interset'' from CPAN, e.g. using |
| |
| **Note:** I decided to put the whole thing under version control. At the same time, I moved it from my lib folder to my project folder. So the current ÚFAL location is ''/home/zeman/projekty/interset/lib''. | <code bash>cpanm Lingua::Interset</code> |
| |
| **Contributions welcome!** If you write your own driver, please share it with others! If you send it to me, I will add it to the package for download here. | **Contributions welcome!** If you write your own driver, please share it with others! If you send it to me, I will add it to the package on CPAN. |
| |
| === Existing drivers === | ==== Existing drivers ==== |
| |
| * tagset::ar::conll - Arabic CoNLL treebank (coarse, fine and feat fields in one string, delimited by tabs) | Use the tool ''bin/driver-test.pl'' from the package (call it without arguments) to list the tagsets/drivers currently available on your system. |
| * tagset::bg::conll - Bulgarian CoNLL treebank | |
| * tagset::cs::pdt - Czech positional tags of the Prague Dependency Treebank | |
| * tagset::da::conll - Danish CoNLL treebank | |
| * tagset::en::conll - English CoNLL treebank | |
| * tagset::en::penn - English Penn Treebank | |
| * tagset::sv::hajic - Tags output by Swedish tagger by Jan Hajič | |
| * tagset::sv::mamba - Swedish Mamba tags from Talbanken05 (CoNLL treebank) | |
| * tagset::sv::svdahybrid - Dan's tagset, aiming at making distribution of tags from sv::hajic and da::conll as close as possible | |
| |
| === Directory Structure === | ==== Directory Structure ==== |
| |
| The drivers are Perl modules and must be somewhere under ''$PERLLIB'' (''@INC''). Their root folder is ''tagset'' (this is what separates the tag set drivers from other Perl libraries). Subfolders of ''tagset'' are two-letter codes of languages (ISO). Some tagsets may be designed for more than one language but most are language-specific. PM files in language folders are drivers. Drivers are called xxx.pm, where xxx is the code name of the tagset. The driver xxx.pm for language ll should be accessible from Perl via | The drivers are Perl modules and must be somewhere under ''$PERLLIB'' (''@INC''). Their root folder is ''Lingua/Interset/Tagset''. Subfolders of ''Tagset'' are two-letter codes of languages ([[http://en.wikipedia.org/wiki/ISO_639-1|ISO 639-1]]), uppercased (because of the convention that Perl modules start with an uppercase letter). Some tagsets may be designed for more than one language but most are language-specific. PM files in language folders are drivers. Drivers are called Xxx.pm, where xxx is the code name of the tagset. The driver Xxx.pm for language ll should be accessible from Perl via |
| |
| <code perl> | <code perl> |
| use tagset::ll::xxx; | use Lingua::Interset::Tagset::LL::Xxx; |
| </code> | </code> |
| |
| Besides drivers, there is a library of useful functions that can be called from within drivers: ''tagset/common.pm''. | but usually it is more convenient to just call the main module and then refer to the tagset using the lowercased identifier: |
| |
| There is also the driver testing script, ''bin/driver-test.pl''. The distribution may contain some sample conversion scripts as well; however, these depend much more on the file format than on the tagset drivers, and thus you'll probably need to write your own anyway. | <code perl> |
| | use Lingua::Interset qw(decode); |
| | my $fs = decode('ll::xxx', $tag); |
| | </code> |
| |
| | The main object in Interset is of the class ''Lingua::Interset::FeatureStructure'', which provides various useful access methods. For details, see the documentation at https://metacpan.org/pod/Lingua::Interset |
| |
| | There is also the driver testing script, ''bin/driver-test.pl''. The distribution may contain some sample conversion scripts as well; however, these depend much more on the file format than on the tagset drivers, and thus you'll probably need to write your own anyway. |
| |
| ==== How to use the Interset ==== | |
| |
| You can write your own tag conversion Perl script, and use the Interset driver library. You have to tell Perl where to find the drivers: | ===== How to use the Interset ===== |
| | |
| | You can write your own Perl script to convert tags, and use the Interset driver library. You may have to tell Perl where to find Interset (the following commands work in ''csh''; you have to use different syntax under ''bash'' or in Windows command line): |
| |
| <code>setenv PERLLIB /home/zeman/projekty/interset/lib:$PERLLIB | <code>setenv PERLLIB /home/zeman/projekty/interset/lib:$PERLLIB |
| setenv PATH /home/zeman/projekty/interset/bin:$PATH</code> | setenv PATH /home/zeman/projekty/interset/bin:$PATH</code> |
| |
| Once the variable is set, writing a conversion script is very easy. For instance, my ''csts-cs-pdt-en-penn.pl'' script (meaning "read and write [[:Formát CSTS|CSTS format]], read Czech PDT tags, write English Penn tags) essentially looks like this: | Once the variable is set, writing a conversion script is very easy. Here is an example (note that in CoNLL-X files we often merge the contents of the CPOS, POS and FEATS columns to create one long string that will be seen by Interset as one “tag”): |
| |
| <code perl> | <code perl> |
| use tagset::cs::pdt; | use Lingua::Interset::Converter; |
| use tagset::en::penn; | |
| |
| | my $c = new Lingua::Interset::Converter ('from' => $tagset1, 'to' => 'mul::uposf'); |
| | |
| | # Read the CoNLL-X file from STDIN or from files given as arguments. |
| while(<>) | while(<>) |
| { | { |
| if(s/<t>([^<]+)/<_tag_to_convert_>/) | unless(m/^\s*$/) |
| { | { |
| my $tag0 = $1; | chomp(); |
| my $features = tagset::cs::pdt::decode($tag0); | my @f = split(/\t/, $_); |
| my $tag1 = tagset::en::penn::encode($features); | my $tag = "$f[3]\t$f[4]\t$f[5]"; |
| s/<_tag_to_convert_>/<t>$tag1/; | my $utag = $c->convert($tag); |
| | my ($upos, $ufeat) = split(/\t/, $utag); |
| | $f[3] = $upos; |
| | $f[5] = $ufeat; |
| | $_ = join("\t", @f)."\n"; |
| } | } |
| print; | # Write the modified line to the standard output. |
| | print(); |
| } | } |
| </code> | </code> |
| |
| Note the two-step replacement of the original tag. I do not dare to use the original tag in a regular expression because there could be special characters in the tag. | |
| |
