[ Skip to the content ]

Institute of Formal and Applied Linguistics Wiki

[ Back to the navigation ]

You are here: start » user » zeman » interset » how-to-use

Differences

This shows you the differences between two versions of the page.

Link to this comparison view

Both sides previous revision Previous revision
Next revision 		Previous revision
Next revision Both sides next revision
user:zeman:interset:how-to-use [2008/03/10 13:10]
zeman New location of the drivers. 		user:zeman:interset:how-to-use [2008/03/13 17:46]
zeman Oprava.
Line 10: Line 10:
  
 === Existing drivers === === Existing drivers ===
 +
 +Note: This list may not be up-to-date. To see what drivers are currently available on your system, call ''driver-test.pl'' without arguments.
  
   * tagset::ar::conll - Arabic CoNLL treebank (coarse, fine and feat fields in one string, delimited by tabs)   * tagset::ar::conll - Arabic CoNLL treebank (coarse, fine and feat fields in one string, delimited by tabs)
Line 31: Line 33:
 Besides drivers, there is a library of useful functions that can be called from within drivers: ''tagset/common.pm''. Besides drivers, there is a library of useful functions that can be called from within drivers: ''tagset/common.pm''.
  
-There is also the driver testing script, ''driver-test.pl''. In the distribution package, this script is in the ''tagset'' folder. However, since ''tagset'' is going to live under one of your ''lib''s, you may prefer to move the script under one of your ''bin''s, e.g. ''~/bin''. The distribution may contain some sample conversion scripts as well; however, these depend much more on the file format than on the tagset drivers, and thus you'll probably need to write your own anyway.+There is also the driver testing script, ''bin/driver-test.pl''. The distribution may contain some sample conversion scripts as well; however, these depend much more on the file format than on the tagset drivers, and thus you'll probably need to write your own anyway. 
 + 
 + 
  
  
Line 38: Line 43:
 You can write your own tag conversion Perl script, and use the Interset driver library. You have to tell Perl where to find the drivers: You can write your own tag conversion Perl script, and use the Interset driver library. You have to tell Perl where to find the drivers:
  
-<code>setenv PERLLIB /home/zeman/projekty/interset/lib:$PERLLIB</code>+<code>setenv PERLLIB /home/zeman/projekty/interset/lib:$PERLLIB 
 +setenv PATH /home/zeman/projekty/interset/bin:$PATH</code>
  
 Once the variable is set, writing a conversion script is very easy. For instance, my ''csts-cs-pdt-en-penn.pl'' script (meaning "read and write [[:Formát CSTS|CSTS format]], read Czech PDT tags, write English Penn tags) essentially looks like this: Once the variable is set, writing a conversion script is very easy. For instance, my ''csts-cs-pdt-en-penn.pl'' script (meaning "read and write [[:Formát CSTS|CSTS format]], read Czech PDT tags, write English Penn tags) essentially looks like this:
Line 60: Line 66:
  
 Note the two-step replacement of the original tag. I do not dare to use the original tag in a regular expression because there could be special characters in the tag. Note the two-step replacement of the original tag. I do not dare to use the original tag in a regular expression because there could be special characters in the tag.
 +
 +Some operations performed by the drivers (especially when encoding) are not trivial. While you may not observe long processing times for toy runs, it might matter once you start converting millions of tags in a big corpus. Then you may want to use up the fact that there are tens to thousands of tags, and cache their translations like in the following example:
 +
 +<code perl>
 +use tagset::cs::pdt;
 +use tagset::en::penn;
 +
 +while(<>)
 +{
 +    if(s/<t>([^<]+)/<_tag_to_convert_>/)
 +    {
 +        my $tag0 = $1;
 +        my $tag1;
 +        if(exists($cache{$tag0}))
 +        {
 +            $tag1 = $cache{$tag0};
 +        }
 +        else
 +        {
 +            my $features = tagset::cs::pdt::decode($tag0);
 +            $tag1 = tagset::en::penn::encode($features);
 +            $cache{$tag0} = $tag1;
 +        }
 +        s/<_tag_to_convert_>/<t>$tag1/;
 +    }
 +    print;
 +}
 +</code>
  

[ Back to the navigation ] [ Back to the content ]