Perl is the language to write a driver. A driver is a simple Perl module (.pm). It should implement the following functions: decode(), encode(), list(). The tagset/common.pm module contains some code you may have use for, so your driver module should start with
use tagset::common;
The input/output tag can be any string. If the information is stored in several kinds of tags, they can be passed in one string, using some unique delimiters. We recommend “\t” (horizontal tab, ASCII 9) as delimiter. If desirable, the input/output tag can be even a multi-line XML!
Empty feature value means “unknown”. It is even not known, whether this feature would be relevant. Some tagsets distinguish between unknown values and irrelevant features. This is not the case of Interset. While something can be irrelevant in one tagset, we can hardly say that it is not relevant in any tagset. So, since we are setting a value in a universal “tagset”, we probably better leave the value empty or even set it to an appropriate default.
This function has one string argument, the tag. The function returns a reference to a hash of features (feature names are hash keys to the feature values).
The decoder is not obliged to set any feature. If the decoder decides to set a feature, it should be one of the pre-defined values. This can be checked by a central procedure. However, it is not mandatory, so if the appropriate value is not available, you can use your own, but please do let me know so I can update the central value pool accordingly. (If you set a value that is not documented as a part of the universal set, no one else can benefit from it. If you combine your driver with another driver to convert from your tag set to the other, the other driver's encode() will not take your invented value into account. It may even behave worse than if the value was empty.)
If the tagset encodes features separately (e.g., each character is a value of a particular feature): The decoder should be tolerant to unexpected combinations of features (or should be able to be tolerant if asked for it).
This function has one argument, a reference to a hash of features (feature names are hash keys to the feature values). The function returns a string - the tag.
The encoder should be able to process all possible values from the central pool. If the tagset does not recognize a value, the most appropriate substitute should be chosen.
Since any feature can in theory have an array of values instead of a single value, the encoder should either be prepared to arrays (more precisely: array references) anywhere, or call tagset::single_values() to get rid of the arrays (or some of them). See Alternative values for more details.
WARNING: Before modifying the contents of the feature structure, you should make a deep copy of it. You cannot assume that the user of the driver will not need the original values of the features after encoding. If you have called single_values(), it made the copy for you.
The list() function returns a reference to an array of all plausible tags in the set. Undefined return value means no list is available and thus any tag is plausible. If there is a list, strict encoding should conform to it.
The list is not necessary for the driver to work. However, it can be useful for testing the driver. If no list is distributed along with the tagset description, you may still be able to acquire a partial list from a corpus.
A feature can have two or more alternative values at the same time. This may be necessary because some tags in some tagsets are defined that way. For instance, the character H on position 8 in the PDT Czech tagset means “tense = past or present.” Alternative values are represented by a single reference to an array of values.
The corresponding decode() code may look like this:
elsif($tense eq "H") { $f{tense} = ["past", "pres"]; }
Note that every encode() implementation should be prepared to work with multiple values of features! Even if your own driver does not create arrays in decode(), you should be able to deal with arrays coming from other tagsets. However, if you do not create arrays, your tagset is probably not capable of storing alternative feature values and you cannot do much more than just pick the first value from the array and use it. In that case it is annoying to have to check for arrays in every single feature. Fortunately, you can get rid of all arrays at once by calling the single_values() function at the beginning of your encode():
use tagset; sub encode { my $f = shift; # Replace any array values by single values. my $single = tagset::single_values($f, "gender", "animateness", "number", "tense"); my %f = %{$single};
The single_values() function will replace any array by its first member value. Note that you can specify features that are exceptions to this rule. In the above example, the features gender, animateness, number and tense will retain multiple values, if present. Thus if you are able to deal with arrays, this is the way how you specify where you can do so and where you cannot.
Now, what do you do with features where you want to encode arrays? You should first check whether the value is an array or not. If it is an array, you may want to grep your values rather than trying exact match, because you do not know what is going to come from other drivers, and the ordering or additional values may not be what matters.
If the array turns out to be incompatible with what you expect, you should pick one value (we suggest you take the first one) and proceed with default single-value processing.
if(ref($f{gender}) eq "ARRAY") { # Processing of any combined values goes here. my @values = @{$f{gender}}; if(scalar(grep{m/^masc$/}(@values)) && scalar(grep{m/^fem$/}(@values))) { $tag[2] = "T"; } elsif(...) { ... } else { $f{gender} = $f{gender}[0]; } } if(ref($f{gender}) eq "") { if($f{gender} eq "masc") { ...
Note: This approach cannot encode situations where some combinations of feature values are plausible and some are not! For instance, if positions [2] and [3] in a tag encode gender and number, respectively, and if NNQW means a logical disjunction of the tags NNFS and NNNP, then you cannot encode the situation in DZ Interset precisely. If you do not want to discard either NNFS or NNNP (by storing the other only), you can say that gender = F or N and number = S or P but by that you have also introduced NNFP and NNNS as possibilities. The approach may be revised in future.
The encoder's problem is that there are more feature values on input than can be encoded on output. If a value cannot be encoded, the encoder must replace it with a suitable default. Although it can control the replacement completely by its own means (e.g. by a system of if-else statements), there is a central system of defaults that can take care of it. The central system however needs the following:
tagset::common. A driver can supply its own, if needed.list() driver function). Then the central system will return the highest-priority permitted value. A value is permitted if the tag set contains a tag that yields the value when decoded.
Building the list of permitted values is expensive (all tags must be decoded!) and you should do it only once when your driver initializes. In your BEGIN block, you should call tagset::common::get_permitted_values() and store the hash reference it returns. The hash (of arrays) will contain a list of permitted values for every feature. When you later need to check a value and replace it if necessary, you pass the hash reference back to tagset::common:
(Note that the list() function must be defined before the BEGIN block that uses it.)
use tagset::common; ... sub list { ... } ... BEGIN { # Store the hash reference in a global variable. $permitvals = tagset::common::get_permitted_values(list(), \&decode); } ... $replacement = tagset::common::check_value($feature, $value, $permitvals);
Alternatively, the following checks and replaces values of all features in a feature structure:
tagset::common::enforce_permitted_values($fstruct, $permitvals);
If an array is a permitted value, all member values are permitted.
If an array is checked, all member values must be permitted in order for the array to be permitted. Otherwise, the array is pruned and the replacement is a subarray where only permitted values are kept. If no member values are permitted (hence the pruned subarray would be empty), the replacement is a single value, the highest-priority replacement of the first element of the array. If the original array was empty (which should never happen but we ought to be careful anyway), the single empty value is checked and possibly replaced.
The above technique does not guarantee that the encoder will only see feature combinations that it normally gets from its own decoder. For instance, the case feature may be known in the given tagset but it only occurs with nouns. When encoding features produced by a different driver, there could be non-empty case also for adjectives, pronouns etc. Even though the encoder may be able to encode the new feature combination (using the tagset's usual way of expressing the values “adjective” and, say, “genitive”) it is not always desirable (depending on the application) to invent new tags. Default encoding should be strict, meaning that only the tags returned by list() can be returned by the encoder.
Similarly to the replacement of separate values, the encoder can ask the Interset common library to replace the whole structure by something the encoder is used to (i.e. by a structure that results from decoding of a tag known by the driver). This usually removes from the encoder the burden of thinking about exotic features and values.
The correcting function tries to lie as little as possible. There is a priority value associated with every known feature. Feature values are checked (and possibly altered) in the order of feature priorities. In the above example (adjective cannot have case), the part of speech would keep its “adjective” value, and the case value would be removed. If the case feature had higher priority than the pos feature (which is not the case), a non-empty value of case would force part of speech to be changed from adjective to noun.
use tagset::common; ... sub list { ... } ... BEGIN { # Store the hash reference in a global variable. $permitted = tagset::common::get_permitted_structures_joint(list(), \&decode); } ... # Give reference to feature structure. Get reference to a new one (deep copy). $fs1 = tagset::common::enforce_permitted_joint($fs0, $permitted);
Replacing feature values with defaults has its limitations. It only works with pre-known feature values. It does not touch the features tagset and other. Such behavior is indeed correct as these features only serve to preserve exotic information when encoding back into the original tagset. However, it also has its downside.
The key problem lies in the method we use to obtain permitted combinations of feature values. All tags of the tagset are decoded into feature structures, which subsequently represent the permitted combinations. Values of tagset and other are disregarded. Now what happens if the resulting feature structure (after erasing tagset and other) is unique, i.e. if if it never occurs without the particular value of other? The encoder cannot create the tag corresponding to the other value (“o-tag”) because the information contained in other has been erased. It will thus create a similar tag (“s-tag”). Unfortunately, the s-tag will be constructed using feature values that can only occur with the o-tag. Thus, the s-tag will not be valid.
Example: Let's assume that the target tagset contains a narrow-purpose tag called pred. It is not intended for verbs but the words tagged with it often have predicative function and work instead of verbs, so we decided to decode it as a strange type of verb. We set pos = “verb” and other = “pred”. Unfortunately, all verb tags in that particular tagset have the feature aspect set, which is not true for pred. The part of encoder that is responsible for replacing does not see the difference between pred and verb because it does not read the value of other. It will learn (wrongly) that aspectless verbs exist. And it will start creating invalid aspectless verb tags once it runs across a feature structure from a tagset that does not know about aspect.
The example is a realistic one. O-tags (tags setting the other feature) are often minor parts of speech. They are used for tokens that hide under broader parts of speech in other tagsets. The specific usage of the o-tags however makes many features of the broader tags unnecessary. Such features are empty in o-tags while they always must be non-empty in corresponding s-tags.
A possible solution would be not to use any o-tags when scanning the possible feature value combinations. This would work for numerous tagset drivers that only resort to other when dealing with a “strange” tag. One would have to make sure when distinguishing a strange tag from its normal counterpart that only the strange tag has other set, and that the normal tag has it empty (in other words, we cannot set other for both, say, other = “strange” for the former and other = “normal” for the latter). Nevertheless, there are instances where most or all the tags of a tagset are o-tags. A good example is zh::conll: poorly documented set of 294 tags, with most distinctions unrepresentable in DZ Interset. Its decoder only sets pos and copies the whole tag into other. Excluding o-tags (meaning all tags here) would not work with this tagset.
Another possible solution is to implement a new subroutine that returns the list of the tags that can be used for scanning of permitted feature value combinations. By default, the subroutine would return the list of non-o-tags. For tagsets such as zh::conll, it could create a taylored list of tags.
Current solution: New (third) parameter to tagset::common::get_permitted_structures_joint(), when set to 1 (true), says that o-tags shall be ignored. Note that it requires changing the beginning of encode(). We must not enforce permitted combinations if source tagset is same as target (because we would be violating o-tags without knowing their permitted values). Instead, we should just do a simple deep copy of the feature structure to protect it from incidential modifications during encoding. The following example is taken from the pl::ipipan driver, which was the first place where the solution was applied.
my $f0 = shift; # Modify the feature structure so that it contains values expected by this # driver. Do not do that if this was also the source tagset (because the # modification would damage tags using 'other'). However, in any case # create a deep copy of the original feature structure so that it is # protected from changes during encoding. my $f; if($f0->{tagset} eq 'pl::ipipan') { $f = tagset::common::duplicate($f0); } else { $f = tagset::common::enforce_permitted_joint($f0, $permitted); } my %f = %{$f};
See Common Problems for a list of suggestions for phenomena difficult to match between tagsets and the Interset.
When you have written a driver for a new tagset, you should test it. The driver package contains a test script called driver-test.pl. When running it, give the driver name as argument, without the tagset:: prefix. You can also use the -d option to turn on debug messages (list of tags being tested).
driver-test.pl ar::conll driver-test.pl bg::conll cs::pdt driver-test.pl -a driver-test.pl -A
Running driver-test.pl without arguments will list the drivers available on the system. Two (or more) arguments test both drivers separately and then conversions from driver A to driver B and vice versa. Running it with the -a option will test all the drivers but no conversions. The -A option tests all drivers and all conversions between all pairs of drivers.
Note that only drivers implementing the list() function can be tested. Most testing involves generating the list of all possible tags and testing the driver on each tag separately.
The following tests will be performed for a single driver:
encode(decode($tag)) eq $tag. While sometimes it can be annoying to try to preserve some obscure information hidden in the tags, this test can also reveal many unwanted bugs. Besides, you should preserve information of your own tagset because people may want to use your driver merely to access the tags, instead of converting them.The following tests will be performed for a pair of drivers: