Perl is the language to write a driver. A driver is a simple Perl module (.pm). It should implement the following functions: decode(), encode(), list(). The tagset/common.pm module contains some code you may have use for, so your driver module should start with

use tagset::common;

The input/output tag can be any string. If the information is stored in several kinds of tags, they can be passed in one string, using some unique delimiters. We recommend “\t” (horizontal tab, ASCII 9) as delimiter. If desirable, the input/output tag can be even a multi-line XML!

Empty feature value means “unknown”. It is even not known, whether this feature would be relevant. Some tagsets distinguish between unknown values and irrelevant features. This is not the case of Interset. While something can be irrelevant in one tagset, we can hardly say that it is not relevant in any tagset. So, since we are setting a value in a universal “tagset”, we probably better leave the value empty or even set it to an appropriate default.

This function has one string argument, the tag. The function returns a reference to a hash of features (feature names are hash keys to the feature values).

The decoder is not obliged to set any feature. If the decoder decides to set a feature, it should be one of the pre-defined values. This can be checked by a central procedure. However, it is not mandatory, so if the appropriate value is not available, you can use your own, but please do let me know so I can update the central value pool accordingly. (If you set a value that is not documented as a part of the universal set, no one else can benefit from it. If you combine your driver with another driver to convert from your tag set to the other, the other driver's encode() will not take your invented value into account. It may even behave worse than if the value was empty.)

If the tagset encodes features separately (e.g., each character is a value of a particular feature): The decoder should be tolerant to unexpected combinations of features (or should be able to be tolerant if asked for it).

This function has one argument, a reference to a hash of features (feature names are hash keys to the feature values). The function returns a string - the tag.

The encoder should be able to process all possible values from the central pool. If the tagset does not recognize a value, the most appropriate substitute should be chosen.

Since any feature can in theory have an array of values instead of a single value, the encoder should either be prepared to arrays (more precisely: array references) anywhere, or call tagset::single_values() to get rid of the arrays (or some of them). See Alternative values for more details.

WARNING: Before modifying the contents of the feature structure, you should make a deep copy of it. You cannot assume that the user of the driver will not need the original values of the features after encoding. If you have called single_values(), it made the copy for you.

The list() function returns a reference to an array of all plausible tags in the set. Undefined return value means no list is available and thus any tag is plausible. If there is a list, strict encoding should conform to it.

The list is not necessary for the driver to work. However, it can be useful for testing the driver. If no list is distributed along with the tagset description, you may still be able to acquire a partial list from a corpus.

A feature can have two or more alternative values at the same time. This may be necessary because some tags in some tagsets are defined that way. For instance, the character H on position 8 in the PDT Czech tagset means “tense = past or present.” Alternative values are represented by a single reference to an array of values.

The corresponding decode() code may look like this:

elsif($tense eq "H")
{
    $f{tense} = ["past", "pres"];
}

Note that every encode() implementation should be prepared to work with multiple values of features! Even if your own driver does not create arrays in decode(), you should be able to deal with arrays coming from other tagsets. However, if you do not create arrays, your tagset is probably not capable of storing alternative feature values and you cannot do much more than just pick the first value from the array and use it. In that case it is annoying to have to check for arrays in every single feature. Fortunately, you can get rid of all arrays at once by calling the single_values() function at the beginning of your encode():

use tagset;
sub encode
{
    my $f = shift;
    # Replace any array values by single values.
    my $single = tagset::single_values($f, "gender", "animateness", "number", "tense");
    my %f = %{$single};

The single_values() function will replace any array by its first member value. Note that you can specify features that are exceptions to this rule. In the above example, the features gender, animateness, number and tense will retain multiple values, if present. Thus if you are able to deal with arrays, this is the way how you specify where you can do so and where you cannot.

Now, what do you do with features where you want to encode arrays? You should first check whether the value is an array or not. If it is an array, you may want to grep your values rather than trying exact match, because you do not know what is going to come from other drivers, and the ordering or additional values may not be what matters.

If the array turns out to be incompatible with what you expect, you should pick one value (we suggest you take the first one) and proceed with default single-value processing.

    if(ref($f{gender}) eq "ARRAY")
    {
        # Processing of any combined values goes here.
        my @values = @{$f{gender}};
        if(scalar(grep{m/^masc$/}(@values)) &&
           scalar(grep{m/^fem$/}(@values)))
        {
            $tag[2] = "T";
        }
        elsif(...)
        {
            ...
        }
        else
        {
            $f{gender} = $f{gender}[0];
        }
    }
    if(ref($f{gender}) eq "")
    {
        if($f{gender} eq "masc")
        {
            ...

Note: This approach cannot encode situations where some combinations of feature values are plausible and some are not! For instance, if positions [2] and [3] in a tag encode gender and number, respectively, and if NNQW means a logical disjunction of the tags NNFS and NNNP, then you cannot encode the situation in DZ Interset precisely. If you do not want to discard either NNFS or NNNP (by storing the other only), you can say that gender = F or N and number = S or P but by that you have also introduced NNFP and NNNS as possibilities. The approach may be revised in future.

The encoder's problem is that there are more feature values on input than can be encoded on output. If a value cannot be encoded, the encoder must replace it with a suitable default. Although it can control the replacement completely by its own means (e.g. by a system of if-else statements), there is a central system of defaults that can take care of it. The central system however needs the following:

1. A table of replacement values for each value, ordered by precedence. There is a default table in tagset::common. A driver can supply its own, if needed.
2. The list of all tags in the tag set (implemented by the list() driver function). Then the central system will return the highest-priority permitted value. A value is permitted if the tag set contains a tag that yields the value when decoded.

Building the list of permitted values is expensive (all tags must be decoded!) and you should do it only once when your driver initializes. In your BEGIN block, you should call tagset::common::get_permitted_values() and store the hash reference it returns. The hash (of arrays) will contain a list of permitted values for every feature. When you later need to check a value and replace it if necessary, you pass the hash reference back to tagset::common:

(Note that the list() function must be defined before the BEGIN block that uses it.)

use tagset::common;
...
sub list { ... }
...
BEGIN
{
    # Store the hash reference in a global variable.
    $permitvals = tagset::common::get_permitted_values(list(), \&decode);
}
...
$replacement = tagset::common::check_value($feature, $value, $permitvals);

Alternatively, the following checks and replaces values of all features in a feature structure:

tagset::common::enforce_permitted_values($fstruct, $permitvals);

If an array is a permitted value, all member values are permitted.

If an array is checked, all member values must be permitted in order for the array to be permitted. Otherwise, the array is pruned and the replacement is a subarray where only permitted values are kept. If no member values are permitted (hence the pruned subarray would be empty), the replacement is a single value, the highest-priority replacement of the first element of the array. If the original array was empty (which should never happen but we ought to be careful anyway), the single empty value is checked and possibly replaced.

See Common Problems for a list of suggestions for phenomena difficult to match between tagsets and the Interset.

When you have written a driver for a new tagset, you should test it. The driver package contains a test script called driver-test.pl. When running it, give the driver name as argument, without the tagset:: prefix. You can also use the -d option to turn on debug messages (list of tags being tested).

driver-test.pl ar::conll
driver-test.pl -a
driver-test.pl bg::conll cs::pdt
driver-test.pl -A

Running driver-test.pl without arguments will list the drivers available on the system. Running it with the -a option will test all the drivers. Two arguments test both drivers separately and then conversions from driver A to driver B and vice versa. The -A option tests all conversions between all pairs of drivers.

Note that only drivers implementing the list() function can be tested. Most testing involves generating the list of all possible tags and testing the driver on each tag separately.

The following tests will be performed for a single driver:

• Decode each tag and check that only known features and values are set. In addition to a built-in list, every feature can have an empty value, and the features “tagset” and “other” can have any value.
• Check for each tag that encode(decode($tag)) eq $tag. While sometimes it can be annoying to try to preserve some obscure information hidden in the tags, this test can also reveal many unwanted bugs. Besides, you should preserve information of your own tagset because people may want to use your driver merely to access the tags, instead of converting them.

The following tests will be performed for a pair of drivers:

• Decode every tag of the first driver, encode it using the second driver and check whether the result is a known tag in the second tagset.