Differences

This shows you the differences between two versions of the page.

--- user:zeman:interset:how-to-write-a-driver [2007/10/01 13:53]
zeman use tagset::common;
+++ user:zeman:interset:how-to-write-a-driver [2007/10/01 17:38]
zeman
@@ Line 93: / Line 93: @@
 **Note:** This approach cannot encode situations where some combinations of feature values are plausible and some are not! For instance, if positions [2] and [3] in a tag encode gender and number, respectively, and if ''NNQW'' means a logical disjunction of the tags ''NNFS'' and ''NNNP'', then you cannot encode the situation in DZ Interset precisely. If you do not want to discard either ''NNFS'' or ''NNNP'' (by storing the other only), you can say that gender = ''F'' or ''N'' and number = ''S'' or ''P'' but by that you have also introduced ''NNFP'' and ''NNNS'' as possibilities. The approach may be revised in future.
+===== Replacing feature values with defaults =====
+The encoder's problem is that there are more feature values on input than can be encoded on output. If a value cannot be encoded, the encoder must replace it with a suitable default. Although it can control the replacement completely by its own means (e.g. by a system of ''if''-''else'' statements), there is a central system of defaults that can take care of it. The central system however needs the following:
+  - A table of replacement values for each value, ordered by precedence. There is a default table in ''tagset::common''. A driver can supply its own, if needed.
+  - The list of all tags in the tag set (implemented by the ''list()'' driver function). Then the central system will return the highest-priority //permitted// value. A value is permitted if the tag set contains a tag that yields the value when decoded.
+Building the list of permitted values is expensive (all tags must be decoded!) and you should do it only once when your driver initializes. In your ''BEGIN'' block, you should call ''tagset::common::get_permitted_values()'' and store the hash reference it returns. The hash (of arrays) will contain a list of permitted values for every feature. When you later need to check a value and replace it if necessary, you pass the hash reference back to ''tagset::common'':
+<code perl>
+use tagset::common;
+BEGIN
+{
+    # Store the hash reference in a global variable.
+    $permitvals = tagset::common::get_permitted_values(list(), \&decode);
+}
+...
+$replacement = tagset::common::check_value($feature, $value, $permitvals);
+</code>
+If an array is a permitted value, all member values are permitted.
+If an array is checked, all member values must be permitted in order for the array to be permitted. Otherwise, the array is pruned and the replacement is a subarray where only permitted values are kept. If no member values are permitted (hence the pruned subarray would be empty), the replacement is a single value, the highest-priority replacement of the first element of the array. If the original array was empty (which should never happen but we ought to be careful anyway), the single empty value is checked and possibly replaced.
 ===== Common problems =====
 See [[user:zeman:interset:Common Problems]] for a list of suggestions for phenomena difficult to match between tagsets and the Interset.
 ===== Test your driver =====
-When you have written a driver for a new tagset, you should test it. An Interset service module can perform the following tests:
+When you have written a driver for a new tagset, you should test it. The driver package contains a test script called ''driver-test.pl''. When running it, give the driver name as argument, without the ''tagset::'' prefix. You can also use the ''-d'' option to turn on debug messages (list of tags being tested):
+<code>
+driver-test.pl ar::conll
+</code>
+Note that only drivers implementing the ''list()'' function can be tested.
+The following tests will be performed:
-  * Get list of possible tags by calling list(). Go through the list and check for each tag that encode(decode($tag))=$tag. While sometimes it can be annoying to try to preserve some obscure information hidden in the tags, this test can also reveal many unwanted bugs. Besides, you should preserve information of your own tagset because people may want to use your driver merely to //access// the tags, instead of //converting// them.
+  * Get list of possible tags by calling ''list()''. Go through the list and check for each tag that ''encode(decode($tag)) eq $tag''. While sometimes it can be annoying to try to preserve some obscure information hidden in the tags, this test can also reveal many unwanted bugs. Besides, you should preserve information of your own tagset because people may want to use your driver merely to //access// the tags, instead of //converting// them.
-To perform the test, run the script ''driver-test.pl'' in the ''tagset'' root folder. Note that the name of the driver to test is currently hard-coded into the source. In future, it will be changed to a command-line argument.

[ Back to the navigation ] [ Back to the content ]

Institute of Formal and Applied Linguistics Wiki

Differences