| Both sides previous revision
Previous revision
Next revision
|
Previous revision
|
user:zeman:mdmake [2023/04/21 16:28]
zeman [Makefile] Examples of values in individual dimensions. |
user:zeman:mdmake [2023/04/21 18:17] (current)
zeman Now versioned at Github. |
| ====== mdmake ====== | ====== mdmake ====== |
| | |
| | [[https://github.com/dan-zeman/mdmake]] |
| |
| Imagine you need to apply the same sequence of tools to a set of data files, and possibly want to be able to repeat the experiment later, i.e. sometime in future you will want recall how precisely the processing would be invoked. One example is a shared task in processing of similarly formatted data in many languages. One may want to use [[http://www.gnu.org/software/make/manual/make.html|make]] and Makefiles where the sequence of application of the various scripts can be well described. However, dealing with some phenomena of such sort of processing is rather tricky in classical Makefiles. | Imagine you need to apply the same sequence of tools to a set of data files, and possibly want to be able to repeat the experiment later, i.e. sometime in future you will want recall how precisely the processing would be invoked. One example is a shared task in processing of similarly formatted data in many languages. One may want to use [[http://www.gnu.org/software/make/manual/make.html|make]] and Makefiles where the sequence of application of the various scripts can be well described. However, dealing with some phenomena of such sort of processing is rather tricky in classical Makefiles. |
| |
| <code>.MDRULE | <code>.MDRULE |
| .md.rul mst.conll < blind.conll mst | .md.rul: mst.conll < blind.conll mst |
| @echo Run the parser here. | @echo Run the parser here. |
| </code> | </code> |
| |
| * A MD-rule ends obligatorily with an empty line (even at the end of the file). | * A MD-rule ends obligatorily with an empty line (even at the end of the file). |
| * MD-make will generate many normal rules from the multidimensional rule. In these generated rules, all combinations of all values in all affected dimensions will appear. As these rules are not templatic any more, we don't have to fear that gnu make will encounter cyclic dependencies or other problems. | * MD-make will generate many normal rules from the multidimensional rule. In these generated rules, all combinations of all values in all affected dimensions will appear. As these rules are not templatic any more, we don't have to fear that gnu make will encounter cyclic dependencies or other problems. For instance, the above multidimensional rule yields the following normal rules, among others: |
| | |
| | <code>cs/dtest-pre1.mst.conll: cs/dtest-pre1.blind.conll cs/dtest-pre1.mst |
| | @echo Run the parser here. |
| | cs/dtest-pre2.mst.conll: cs/dtest-pre2.blind.conll cs/dtest-pre2.mst |
| | @echo Run the parser here. |
| | cs/etest-pre1.mst.conll: cs/etest-pre1.blind.conll cs/etest-pre1.mst |
| | @echo Run the parser here. |
| | cs/etest-pre2.mst.conll: cs/etest-pre2.blind.conll cs/etest-pre2.mst |
| | @echo Run the parser here. |
| | ... |
| | en/etest-pre2.mst.conll: en/etest-pre2.blind.conll en/etest-pre2.mst |
| | @echo Run the parser here. |
| | </code> |
| * The following parameters can be supplied, too: | * The following parameters can be supplied, too: |
| * The ''.md.for'' parameter specifies in what dimensions the target file exists. (The other dimensions will not appear in the file name.) If there is no parameter ''.md.for'' the rule is generated for all known dimensions except the last one (''STATES'' in our case). | * The ''.md.for'' parameter specifies in what dimensions the target file exists. (The other dimensions will not appear in the file name.) If there is no parameter ''.md.for'' the rule is generated for all known dimensions except the last one (''STATES'' in our case). |
| | * The ''.md.fix'' parameter contains values that are fixed in this rule, i.e. the rule is not generated for other values of the same dimension. So far it is not allowed to include more values in one dimension (although in theory we may want to use it to constrain partial generation). |
| | * If ''.md.fix'' contains a dimension that at the same time appears in ''.md.for'', it means that the target type exists in this dimension, has its value in its name/path but this particular rule generates this file only for one value of that dimension. |
| | * If ''.md.fix'' contains a dimension that does not appear in ''.md.for'', it means that the target file type does not know this dimension and does not have it in its name/path but one of the source files knows the dimension and needs to know what value ve have on mind. We can figure out from the rules generating the source files what dimensions they exist in. |
| | * If a source file requires a dimension not contained in the target file, and the dimension is not fixed, the rule will be generated for all values of this dimension. This means that there will be several competing rules for the same target file. |
| | * Example: The rule defined above is intended for parsing, not training, so it should only operate on test conll files. We thus freeze the TRAINTEST dimension on the test value. |
| | |
| | <code>.MDRULE |
| | .md.rul: mst.conll < blind.conll mst |
| | .md.for: LANGUAGES DE PREPROCESSINGS |
| | .md.fix: test |
| | @echo Run the parser here. |
| | </code> |
| | |
| * What are the constraints for the values in the respective dimensions. (Standard way is the ''.md.if'' directive but we would like to be able to constrain the ''.STATES'' dimension (or the last dimension in the list) directly in the rule. | * What are the constraints for the values in the respective dimensions. (Standard way is the ''.md.if'' directive but we would like to be able to constrain the ''.STATES'' dimension (or the last dimension in the list) directly in the rule. |
| * New variables ''$(*1)'' (or other number instead of 1, for n-th dependency) are available within the commands in the rule. MD-make finds the rule that creates this dependency, uses it to determine the set of dimensions of the dependency, constructs the name of the file and replaces the variable by the file name. MD-make leaves intact ''$<'' and ''$^'' that will still work in the generated makefile. However, don't use ''$*'' that does not make sense in MD-rules (unlike in normal pattern rules). | * New variables ''$(*1)'' (or other number instead of 1, for n-th dependency) are available within the commands in the rule. MD-make finds the rule that creates this dependency, uses it to determine the set of dimensions of the dependency, constructs the name of the file and replaces the variable by the file name. MD-make leaves intact ''$<'' and ''$^'' that will still work in the generated makefile. However, don't use ''$*'' that does not make sense in MD-rules (unlike in normal pattern rules). |
| * The ''.md.fix'' parameter contains values that are fixed in this rule, i.e. the rule is not generated for other values of the same dimension. So far it is not allowed to include more values in one dimension (although in theory we may want to use it to constrain partial generation). | |
| * If ''.md.fix'' contains a dimension that at the same time appears in ''.md.for'', it means that the target type exists in this dimension, has its value in its name/path but this particular rule generates this file only for one value of that dimension. | |
| * If ''.md.fix'' contains a dimension that does not appear in ''.md.for'', it means that the target file type does not know this dimension and does not have it in its name/path but one of the source files knows the dimension and needs to know what value ve have on mind. We can figure out from the rules generating the source files what dimensions they exist in. | |
| * If a source file requires a dimension not contained in the target file, and the dimension is not fixed, the rule will be generated for all values of this dimension. This means that there will be several competing rules for the same target file. | |
| * ''.md.del'' removes dimensions from ''.md.for'' (handy if ''.md.for'' is not explicitly stated and contains all dimensions by default) | * ''.md.del'' removes dimensions from ''.md.for'' (handy if ''.md.for'' is not explicitly stated and contains all dimensions by default) |
| * ''.md.fxd'' combines ''.md.fix'' and ''.md.del''. Contains values, not dimensions (like ''.md.fix'' and unlike ''.md.del'') | * ''.md.fxd'' combines ''.md.fix'' and ''.md.del''. Contains values, not dimensions (like ''.md.fix'' and unlike ''.md.del'') |
| |
| <code>.MDRULE | <code>.MDRULE |
| .md.rul mst.conll < blind.conll mst | .md.rul: mst.conll < blind.conll mst |
| .md.dep $(TOOLDIR)/runmst.pl | .md.dep: $(TOOLDIR)/runmst.pl |
| .md.for: LANGUAGES DE PREPROCESSINGS | .md.for: LANGUAGES DE PREPROCESSINGS |
| .md.fix: test | .md.fix: test |
