[ Skip to the content ]

Institute of Formal and Applied Linguistics Wiki

[ Back to the navigation ]

You are here: start » user » zeman » mdmake

Differences

This shows you the differences between two versions of the page.

Link to this comparison view

Both sides previous revision Previous revision
Next revision 		Previous revision
user:zeman:mdmake [2023/04/21 16:09]
zeman [Makefile] 		user:zeman:mdmake [2023/04/21 18:17] (current)
zeman Now versioned at Github.
Line 1: Line 1:
 ====== mdmake ====== ====== mdmake ======
 +
 +[[https://github.com/dan-zeman/mdmake]]
  
 Imagine you need to apply the same sequence of tools to a set of data files, and possibly want to be able to repeat the experiment later, i.e. sometime in future you will want recall how precisely the processing would be invoked. One example is a shared task in processing of similarly formatted data in many languages. One may want to use [[http://www.gnu.org/software/make/manual/make.html|make]] and Makefiles where the sequence of application of the various scripts can be well described. However, dealing with some phenomena of such sort of processing is rather tricky in classical Makefiles. Imagine you need to apply the same sequence of tools to a set of data files, and possibly want to be able to repeat the experiment later, i.e. sometime in future you will want recall how precisely the processing would be invoked. One example is a shared task in processing of similarly formatted data in many languages. One may want to use [[http://www.gnu.org/software/make/manual/make.html|make]] and Makefiles where the sequence of application of the various scripts can be well described. However, dealing with some phenomena of such sort of processing is rather tricky in classical Makefiles.
Line 11: Line 13:
   * A MD-makefile (''makefile.mdm'') may contain all syntactic constructions that a normal makefile can contain. The constructions will be copied to a generated makefile and normal gnu make will be responsible for their interpretation. It has to be borne in mind however that they will be processed //after// the makefile will be generated. So for example if we include nested makefiles, these must be normal makefiles, not MD-makefiles.   * A MD-makefile (''makefile.mdm'') may contain all syntactic constructions that a normal makefile can contain. The constructions will be copied to a generated makefile and normal gnu make will be responsible for their interpretation. It has to be borne in mind however that they will be processed //after// the makefile will be generated. So for example if we include nested makefiles, these must be normal makefiles, not MD-makefiles.
   * Enumerate variables that contain values of respective dimensions. At the same time tell how to combine them into file names (paths). (The spaces will be deleted, their purpose here is to show what delimiter should be omitted if a dimension is omitted. Permitted delimiters are slash, hyphen and period.)   * Enumerate variables that contain values of respective dimensions. At the same time tell how to combine them into file names (paths). (The spaces will be deleted, their purpose here is to show what delimiter should be omitted if a dimension is omitted. Permitted delimiters are slash, hyphen and period.)
 +
 <code>.MDIMS: LANGUAGES/ DE TRAINTEST -PREPROCESSINGS .STATES</code> <code>.MDIMS: LANGUAGES/ DE TRAINTEST -PREPROCESSINGS .STATES</code>
 +
   * The delimiters are not mandatory but MD-make checks whether missing delimiters do not cause ambiguities (e.g. if LANGUAGES = hi him, DOMAINS = mix ix, then .MDIMS: LANGUAGES DOMAINS would cause problems).   * The delimiters are not mandatory but MD-make checks whether missing delimiters do not cause ambiguities (e.g. if LANGUAGES = hi him, DOMAINS = mix ix, then .MDIMS: LANGUAGES DOMAINS would cause problems).
   * The last dimension in the list of dimensions is special. It need not be named STATES and it need not be delimited by a period (although it is recommended - in some operating systems it is desirable that the file name extension defines the type of the contents), nevertheless the value of this dimension is considered the type of the file. Among others, the file type defines, in what dimensions the files of this type exist. MD-make gets that information from the rule that generates files of this type as its goal. For every type there must be at least one such rule. Theoretically there can be more if e.g. we want to perform different actions for different languages. In that case all such rules must lead to the same list of dimensions of the goal. However, they are not required to cover together all values of all these dimensions.   * The last dimension in the list of dimensions is special. It need not be named STATES and it need not be delimited by a period (although it is recommended - in some operating systems it is desirable that the file name extension defines the type of the contents), nevertheless the value of this dimension is considered the type of the file. Among others, the file type defines, in what dimensions the files of this type exist. MD-make gets that information from the rule that generates files of this type as its goal. For every type there must be at least one such rule. Theoretically there can be more if e.g. we want to perform different actions for different languages. In that case all such rules must lead to the same list of dimensions of the goal. However, they are not required to cover together all values of all these dimensions.
   * The respective variables with values of the respective dimensions must be normal variables containing only a list of words separated by spaces. MD-make will not search them for references to other variables or macros. If it encounters a dollar sign in these variables, it will throw an exception and terminate. These variables will be visible in the generated makefile as well.   * The respective variables with values of the respective dimensions must be normal variables containing only a list of words separated by spaces. MD-make will not search them for references to other variables or macros. If it encounters a dollar sign in these variables, it will throw an exception and terminate. These variables will be visible in the generated makefile as well.
   * No value in no dimension can be identical with any other value of any dimension. In other words, a value uniquely identifies its dimension. (This helps prevent ambiguities in file names that do not contain all dimensions.)   * No value in no dimension can be identical with any other value of any dimension. In other words, a value uniquely identifies its dimension. (This helps prevent ambiguities in file names that do not contain all dimensions.)
 +
 +<code>LANGUAGES = cs en
 +DE = d e
 +TRAINTEST = train test
 +PREPROCESSINGS = pre1 pre2
 +STATES = mst blind.conll mst.conll</code>
 +
   * There are special keywords to mark a multidimensional rule: ''.MDRULE'', ''.MDALL'', ''.MDIN''   * There are special keywords to mark a multidimensional rule: ''.MDRULE'', ''.MDALL'', ''.MDIN''
   * ''.MDRULE'' introduces the main type of pattern rule. It has the parameter ''.md.rul'', which specifies the target and source states / file types (values of the last dimension). For example, we may state that the target file type ''mst.conll'' (a file parsed by the MST parser) needs source files of two types: ''blind.conll'' (the text to be parsed) and ''mst'' (the trained model for the MST parser).   * ''.MDRULE'' introduces the main type of pattern rule. It has the parameter ''.md.rul'', which specifies the target and source states / file types (values of the last dimension). For example, we may state that the target file type ''mst.conll'' (a file parsed by the MST parser) needs source files of two types: ''blind.conll'' (the text to be parsed) and ''mst'' (the trained model for the MST parser).
  
 <code>.MDRULE <code>.MDRULE
-.md.rul mst.conll < blind.conll mst</code>+.md.rul: mst.conll < blind.conll mst 
 +        @echo Run the parser here. 
 +</code> 
 + 
 +  * A MD-rule ends obligatorily with an empty line (even at the end of the file). 
 +  * MD-make will generate many normal rules from the multidimensional rule. In these generated rules, all combinations of all values in all affected dimensions will appear. As these rules are not templatic any more, we don't have to fear that gnu make will encounter cyclic dependencies or other problems. For instance, the above multidimensional rule yields the following normal rules, among others: 
 + 
 +<code>cs/dtest-pre1.mst.conll: cs/dtest-pre1.blind.conll cs/dtest-pre1.mst 
 +        @echo Run the parser here. 
 +cs/dtest-pre2.mst.conll: cs/dtest-pre2.blind.conll cs/dtest-pre2.mst 
 +        @echo Run the parser here. 
 +cs/etest-pre1.mst.conll: cs/etest-pre1.blind.conll cs/etest-pre1.mst 
 +        @echo Run the parser here. 
 +cs/etest-pre2.mst.conll: cs/etest-pre2.blind.conll cs/etest-pre2.mst 
 +        @echo Run the parser here. 
 +... 
 +en/etest-pre2.mst.conll: en/etest-pre2.blind.conll en/etest-pre2.mst 
 +        @echo Run the parser here. 
 +</code>
  
   * The following parameters can be supplied, too:   * The following parameters can be supplied, too:
-    * The ''.md.for'' parameter specifies in what dimensions the target file exists. (The other dimensions will not appear in the file name.)+    * The ''.md.for'' parameter specifies in what dimensions the target file exists. (The other dimensions will not appear in the file name.) If there is no parameter ''.md.for'' the rule is generated for all known dimensions except the last one (''STATES'' in our case). 
 +    * The ''.md.fix'' parameter contains values that are fixed in this rule, i.e. the rule is not generated for other values of the same dimension. So far it is not allowed to include more values in one dimension (although in theory we may want to use it to constrain partial generation). 
 +      * If ''.md.fix'' contains a dimension that at the same time appears in ''.md.for'', it means that the target type exists in this dimension, has its value in its name/path but this particular rule generates this file only for one value of that dimension. 
 +      * If ''.md.fix'' contains a dimension that does not appear in ''.md.for'', it means that the target file type does not know this dimension and does not have it in its name/path but one of the source files knows the dimension and needs to know what value ve have on mind. We can figure out from the rules generating the source files what dimensions they exist in. 
 +      * If a source file requires a dimension not contained in the target file, and the dimension is not fixed, the rule will be generated for all values of this dimension. This means that there will be several competing rules for the same target file. 
 +      * Example: The rule defined above is intended for parsing, not training, so it should only operate on test conll files. We thus freeze the TRAINTEST dimension on the test value. 
 + 
 +<code>.MDRULE 
 +.md.rul: mst.conll < blind.conll mst 
 +.md.for: LANGUAGES DE PREPROCESSINGS 
 +.md.fix: test 
 +        @echo Run the parser here. 
 +</code> 
     * What are the constraints for the values in the respective dimensions. (Standard way is the ''.md.if'' directive but we would like to be able to constrain the ''.STATES'' dimension (or the last dimension in the list) directly in the rule.     * What are the constraints for the values in the respective dimensions. (Standard way is the ''.md.if'' directive but we would like to be able to constrain the ''.STATES'' dimension (or the last dimension in the list) directly in the rule.
-    * MD-make will generate many normal rules from the multidimensional rule. In these generated rules, all combinations of all values in all affected dimensions will appear. As these rules are not templatic any more, we don't have to fear that gnu make will encounter cyclic dependencies or other problems. 
     * New variables ''$(*1)'' (or other number instead of 1, for n-th dependency) are available within the commands in the rule. MD-make finds the rule that creates this dependency, uses it to determine the set of dimensions of the dependency, constructs the name of the file and replaces the variable by the file name. MD-make leaves intact ''$<'' and ''$^'' that will still work in the generated makefile. However, don't use ''$*'' that does not make sense in MD-rules (unlike in normal pattern rules).     * New variables ''$(*1)'' (or other number instead of 1, for n-th dependency) are available within the commands in the rule. MD-make finds the rule that creates this dependency, uses it to determine the set of dimensions of the dependency, constructs the name of the file and replaces the variable by the file name. MD-make leaves intact ''$<'' and ''$^'' that will still work in the generated makefile. However, don't use ''$*'' that does not make sense in MD-rules (unlike in normal pattern rules).
-    * A MD-rule ends obligatorily with an empty line (even at the end of the file). 
-    * If there is no parameter ''.md.for'' the rule is generated for all known dimensions except the last one (''STATES'' in our case). 
-    * The ''.md.fix'' parameter contains values that are fixed in this rule, i.e. the rule is not generated for other values of the same dimension. So far it is not allowed to include more values in one dimension (although in theory we may want to use it to constrain partial generation). 
-    * If ''.md.fix'' contains a dimension that at the same time appears in ''.md.for'', it means that the target type exists in this dimension, has its value in its name/path but this particular rule generates this file only for one value of that dimension. 
-    * If ''.md.fix'' contains a dimension that does not appear in ''.md.for'', it means that the target file type does not know this dimension and does not have it in its name/path but one of the source files knows the dimension and needs to know what value ve have on mind. We can figure out from the rules generating the source files what dimensions they exist in. 
-    * If a source file requires a dimension not contained in the target file, and the dimension is not fixed, the rule will be generated for all values of this dimension. This means that there will be several competing rules for the same target file. 
     * ''.md.del'' removes dimensions from ''.md.for'' (handy if ''.md.for'' is not explicitly stated and contains all dimensions by default)     * ''.md.del'' removes dimensions from ''.md.for'' (handy if ''.md.for'' is not explicitly stated and contains all dimensions by default)
     * ''.md.fxd'' combines ''.md.fix'' and ''.md.del''. Contains values, not dimensions (like ''.md.fix'' and unlike ''.md.del'')     * ''.md.fxd'' combines ''.md.fix'' and ''.md.del''. Contains values, not dimensions (like ''.md.fix'' and unlike ''.md.del'')
Line 38: Line 73:
  
 <code>.MDRULE <code>.MDRULE
-.md.rul mst.conll < blind.conll mst +.md.rul: mst.conll < blind.conll mst 
-.md.dep $(TOOLDIR)/runmst.pl+.md.dep: $(TOOLDIR)/runmst.pl
 .md.for: LANGUAGES DE PREPROCESSINGS .md.for: LANGUAGES DE PREPROCESSINGS
 .md.fix: test .md.fix: test

[ Back to the navigation ] [ Back to the content ]