Differences
This shows you the differences between two versions of the page.
| Both sides previous revision Previous revision Next revision | Previous revision | ||
|
user:zeman:mdmake [2010/11/05 15:28] zeman |
user:zeman:mdmake [2023/04/21 18:17] (current) zeman Now versioned at Github. |
||
|---|---|---|---|
| Line 1: | Line 1: | ||
| ====== mdmake ====== | ====== mdmake ====== | ||
| + | |||
| + | [[https:// | ||
| Imagine you need to apply the same sequence of tools to a set of data files, and possibly want to be able to repeat the experiment later, i.e. sometime in future you will want recall how precisely the processing would be invoked. One example is a shared task in processing of similarly formatted data in many languages. One may want to use [[http:// | Imagine you need to apply the same sequence of tools to a set of data files, and possibly want to be able to repeat the experiment later, i.e. sometime in future you will want recall how precisely the processing would be invoked. One example is a shared task in processing of similarly formatted data in many languages. One may want to use [[http:// | ||
| Line 11: | Line 13: | ||
| * A MD-makefile ('' | * A MD-makefile ('' | ||
| * Enumerate variables that contain values of respective dimensions. At the same time tell how to combine them into file names (paths). (The spaces will be deleted, their purpose here is to show what delimiter should be omitted if a dimension is omitted. Permitted delimiters are slash, hyphen and period.) | * Enumerate variables that contain values of respective dimensions. At the same time tell how to combine them into file names (paths). (The spaces will be deleted, their purpose here is to show what delimiter should be omitted if a dimension is omitted. Permitted delimiters are slash, hyphen and period.) | ||
| + | |||
| < | < | ||
| + | |||
| * The delimiters are not mandatory but MD-make checks whether missing delimiters do not cause ambiguities (e.g. if LANGUAGES = hi him, DOMAINS = mix ix, then .MDIMS: LANGUAGES DOMAINS would cause problems). | * The delimiters are not mandatory but MD-make checks whether missing delimiters do not cause ambiguities (e.g. if LANGUAGES = hi him, DOMAINS = mix ix, then .MDIMS: LANGUAGES DOMAINS would cause problems). | ||
| * The last dimension in the list of dimensions is special. It need not be named STATES and it need not be delimited by a period (although it is recommended - in some operating systems it is desirable that the file name extension defines the type of the contents), nevertheless the value of this dimension is considered the type of the file. Among others, the file type defines, in what dimensions the files of this type exist. MD-make gets that information from the rule that generates files of this type as its goal. For every type there must be at least one such rule. Theoretically there can be more if e.g. we want to perform different actions for different languages. In that case all such rules must lead to the same list of dimensions of the goal. However, they are not required to cover together all values of all these dimensions. | * The last dimension in the list of dimensions is special. It need not be named STATES and it need not be delimited by a period (although it is recommended - in some operating systems it is desirable that the file name extension defines the type of the contents), nevertheless the value of this dimension is considered the type of the file. Among others, the file type defines, in what dimensions the files of this type exist. MD-make gets that information from the rule that generates files of this type as its goal. For every type there must be at least one such rule. Theoretically there can be more if e.g. we want to perform different actions for different languages. In that case all such rules must lead to the same list of dimensions of the goal. However, they are not required to cover together all values of all these dimensions. | ||
| * The respective variables with values of the respective dimensions must be normal variables containing only a list of words separated by spaces. MD-make will not search them for references to other variables or macros. If it encounters a dollar sign in these variables, it will throw an exception and terminate. These variables will be visible in the generated makefile as well. | * The respective variables with values of the respective dimensions must be normal variables containing only a list of words separated by spaces. MD-make will not search them for references to other variables or macros. If it encounters a dollar sign in these variables, it will throw an exception and terminate. These variables will be visible in the generated makefile as well. | ||
| * No value in no dimension can be identical with any other value of any dimension. In other words, a value uniquely identifies its dimension. (This helps prevent ambiguities in file names that do not contain all dimensions.) | * No value in no dimension can be identical with any other value of any dimension. In other words, a value uniquely identifies its dimension. (This helps prevent ambiguities in file names that do not contain all dimensions.) | ||
| - | * There are special keywords to mark a multidimensional pattern rule. The following parameters can be supplied, too: | ||
| - | * In what dimensions the target file exists. (The other dimensions will not appear in the file name.) | ||
| - | * What are the constraints for the values in the respective dimensions. (Standard way is the '' | ||
| - | * MD-make will generate many normal rules from the multidimensional rule. In these generated rules, all combinations of all values in all affected dimensions will appear. As these rules are not templatic any more, we don't have to fear that gnu make will encounter cyclic dependencies or other problems. | ||
| - | * Uvnitř příkazů lze použít nové proměnné '' | + | <code> |
| - | * MD pravidlo končí povinně prázdným řádkem (dokonce i na konci souboru). | + | DE = d e |
| - | * Není-li uveden parametr '' | + | TRAINTEST = train test |
| - | * Parametr | + | PREPROCESSINGS = pre1 pre2 |
| - | * Jestliže | + | STATES = mst blind.conll mst.conll</ |
| - | * Jestliže | + | |
| - | * Jestliže některý zdrojový soubor vyžaduje rozměr, který cílový soubor neobsahuje, a tento rozměr není zafixován, pravidlo se rozgeneruje i pro všechny hodnoty tohoto rozměru. Bude pak existovat několik konkurenčních pravidel, která vytvářejí tentýž cílový soubor. | + | |
| - | * '' | + | * '' |
| - | * '' | + | |
| - | * Odkaz na hodnotu rozměru z& | + | |
| < | < | ||
| - | .md.rul mst.conll < blind.conll mst | + | .md.rul: mst.conll < blind.conll mst |
| - | .md.dep $(TOOLDIR)/ | + | @echo Run the parser here. |
| + | </ | ||
| + | |||
| + | * A MD-rule ends obligatorily with an empty line (even at the end of the file). | ||
| + | * MD-make will generate many normal rules from the multidimensional rule. In these generated rules, all combinations of all values in all affected dimensions will appear. As these rules are not templatic any more, we don't have to fear that gnu make will encounter cyclic dependencies or other problems. For instance, the above multidimensional rule yields the following normal rules, among others: | ||
| + | |||
| + | < | ||
| + | @echo Run the parser here. | ||
| + | cs/ | ||
| + | @echo Run the parser here. | ||
| + | cs/ | ||
| + | @echo Run the parser here. | ||
| + | cs/ | ||
| + | @echo Run the parser here. | ||
| + | ... | ||
| + | en/ | ||
| + | @echo Run the parser here. | ||
| + | </ | ||
| + | |||
| + | * The following parameters can be supplied, too: | ||
| + | * The '' | ||
| + | * The '' | ||
| + | * If '' | ||
| + | * If '' | ||
| + | * If a source file requires a dimension not contained in the target file, and the dimension is not fixed, the rule will be generated for all values of this dimension. This means that there will be several competing rules for the same target file. | ||
| + | * Example: The rule defined above is intended for parsing, not training, so it should only operate on test conll files. We thus freeze the TRAINTEST dimension on the test value. | ||
| + | |||
| + | < | ||
| + | .md.rul: mst.conll < blind.conll mst | ||
| + | .md.for: LANGUAGES DE PREPROCESSINGS | ||
| + | .md.fix: test | ||
| + | @echo Run the parser here. | ||
| + | </ | ||
| + | |||
| + | * What are the constraints for the values in the respective dimensions. (Standard way is the '' | ||
| + | * New variables '' | ||
| + | * '' | ||
| + | * '' | ||
| + | * If a command in a rule refers a dimension (e.g. '' | ||
| + | |||
| + | < | ||
| + | .md.rul: | ||
| + | .md.dep: $(TOOLDIR)/ | ||
| .md.for: LANGUAGES DE PREPROCESSINGS | .md.for: LANGUAGES DE PREPROCESSINGS | ||
| .md.fix: test | .md.fix: test | ||
| Line 40: | Line 80: | ||
| $(TOOLDIR)/ | $(TOOLDIR)/ | ||
| - | * Je možné definovat vstupní soubory. Ty typicky leží úplně v& | + | * It is possible to define input files. These are typically in a completely different folder or they names are different so that they are not confused with the files named according to dimension values and they are not in danger of being removed by make. We can describe their properties in various dimensions if we create a normal rule where the input file is a dependency (source) and the goal is a file named by dimension values. We write '' |
| - | * Vygenerovaný | + | * The generated |
| - | * V& | + | * Collect names of all generated target files during generating of the multidimensional rules. Create a hash for every target file where the key is the value in a dimension and the value under that key is non-zero if the value of the dimension is contained in the name of the file. At the end of the makefile there can be a rule '' |
| < | < | ||
| - | se přepíše jako | + | would be rewritten as |
| < | < | ||
| - | all_d_hi_conll: | + | all_d_hi_conll: |
| + | |||
| + | ===== Download ===== | ||
| + | |||
| + | Copyright © 2009 Daniel Zeman | ||
| + | |||
| + | All software supplied with this package is released under the GNU | ||
| + | General Public License. | ||
| + | redistribute it and/or modify it under the terms of the GNU General | ||
| + | Public License as published by the Free Software Foundation; either | ||
| + | version 2, or (at your option) any later version. | ||
| + | |||
| + | This program is distributed in the hope that it will be useful, | ||
| + | but WITHOUT ANY WARRANTY; without even the implied warranty of | ||
| + | MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. | ||
| + | GNU General Public License (below or at http:// | ||
| + | gpl.html) for more details. | ||
| + | |||
| + | {{: | ||
| + | |||
| + | ===== Acknowledgements ===== | ||
| + | |||
| + | This research has been supported by the grant of the Czech Ministry of Education no. MSM0021620838. | ||