|
Next revision
|
Previous revision
Next revision
Both sides next revision
|
user:zeman:mdmake [2010/11/05 14:24]
zeman vytvořeno |
user:zeman:mdmake [2010/11/05 15:47]
zeman |
| Imagine you need to apply the same sequence of tools to a set of data files, and possibly want to be able to repeat the experiment later, i.e. sometime in future you will want recall how precisely the processing would be invoked. One example is a shared task in processing of similarly formatted data in many languages. One may want to use [[http://www.gnu.org/software/make/manual/make.html|make]] and Makefiles where the sequence of application of the various scripts can be well described. However, dealing with some phenomena of such sort of processing is rather tricky in classical Makefiles. | Imagine you need to apply the same sequence of tools to a set of data files, and possibly want to be able to repeat the experiment later, i.e. sometime in future you will want recall how precisely the processing would be invoked. One example is a shared task in processing of similarly formatted data in many languages. One may want to use [[http://www.gnu.org/software/make/manual/make.html|make]] and Makefiles where the sequence of application of the various scripts can be well described. However, dealing with some phenomena of such sort of processing is rather tricky in classical Makefiles. |
| |
| The most prominent phenomenon that is difficult to capture is what I call //multidimensionality// of the data. Every data file undergoes a sequence of processing steps, i.e. it appears in many different states (and intermediate data formats). Some processing tools may have alternative implementations, so you may have the same piece of data in the same stage of processing (e.g. syntactically parsed) but with different processing results (e.g. parsed either by Malt parser, or MST parser). | The most prominent phenomenon that is difficult to capture is what I call //multidimensionality// of the data. Every data file undergoes a sequence of processing steps, i.e. it appears in many different states (and intermediate data formats). Some processing tools may have alternative implementations, so you may have the same piece of data in the same stage of processing (e.g. syntactically parsed) but with different processing results (e.g. parsed either by Malt parser, or MST parser). Besides that, you may be applying the same processing to data in ten different languages, several domains per language, separately to development and evaluation test data etc. All these //dimensions// will probably be somehow reflected in the path to your data files. You probably would want to use pattern (template) rules in your Makefile to describe the same action applied to many files. However, gnu make allows you only one ''%'' (variable) per pattern rule, which makes it rather difficult to define templates in the multidimensional space. This is where **mdmake,** or “multidimensional make” may be useful. |
| | |
| | An older, more detailed discussion of the related problems is described [[rizeni-pokusu-pomoci-makefilu|here]] but it's in Czech. |
| | |
| | ===== Makefile ===== |
| | |
| | * A MD-makefile (''makefile.mdm'') may contain all syntactic constructions that a normal makefile can contain. The constructions will be copied to a generated makefile and normal gnu make will be responsible for their interpretation. It has to be borne in mind however that they will be processed //after// the makefile will be generated. So for example if we include nested makefiles, these must be normal makefiles, not MD-makefiles. |
| | * Enumerate variables that contain values of respective dimensions. At the same time tell how to combine them into file names (paths). (The spaces will be deleted, their purpose here is to show what delimiter should be omitted if a dimension is omitted. Permitted delimiters are slash, hyphen and period.) |
| | <code>.MDIMS: LANGUAGES/ DE TRAINTEST -PREPROCESSINGS .STATES</code> |
| | * The delimiters are not mandatory but MD-make checks whether missing delimiters do not cause ambiguities (e.g. if LANGUAGES = hi him, DOMAINS = mix ix, then .MDIMS: LANGUAGES DOMAINS would cause problems). |
| | * The last dimension in the list of dimensions is special. It need not be named STATES and it need not be delimited by a period (although it is recommended - in some operating systems it is desirable that the file name extension defines the type of the contents), nevertheless the value of this dimension is considered the type of the file. Among others, the file type defines, in what dimensions the files of this type exist. MD-make gets that information from the rule that generates files of this type as its goal. For every type there must be at least one such rule. Theoretically there can be more if e.g. we want to perform different actions for different languages. In that case all such rules must lead to the same list of dimensions of the goal. However, they are not required to cover together all values of all these dimensions. |
| | * The respective variables with values of the respective dimensions must be normal variables containing only a list of words separated by spaces. MD-make will not search them for references to other variables or macros. If it encounters a dollar sign in these variables, it will throw an exception and terminate. These variables will be visible in the generated makefile as well. |
| | * No value in no dimension can be identical with any other value of any dimension. In other words, a value uniquely identifies its dimension. (This helps prevent ambiguities in file names that do not contain all dimensions.) |
| | * There are special keywords to mark a multidimensional pattern rule. The following parameters can be supplied, too: |
| | * In what dimensions the target file exists. (The other dimensions will not appear in the file name.) |
| | * What are the constraints for the values in the respective dimensions. (Standard way is the ''.md.if'' directive but we would like to be able to constrain the ''.STATES'' dimension (or the last dimension in the list) directly in the rule. |
| | * MD-make will generate many normal rules from the multidimensional rule. In these generated rules, all combinations of all values in all affected dimensions will appear. As these rules are not templatic any more, we don't have to fear that gnu make will encounter cyclic dependencies or other problems. |
| | * New variables ''$(*1)'' (or other number instead of 1, for n-th dependency) are available within the commands in the rule. MD-make finds the rule that creates this dependency, uses it to determine the set of dimensions of the dependency, constructs the name of the file and replaces the variable by the file name. MD-make leaves intact ''$<'' and ''$^'' that will still work in the generated makefile. However, don't use ''$*'' that does not make sense in MD-rules (unlike in normal pattern rules). |
| | * A MD-rule ends obligatorily with an empty line (even at the end of the file). |
| | * If there is no parameter ''.md.for'' the rule is generated for all known dimensions except the last one (''STATES'' in our case). |
| | * The ''.md.fix'' parameter contains values that are fixed in this rule, i.e. the rule is not generated for other values of the same dimension. So far it is not allowed to include more values in one dimension (although in theory we may want to use it to constrain partial generation). |
| | * If ''.md.fix'' contains a dimension that at the same time appears in ''.md.for'', it means that the target type exists in this dimension, has its value in its name/path but this particular rule generates this file only for one value of that dimension. |
| | * If ''.md.fix'' contains a dimension that does not appear in ''.md.for'', it means that the target file type does not know this dimension and does not have it in its name/path but one of the source files knows the dimension and needs to know what value ve have on mind. We can figure out from the rules generating the source files what dimensions they exist in. |
| | * If a source file requires a dimension not contained in the target file, and the dimension is not fixed, the rule will be generated for all values of this dimension. This means that there will be several competing rules for the same target file. |
| | * ''.md.del'' removes dimensions from ''.md.for'' (handy if ''.md.for'' is not explicitly stated and contains all dimensions by default) |
| | * ''.md.fxd'' combines ''.md.fix'' and ''.md.del''. Contains values, not dimensions (like ''.md.fix'' and unlike ''.md.del'') |
| | * If a command in a rule refers a dimension (e.g. ''$(*LANGUAGES)'') the reference will be converted to the actual value of the dimension. If different source files have different values of the same dimension within one generated rule the reference will be replaced by the value that the target file takes in this dimension (i.e. by the variable value). Anyway the purpose of such references is to refer to the variable dimensions. Modified values of particular source files are fixed exceptions, we know them in advance and can write them to the command directly, if necessary. |
| | |
| | |
| | <code>.MDRULE |
| | .md.rul mst.conll < blind.conll mst |
| | .md.dep $(TOOLDIR)/runmst.pl |
| | .md.for: LANGUAGES DE PREPROCESSINGS |
| | .md.fix: test |
| | @echo Running MST for language $(*LANGUAGES): |
| | $(TOOLDIR)/runmst.pl -m $(*2) < $< > $@</code> |
| | |
| | * Je možné definovat vstupní soubory. Ty typicky leží úplně v jiné cestě, nebo se alespoň jmenují tak, aby se nepletly se soubory pojmenovanými pomocí hodnot rozměrů, a nehrozilo tudíž jejich smazání makem. Můžeme popsat jejich vlastnosti v jednotlivých rozměrech prostě tak, že vytvoříme obyčejné pravidlo, kde dotyčný vstupní soubor bude jako závislost, zatímco cíl bude soubor pojmenovaný příslušnými hodnotami rozměrů. Před pravidlo připíšeme ''.md.in:''. MD-make pak doplní příkaz pro zkopírování závislosti do cíle (''cp $< $@'') a navíc zkontroluje, že cílový soubor má hodnoty všech rozměrů, které soubor v daném stavu (hodnota posledního rozměru) má mít. |
| | * Vygenerovaný makefile by navíc mohl obsahovat pro každou hodnotu každého rozměru seznam souborů, v nichž je tato hodnota zafixovaná. Např. všechny cílové soubory v jazyce "hi". Kromě proměnné obsahující jména těchto souborů (HIFILES) by vygenerovaný makefile obsahoval cíl, který všechny tyto soubory vyrobí (hi), a cíl, který je smaže (clean_hi). |
| | * V průběhu generování vícerozměrných pravidel si pamatovat seznam všech vygenerovaných cílových souborů. Ke každému cílovému souboru vytvořit hash, jehož klíčem je hodnota libovolného rozměru a hodnota u daného klíče je nenulová, jestliže příslušná hodnota rozměru je v názvu souboru obsažena. Na konci makefilu lze použít pravidlo ''.MDALL'', které vytvoří ''.PHONY'' cíl závisející na všech souborech obsahujících určité hodnoty. Např. |
| | |
| | <code>.MDALL: d hi conll</code> |
| | |
| | se přepíše jako |
| | |
| | <code>.PHONY: all_d_hi_conll |
| | all_d_hi_conll: <seznam všech souborů obsahujících hodnoty "d", "hi" a "conll"></code> |
