mdmake

https://github.com/dan-zeman/mdmake

Imagine you need to apply the same sequence of tools to a set of data files, and possibly want to be able to repeat the experiment later, i.e. sometime in future you will want recall how precisely the processing would be invoked. One example is a shared task in processing of similarly formatted data in many languages. One may want to use make and Makefiles where the sequence of application of the various scripts can be well described. However, dealing with some phenomena of such sort of processing is rather tricky in classical Makefiles.

The most prominent phenomenon that is difficult to capture is what I call multidimensionality of the data. Every data file undergoes a sequence of processing steps, i.e. it appears in many different states (and intermediate data formats). Some processing tools may have alternative implementations, so you may have the same piece of data in the same stage of processing (e.g. syntactically parsed) but with different processing results (e.g. parsed either by Malt parser, or MST parser). Besides that, you may be applying the same processing to data in ten different languages, several domains per language, separately to development and evaluation test data etc. All these dimensions will probably be somehow reflected in the path to your data files. You probably would want to use pattern (template) rules in your Makefile to describe the same action applied to many files. However, gnu make allows you only one % (variable) per pattern rule, which makes it rather difficult to define templates in the multidimensional space. This is where mdmake, or “multidimensional make” may be useful.

An older, more detailed discussion of the related problems is described here but it's in Czech.

• A MD-makefile (makefile.mdm) may contain all syntactic constructions that a normal makefile can contain. The constructions will be copied to a generated makefile and normal gnu make will be responsible for their interpretation. It has to be borne in mind however that they will be processed after the makefile will be generated. So for example if we include nested makefiles, these must be normal makefiles, not MD-makefiles.
• Enumerate variables that contain values of respective dimensions. At the same time tell how to combine them into file names (paths). (The spaces will be deleted, their purpose here is to show what delimiter should be omitted if a dimension is omitted. Permitted delimiters are slash, hyphen and period.)

.MDIMS: LANGUAGES/ DE TRAINTEST -PREPROCESSINGS .STATES

• The delimiters are not mandatory but MD-make checks whether missing delimiters do not cause ambiguities (e.g. if LANGUAGES = hi him, DOMAINS = mix ix, then .MDIMS: LANGUAGES DOMAINS would cause problems).
• The last dimension in the list of dimensions is special. It need not be named STATES and it need not be delimited by a period (although it is recommended - in some operating systems it is desirable that the file name extension defines the type of the contents), nevertheless the value of this dimension is considered the type of the file. Among others, the file type defines, in what dimensions the files of this type exist. MD-make gets that information from the rule that generates files of this type as its goal. For every type there must be at least one such rule. Theoretically there can be more if e.g. we want to perform different actions for different languages. In that case all such rules must lead to the same list of dimensions of the goal. However, they are not required to cover together all values of all these dimensions.
• The respective variables with values of the respective dimensions must be normal variables containing only a list of words separated by spaces. MD-make will not search them for references to other variables or macros. If it encounters a dollar sign in these variables, it will throw an exception and terminate. These variables will be visible in the generated makefile as well.
• No value in no dimension can be identical with any other value of any dimension. In other words, a value uniquely identifies its dimension. (This helps prevent ambiguities in file names that do not contain all dimensions.)

LANGUAGES = cs en
DE = d e
TRAINTEST = train test
PREPROCESSINGS = pre1 pre2
STATES = mst blind.conll mst.conll

• There are special keywords to mark a multidimensional rule: .MDRULE, .MDALL, .MDIN
• .MDRULE introduces the main type of pattern rule. It has the parameter .md.rul, which specifies the target and source states / file types (values of the last dimension). For example, we may state that the target file type mst.conll (a file parsed by the MST parser) needs source files of two types: blind.conll (the text to be parsed) and mst (the trained model for the MST parser).

.MDRULE
.md.rul: mst.conll < blind.conll mst
        @echo Run the parser here.

• A MD-rule ends obligatorily with an empty line (even at the end of the file).
• MD-make will generate many normal rules from the multidimensional rule. In these generated rules, all combinations of all values in all affected dimensions will appear. As these rules are not templatic any more, we don't have to fear that gnu make will encounter cyclic dependencies or other problems. For instance, the above multidimensional rule yields the following normal rules, among others:

cs/dtest-pre1.mst.conll: cs/dtest-pre1.blind.conll cs/dtest-pre1.mst
        @echo Run the parser here.
cs/dtest-pre2.mst.conll: cs/dtest-pre2.blind.conll cs/dtest-pre2.mst
        @echo Run the parser here.
cs/etest-pre1.mst.conll: cs/etest-pre1.blind.conll cs/etest-pre1.mst
        @echo Run the parser here.
cs/etest-pre2.mst.conll: cs/etest-pre2.blind.conll cs/etest-pre2.mst
        @echo Run the parser here.
...
en/etest-pre2.mst.conll: en/etest-pre2.blind.conll en/etest-pre2.mst
        @echo Run the parser here.

• The following parameters can be supplied, too:
  • The .md.for parameter specifies in what dimensions the target file exists. (The other dimensions will not appear in the file name.) If there is no parameter .md.for the rule is generated for all known dimensions except the last one (STATES in our case).
  • The .md.fix parameter contains values that are fixed in this rule, i.e. the rule is not generated for other values of the same dimension. So far it is not allowed to include more values in one dimension (although in theory we may want to use it to constrain partial generation).
    • If .md.fix contains a dimension that at the same time appears in .md.for, it means that the target type exists in this dimension, has its value in its name/path but this particular rule generates this file only for one value of that dimension.
    • If .md.fix contains a dimension that does not appear in .md.for, it means that the target file type does not know this dimension and does not have it in its name/path but one of the source files knows the dimension and needs to know what value ve have on mind. We can figure out from the rules generating the source files what dimensions they exist in.
    • If a source file requires a dimension not contained in the target file, and the dimension is not fixed, the rule will be generated for all values of this dimension. This means that there will be several competing rules for the same target file.
    • Example: The rule defined above is intended for parsing, not training, so it should only operate on test conll files. We thus freeze the TRAINTEST dimension on the test value.

.MDRULE
.md.rul: mst.conll < blind.conll mst
.md.for: LANGUAGES DE PREPROCESSINGS
.md.fix: test
        @echo Run the parser here.

• What are the constraints for the values in the respective dimensions. (Standard way is the .md.if directive but we would like to be able to constrain the .STATES dimension (or the last dimension in the list) directly in the rule.
• New variables $(*1) (or other number instead of 1, for n-th dependency) are available within the commands in the rule. MD-make finds the rule that creates this dependency, uses it to determine the set of dimensions of the dependency, constructs the name of the file and replaces the variable by the file name. MD-make leaves intact $< and $^ that will still work in the generated makefile. However, don't use $* that does not make sense in MD-rules (unlike in normal pattern rules).
• .md.del removes dimensions from .md.for (handy if .md.for is not explicitly stated and contains all dimensions by default)
• .md.fxd combines .md.fix and .md.del. Contains values, not dimensions (like .md.fix and unlike .md.del)
• If a command in a rule refers a dimension (e.g. $(*LANGUAGES)) the reference will be converted to the actual value of the dimension. If different source files have different values of the same dimension within one generated rule the reference will be replaced by the value that the target file takes in this dimension (i.e. by the variable value). Anyway the purpose of such references is to refer to the variable dimensions. Modified values of particular source files are fixed exceptions, we know them in advance and can write them to the command directly, if necessary.

.MDRULE
.md.rul: mst.conll < blind.conll mst
.md.dep: $(TOOLDIR)/runmst.pl
.md.for: LANGUAGES DE PREPROCESSINGS
.md.fix: test
        @echo Running MST for language $(*LANGUAGES):
        $(TOOLDIR)/runmst.pl -m $(*2) < $< > $@

• It is possible to define input files. These are typically in a completely different folder or they names are different so that they are not confused with the files named according to dimension values and they are not in danger of being removed by make. We can describe their properties in various dimensions if we create a normal rule where the input file is a dependency (source) and the goal is a file named by dimension values. We write .md.in: before such rule. MD-make adds a command to copy the source file to the target (cp $< $@) and checks that the target file has values for all dimensions that a file in its state (last dimension's value) ought to have.
• The generated makefile could further contain for each value of each dimension the list of files for which this value is fixed. For instance, all target files in the language “hi”. Besides the variable containing names of such files (HIFILES) there would also be a goal that creates/updates all such files (hi) and a goal that removes them (clean_hi).
• Collect names of all generated target files during generating of the multidimensional rules. Create a hash for every target file where the key is the value in a dimension and the value under that key is non-zero if the value of the dimension is contained in the name of the file. At the end of the makefile there can be a rule .MDALL that creates a .PHONY goal depending on all files containing given values. E.g.

.MDALL: d hi conll

would be rewritten as

.PHONY: all_d_hi_conll
all_d_hi_conll: <list of all files containing values "d", "hi" and "conll">

Copyright © 2009 Daniel Zeman

All software supplied with this package is released under the GNU
General Public License. This program is free software; you can
redistribute it and/or modify it under the terms of the GNU General
Public License as published by the Free Software Foundation; either
version 2, or (at your option) any later version.

This program is distributed in the hope that it will be useful,
but WITHOUT ANY WARRANTY; without even the implied warranty of
MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
GNU General Public License (below or at http://www.gnu.org/licenses/
gpl.html) for more details.

mdmake.zip contains the script mdmake.pl (you need a Perl interpreter to use it), a sample multidimensional makefile and the normal makefile generated from it.

This research has been supported by the grant of the Czech Ministry of Education no. MSM0021620838.