Motivation
The current ACS Makefile has been improved over the years allowing a per-module parallel build, however it has always been an issue to build the whole system efficiently. There have been improvements over the years (like building subsystems in parallel), but we're far from a lean process when small changes are made to some module. It actually means to build everything again, leading to an ~8 hours compilation process.
There are at least 3 main areas where we aim to improve:
- Dependencies Calculation: This helps in the creation of patches, without missing updating libraries or dependent code, because we didn't force to build some dependencies we were not aware of. It could also be used at some point to create diagrams automatically, which may help to estimate the impact of a new change in the code. It is also a requirement for continuous integration.
- Inclusive Makefile: Currently, we make use of a recursive strategy to build all the subsystems, but the problem is that this isolates the dependencies calculations to single modules. The idea is to make use of an inclusive makefile, which goes through the subsystems trees, parsing the makefiles, leading to a single process with the information from all modules.
- Continuous integration: This would allow to considerably lower the building times, as only the change and its dependencies would need to be recompiled. This is very useful to maintain a build always compiling, and one could also trigger testing jobs related with the changes.
Description
For transition purposes, the idea is for both the current and the new Makefile to coexist. In the modules, this means that there will be both a Makefile (current) and Makefile.mk + module.mk (new). In the groups, there will also be a Makefile (current) and Makefile.mk + group.mk (new).
- Makefile.mk: This is just a placeholder that allows building "from this point on..." to mimic what the current Makefile does. Once you execute one of these makefiles (make -f Makefile.mk ...), all the rest of Makefile.mk files in the tree will be ignored and this will be considered the top directory
- module.mk: This is the equivalent to the old modules Makefile, but is not possible to execute it directly, it has to be done through the Makefile.mk
- It has all the definitions from the previous Makefile that could be reused, but adds a bit more (specially for dependencies calculations)
- It also has changed how all/clean/install/etc. rules are defined
- group.mk: This is the equivalent to the old group Makefile, but is not possible to execute it directly, it has to be done through the Makefile.mk
- It has the list of modules to be considered
- It abstracts the logic to iterate between the different modules, and makes use of the inclusive strategy instead of the recursive one
- It also has changed how all/clean/install/etc. rules are defined
The idea is to minimize the changes required on each module, so at this point I would like to create a script that generates the Makefile.mk files and creates a template module.mk from the existing Makefile file, so developers time fixing this is as low as possible. That said, there would be need to do manual changes, as paths are now considered absolute instead of relative to module's path (There are helpers to get the absolute path in a generic way).
The main Makefile in this definition is the InclusiveMakefile.mk file. This Makefile is supposed to get to a point which is agnostic of ACS and the ALMASW (it is not at this point). There's a variable MAKEID, which points to the specific implementation (In this case 'acs'). With this information it includes several specific files:
- $(MAKEID)MakefileConfig: General configuration variables and macros required for the specific MAKEID
- $(MAKEID)MakefileProfile.$(DIST).mk: Specific changes required for a given distribution
- group.mk: As explained before
- module.mk: As explained before
Core execution:
- When "make -f Makefile.mk ..." command is issued on a module, the InclusiveMakefile.mk file gets included, and the 'genModule' macro gets executed
- genModule:
- Defines helper variables (MODRULE, MODDEP and MODPATH) to be used by the module itself for rule definition and absolute path usage
- Stores the repeated module variables inside specific unique ones (for example LIBRARIES gets stored in <MOD>_LIBRARIES) and cleans the generic ones (so they're free for the next module)
- calls the makeModule macro
- makeModule:
- Goes through all the modules variables (LIBRARIES, EXECUTABLES, JARFILES, etc.) and issues specific macros for each case (makeLibraries, makeExecutables, makePyModules, makeJarFiles, etc.)
- Defines generic rules that don't depend on other details (object compilation, header dependencies, mkdir, etc.)
- I would not go into detail of each existing macro for each case at this point, but that is definitely one of the most important pieces of code
- When "make -f Makefile.mk ..." command is issued on a group, the InclusiveMakefile.mk file gets included, and the 'genGroup' macro gets executed
- genGroup:
- Defines helper variables (GRPRULE, GRPDEP) to be used by the module itself for rule definition and absolute path usage
- Stores the repeated group variables inside specific unique ones (GROUPS and MODULES gets stored in <GRP>_GROUPS and <GRP>_MODULES) and cleans the generic ones (so they're free for the next module)
- calls the makeGroup macro
- makeGroup:
- Iterates in the list of modules calling the genModules macro
- Iterates in the list of groups calling the genGroups macro
- Defines generic rules (<GRP>, clean_<GRP>, install_<GRP>, etc.)
- genGroups and genModules differ from genGroup and genModule in the sense that they do have a parent
Variables
General Variables:
- ...
Module Variables:
- LIBRARIES
- LIBRARIES_L
- EXECUTABLES
- EXECUTABLES_L
- SCRIPTS
- SCRIPTS_L
- PY_SCRIPTS
- PY_SCRIPTS_L
- PY_MODULES
- PY_MODULES_L
- PY_PACKAGES
- PY_PACKAGES_L
- JARFILES
- JARFILE_L
- COMPONENTS_JARFILES
- COMPONENTS_JARFILES_L
- IDL_FILES
- IDL_FILES_L
- TCL_SCRIPTS
- TCL_SCRIPTS_L
- INCLUDES
- CONFIGS
- INSTALL_FILES
- XSDBIND
- XSDBIND_L
- ACSERRDEF
- ACSERRDEF_L
- ACSLOGTSDEF
- ACSLOGTSDEF_L
- CDB_SCHEMAS
- TAO_IDLFLAGS
- CLASSPATH
- DEBUG
General Sub-Variables:
- <TYPE>_DEPS
Libraries Sub-Variables:
- <LIB>_OBJECTS
- <LIB>_LIBS
- <LIB>_CFLAGS
- <LIB>_LDFLAGS
Executable Sub-Variables:
- <EXE>_OBJECTS
- <EXE>_LIBS
- <EXE>_CFLAGS
- <EXE>_LDFLAGS
Object Sub-Variables:
- <OBJ>_CFLAGS
JAR Sub-Variables:
- <JAR>_DIRS
- <JAR>_EXTRAS
- <JAR>_JARS
