Build System Migration - Meson + Autotools + CMake = PkgConfig!

The Problem

• What happens when you use one build system and try to use libraries with another build system?
  • CMake can't include Meson projects
  • Meson can't include Autotools projects
  • Autotools + Make doesn't generate a compile_commands.json file
  • Custom scripts in build system specific scripting languages get ugly quite quickly

Potential Ways to Fix it

• The Better One of the Bunch - Meson!

  • It can integrate CMake projects
  • It can build Autotools projects as one would do manually with some custom scripting (haven't tried it myself).

• How build systems talk to each other

  • There's no standard way to do it at build time
  • Each build system supports pkgconfig, so the installed artifacts can be consumed through that interface.
  • It is build system agnostic, so as long as a .pc file is generated, we can consume that project.
  • CollaboraOffice uses this approach to consume external libraries (with some Make metaprogramming)

The Migration Process

Data & Code Separation

• Architectural overview
  • The configure & build commands are pretty standard for any build system
  • The only thing which changes is the configure options. We separated them from the build commands.
  • For each exteral project foo, there are 4 config files, config/external/foo/flags.{common,linux,apple,windows}
  • These contain the configure flags for the specific build system.
  • We then have a scripts/make/exports.mk script which looks like this...

 1# scripts/make/exports.mk
 2EXTERNALS = $(shell realpath ./external)
 3EXTERNALS_CONFIG = $(shell realpath ./config/external)
 4INSTALL_DIRECTORY = $(shell realpath ./install)
 5BUILD_DIRECTORY = $(shell realpath ./build)
 6CLANGD_CONFIG = $(shell realpath ./.clangd)
 7PLATFORM = $(shell uname -s)
 8MESON_BUILD_TYPE = debug
 9CMAKE_BUILD_TYPE = Debug
10SHELL := /bin/bash
11
12ifeq ($(PLATFORM),Linux)
13	PLATFORM_CONFIG_EXTENSION = linux
14else ifeq ($(PLATFORM),Darwin)
15	PLATFORM_CONFIG_EXTENSION = apple
16else ifeq ($(PLATFORM),MINGW64*)
17	PLATFORM_CONFIG_EXTENSION = windows
18else
19	PLATFORM_CONFIG_EXTENSION = unknown
20endif

• About exports.mk

  • Here we define some paths and some variables which would be used in the build scripts
  • Platform detection sets the flags file extension which we need to include other than flags.common

• Make metaprogramming

  • The metaprograms define build wrappers in scripts/make/build_wrappers.mk and eval call them from Makefile
  • There's one for Meson, Autotools and CMake
  • For a project foo, we generate foo, foo.configure, foo.build, foo.clangd, foo.install and foo.clean
  • We get the common and platform specific configure flags from the flag files in config/external/foo
  • Target foo depends on all those foo prefixed targets

 1# scripts/make/build_wrappers.mk
 2define build_wrapper_cmake
 3$(1)_CONFIGURE_FLAGS=$$(shell cat $$(EXTERNALS_CONFIG)/$(1)/{flags.common,flags.$$(PLATFORM_CONFIG_EXTENSION)} | xargs)
 4
 5$(1): $(1).configure $(1).build $(1).install $(1).clangd
 6
 7$(1).configure:
 8	cmake \
 9		-DCMAKE_BUILD_TYPE=$$(CMAKE_BUILD_TYPE) \
10		-DCMAKE_INSTALL_PREFIX=$$(INSTALL_DIRECTORY) \
11		-DCMAKE_EXPORT_COMPILE_COMMANDS=ON \
12		$$($(1)_CONFIGURE_FLAGS) \
13		-B "$$(BUILD_DIRECTORY)/$(1)" \
14		-S "$$(EXTERNALS)/$(1)"
15
16$(1).build:
17	cmake --build "$$(BUILD_DIRECTORY)/$(1)"
18
19$(1).clangd:
20	$$(call clangd_config_string,$(1),$$(BUILD_DIRECTORY)/$(1))
21
22$(1).install:
23	cmake --install "$$(BUILD_DIRECTORY)/$(1)" --prefix "$$(INSTALL_DIRECTORY)"
24
25$(1).clean:
26	rm -rf "$$(BUILD_DIRECTORY)/$(1)"
27endef
28...

 1# Makefile
 2$(eval $(call build_wrapper_meson,fontconfig))
 3$(eval $(call build_wrapper_cmake,cglm))
 4$(eval $(call build_wrapper_cmake,freetype))
 5$(eval $(call build_wrapper_meson,harfbuzz))
 6$(eval $(call build_wrapper_cmake,glfw))
 7$(eval $(call build_wrapper_cmake,glad))
 8$(eval $(call build_wrapper_autotools,ffmpeg))

A Note on Clangd integration

• Clangd & CompDB

  • By default, it looks for compile_commands.json file in the project root and the build/ directory
  • External projects generate the compdb in their respective build directories
  • Compdb file generated with Meson + Ninja has relative paths, so it's good only if read from the build directory
  • So we can't merge multiple compdb files into one, that would unnecessarily complicate things

• One way is to use an external tool like bear to wrap all the build commands & append their compdb files

  • This is actually not that impractical since we have to do that for Autotools projects anyways
  • Also bear is available on all major desktop platforms.
  • I did this successfully some time ago in bash. I had to create a config file for bear.
  • If this works for you, it's not a bad solution. But since I had to redo it, I was looking for something better

• Clangd configuration

  • We can set CompileFlags.CompilationDatabase in .clangd to tell clang where to look for the compdb file
  • But this is only for a single compdb file, we have many of them in different build directories
  • We can use pattern matching and conditionally change the CompilationDatabase path
    • This was suggested by Arsen on the mesonbuild matrix channel, thanks!
    • Configuration - CompilationDatabase
    • Configuration - If
    • LSP setup - Managarm Handbook
  • This solution is elegant and we can integrate this with some metaprogramming

 1# scripts/make/build_wrappers.mk
 2define clangd_config_string
 3	@touch $(CLANGD_CONFIG)
 4	@if ! grep -q $(1) $(CLANGD_CONFIG); then \
 5		{ echo "---"; \
 6		  echo "If:"; \
 7		  echo "  PathMatch: .*$(1).*"; \
 8		  echo "CompileFlags:"; \
 9		  echo "  CompilationDatabase: $(2)"; \
10		} | tee -a $(CLANGD_CONFIG); \
11endef
12
13# Makefile
14clangd:
15	$(call clangd_config_string,engine,$(BUILD_DIRECTORY))
16	$(call clangd_config_string,projects,$(BUILD_DIRECTORY))

• Clangd config generation
  • We grep from .clangd to avoid duplication
  • For each external library foo we generate foo.clangd where we call this metaprogram
  • This works!

Advantages

• Officially supported build systems are always better
  • Harfbuzz had some issues linking static libraries built with CMake. With Meson, it links just fine.
  • I don't like to use LLMs, and manually writing/maintaining build scripts is just not practical
  • Bash/Make is easy to debug

• Github PRs
  • build system migration by printfdebugging · Pull Request #5 · printfdebugging/graphics
  • compilation database fixes by printfdebugging · Pull Request #6 · printfdebugging/graphics