printfdebugging - Text Editor

Unicode - A character is not a char!

2026-08-01T00:00:00+00:00

Unicode</h1>

A character is not a char...</h2>

... unless you just write english.</li>
ASCII covers all the latin english characters and that at most takes 1 byte (7 bits to be precise).</li>
If we also include the 8th bit, then we can represent another 127 characters.</li>

We can't represent all the characters from all the languages using one byte.</li> </ul>
Codepoint</h2>

We can use an integer instead, it has 32 bits (4 bytes).</li>
With that, we can assign a unique number to each character from all the languages and still have plenty left.</li>
This integer is called a unicode codepoint</code>, a unique number used to identify a character.</li> </ul> Unicode Transformation Format 8 (UTF-8)</h2> We can write these codepoints to a file and we are done! Hold on..</li> ASCII characters only need 1 byte, using 4 bytes for that would mean 3 wasted bytes</li> Because of these LLMs, ssd prices have trippled, storage isn't cheap anymore!</li> So we can't write the characters as 4 byte sequences to the file, what should we do then?</li> Codepoints are just positive integers.</li> All the codepoints from 0 to 127 can be represented by 1 byte, 8th bit is used to flag a 1 byte long codepoint.</li> Similarly all the codepoints can be encoded with varying number of bytes & we can use some bits to say how many.</li> There are other UTF encoding schemes (UTF-16, UTF-32) which use 2 & 4 bytes at least to represent a codepoint resp.</li> </ul> /* source: https://www.unicode.org/versions/Unicode17.0.0/core-spec/chapter-3/#G27506 */ 1 byte 2 bytes 3 bytes 4 bytes <---------------------|----------------------|-----------------------------------|-------------------------------> U+0000 .. U+007F U+0080 .. U+07FF U+0800 .. U+0FFF U+1000 .. U+CFFF U+D000 .. U+D7FF U+E000 .. U+FFFF U+10000 .. U+3FFFF U+40000 .. U+FFFFF U+100000 .. U+10FFFF</code></pre>UTF-8 Encoding/Decoding</h2> This RFC document</a> shows how codepoints are encoded with UTF-8.</li> Notice how the first few bits on each encoded byte are marked.</li> If the first byte of the sequence starts with 0</code>, we can read the next 7 bits to find the codepoint</li> If it's marked with 110</code>, 1110</code> or 11110</code>, we need to read the next 1,2,3 bytes resp to decode the codepoint.</li> Once we read all the bytes, we can put those x</code> bits into an integer from left to right.</li> The resulting number (max 21 bits) would be our codepoint.</li> </ul> /* source: https://www.rfc-editor.org/info/rfc3629/#ref-UNICODE */ Char. number range | UTF-8 octet sequence (hexadecimal) | (binary) --------------------+--------------------------------------------- 0000 0000-0000 007F | 0xxxxxxx 0000 0080-0000 07FF | 110xxxxx 10xxxxxx 0000 0800-0000 FFFF | 1110xxxx 10xxxxxx 10xxxxxx 0001 0000-0010 FFFF | 11110xxx 10xxxxxx 10xxxxxx 10xxxxxx</code></pre> Encoding is quite simple, we repeat the decoding steps in reverse</li> Use the codepoint to find out how many bytes it will take to encode it, see the numberline above.</li> Then starting from left to right, start filling in those x</code>s, see the table in the decoding section.</li> See the example below... you can read it in reverse for decoding.</li> </ul> character =>'世' codepoint => U+4E16 encodedLen => 3 bytes /* U+0800 <= 0x4E16 <= U+FFFF */ binary => '0b100111000010110' binary => ' 100 111000 010110' pattern => 1110|xxxx 10|xxxxxx 10|xxxxxx encoded => 1110|0100 10|111000 10|010110 ------------------------------------------------ UTF-8 => 11100100 10111000 10010110 </code></pre>Unicode Library</h1> I wrote a simple library for UTF-8 encoding/decoding, it lives on github, at printfdebugging/unicode</a>.</li> It has a ton of tests for encoding-decoding roundtrip integrity testing & for checking invalid byte sequences.</li> </ul>

Build System Migration - Meson + Autotools + CMake = PkgConfig!

2026-07-01T00:00:00+00:00

The Problem</h1>

What happens when you use one build system and try to use libraries with another build system?

CMake can't include Meson projects</li>
Meson can't include Autotools projects</li>
Autotools + Make doesn't generate a `compile_commands.json</code> file</li>`
`Custom scripts in build system specific scripting languages get ugly quite quickly</li> </ul> </li> </ul>`Potential Ways to Fix it</h1> The Better One of the Bunch - Meson! It can integrate CMake projects</li> It can build Autotools projects as one would do manually with some custom scripting (haven't tried it myself).</li> </ul> </li> How build systems talk to each other There's no standard way to do it at build time</li> Each build system supports pkgconfig</a>, so the installed artifacts can be consumed through that interface.</li> It is build system agnostic, so as long as a .pc</code> file is generated, we can consume that project.</li> CollaboraOffice</a> uses this approach to consume external libraries (with some Make metaprogramming)</li> </ul> </li> </ul> The Migration Process</h1> Data & Code Separation</h2> Architectural overview The configure & build commands are pretty standard for any build system</li> The only thing which changes is the configure options. We separated them from the build commands.</li> For each exteral project foo, there are 4 config files, config/external/foo/flags.{common,linux,apple,windows}</code></li> These contain the configure flags for the specific build system.</li> We then have a scripts/make/exports.mk</code></a> script which looks like this...</li> </ul> </li> </ul> 1# scripts/make/exports.mk 2EXTERNALS = $(shell realpath ./external) 3EXTERNALS_CONFIG = $(shell realpath ./config/external) 4INSTALL_DIRECTORY = $(shell realpath ./install) 5BUILD_DIRECTORY = $(shell realpath ./build) 6CLANGD_CONFIG = $(shell realpath ./.clangd) 7PLATFORM = $(shell uname -s) 8MESON_BUILD_TYPE = debug 9CMAKE_BUILD_TYPE = Debug 10SHELL := /bin/bash 11 12ifeq ($(PLATFORM),Linux) 13 PLATFORM_CONFIG_EXTENSION = linux 14else ifeq ($(PLATFORM),Darwin) 15 PLATFORM_CONFIG_EXTENSION = apple 16else ifeq ($(PLATFORM),MINGW64) 17 PLATFORM_CONFIG_EXTENSION = windows 18else 19 PLATFORM_CONFIG_EXTENSION = unknown 20endif</code></pre> About exports.mk Here we define some paths and some variables which would be used in the build scripts</li> Platform detection sets the flags file extension which we need to include other than flags.common</code></li> </ul> </li> Make metaprogramming The metaprograms define build wrappers in scripts/make/build_wrappers.mk</code></a> and eval call them from Makefile</code></a></li> There's one for Meson, Autotools and CMake</li> For a project foo</code>, we generate foo</code>, foo.configure</code>, foo.build</code>, foo.clangd</code>, foo.install</code> and foo.clean</code></li> We get the common and platform specific configure flags from the flag files in config/external/foo</code></li> Target foo</code> depends on all those foo</code> prefixed targets</li> </ul> </li> </ul> 1# scripts/make/build_wrappers.mk 2define build_wrapper_cmake 3$(1)_CONFIGURE_FLAGS=$$(shell cat $$(EXTERNALS_CONFIG)/$(1)/{flags.common,flags.$$(PLATFORM_CONFIG_EXTENSION)} | xargs) 4 5$(1): $(1).configure $(1).build $(1).install $(1).clangd 6 7$(1).configure: 8 cmake \ 9 -DCMAKE_BUILD_TYPE=$$(CMAKE_BUILD_TYPE) \ 10 -DCMAKE_INSTALL_PREFIX=$$(INSTALL_DIRECTORY) \ 11 -DCMAKE_EXPORT_COMPILE_COMMANDS=ON \ 12 $$($(1)_CONFIGURE_FLAGS) \ 13 -B "$$(BUILD_DIRECTORY)/$(1)" \ 14 -S "$$(EXTERNALS)/$(1)" 15 16$(1).build: 17 cmake --build "$$(BUILD_DIRECTORY)/$(1)" 18 19$(1).clangd: 20 $$(call clangd_config_string,$(1),$$(BUILD_DIRECTORY)/$(1)) 21 22$(1).install: 23 cmake --install "$$(BUILD_DIRECTORY)/$(1)" --prefix "$$(INSTALL_DIRECTORY)" 24 25$(1).clean: 26 rm -rf "$$(BUILD_DIRECTORY)/$(1)" 27endef 28...</code></pre> 1# Makefile 2$(eval $(call build_wrapper_meson,fontconfig)) 3$(eval $(call build_wrapper_cmake,cglm)) 4$(eval $(call build_wrapper_cmake,freetype)) 5$(eval $(call build_wrapper_meson,harfbuzz)) 6$(eval $(call build_wrapper_cmake,glfw)) 7$(eval $(call build_wrapper_cmake,glad)) 8$(eval $(call build_wrapper_autotools,ffmpeg))</code></pre>A Note on Clangd integration</h2> Clangd & CompDB By default, it looks for compile_commands.json</code> file in the project root and the build/</code> directory</li> External projects generate the compdb in their respective build directories</li> Compdb file generated with Meson + Ninja has relative paths, so it's good only if read from the build directory</li> So we can't merge multiple compdb files into one, that would unnecessarily complicate things</li> </ul> </li> One way is to use an external tool like bear</code> to wrap all the build commands & append their compdb files This is actually not that impractical since we have to do that for Autotools projects anyways</li> Also bear is available on all major desktop platforms.</li> I did this successfully</a> some time ago in bash. I had to create a config file</a> for bear.</li> If this works for you, it's not a bad solution. But since I had to redo it, I was looking for something better</li> </ul> </li> Clangd configuration We can set CompileFlags.CompilationDatabase</code> in .clangd</code> to tell clang where to look for the compdb file</li> But this is only for a single compdb file, we have many of them in different build directories</li> We can use pattern matching and conditionally change the CompilationDatabase</code> path This was suggested by Arsen</a> on the mesonbuild matrix channel</a>, thanks!</li> Configuration - CompilationDatabase</a></li> Configuration - If</a></li> LSP setup - Managarm Handbook</a></li> </ul> </li> This solution is elegant and we can integrate this with some metaprogramming</li> </ul> </li> </ul> 1# scripts/make/build_wrappers.mk 2define clangd_config_string 3 @touch $(CLANGD_CONFIG) 4 @if ! grep -q $(1) $(CLANGD_CONFIG); then \ 5 { echo "---"; \ 6 echo "If:"; \ 7 echo " PathMatch: .$(1).*"; \ 8 echo "CompileFlags:"; \ 9 echo " CompilationDatabase: $(2)"; \ 10 } | tee -a $(CLANGD_CONFIG); \ 11endef 12 13# Makefile 14clangd: 15 $(call clangd_config_string,engine,$(BUILD_DIRECTORY)) 16 $(call clangd_config_string,projects,$(BUILD_DIRECTORY))</code></pre> Clangd config generation We grep from .clangd</code> to avoid duplication</li> For each external library foo we generate foo.clangd</code> where we call this metaprogram</li> This works!</li> </ul> </li> </ul> Advantages</h1> Officially supported build systems are always better Harfbuzz had some issues</a> linking static libraries built with CMake. With Meson, it links just fine.</li> I don't like to use LLMs, and manually writing/maintaining build scripts is just not practical</li> Bash/Make is easy to debug</li> </ul> </li> </ul> Github PRs build system migration by printfdebugging · Pull Request #5 · printfdebugging/graphics</a></li> compilation database fixes by printfdebugging · Pull Request #6 · printfdebugging/graphics</a></li> </ul> </li> </ul>

Unicode Library</h1>

I wrote a simple library for UTF-8 encoding/decoding, it lives on github, at printfdebugging/unicode</a>.</li>
It has a ton of tests for encoding-decoding roundtrip integrity testing & for checking invalid byte sequences.</li> </ul>

The Migration Process</h1>

Project Arcitecture</h1>

printfdebugging - Text Editor

Unicode - A character is not a char!

Unicode</h1>

Build System Migration - Meson + Autotools + CMake = PkgConfig!

Overview - Build System and the Project Arcitecture

printfdebugging - Text Editor

Unicode - A character is not a char!

Unicode</h1>

Unicode Library</h1> I wrote a simple library for UTF-8 encoding/decoding, it lives on github, at printfdebugging/unicode</a>.</li> It has a ton of tests for encoding-decoding roundtrip integrity testing & for checking invalid byte sequences.</li> </ul>

Build System Migration - Meson + Autotools + CMake = PkgConfig!

The Problem</h1> What happens when you use one build system and try to use libraries with another build system? CMake can't include Meson projects</li> Meson can't include Autotools projects</li> Autotools + Make doesn't generate a compile_commands.json</code> file</li>

The Migration Process</h1>

A Note on Clangd integration</h2> Clangd & CompDB</p>

Overview - Build System and the Project Arcitecture

The Build System</h1> This project is written in C</strong>.</p>

Project Arcitecture</h1>

Unicode Library</h1>

I wrote a simple library for UTF-8 encoding/decoding, it lives on github, at printfdebugging/unicode</a>.</li>
It has a ton of tests for encoding-decoding roundtrip integrity testing & for checking invalid byte sequences.</li> </ul>