Unicode - A character is not a char!

2026-08-01T00:00:00+00:00

Unicode</h1>

A character is not a char...</h2>

... unless you just write english.</li>
ASCII covers all the latin english characters and that at most takes 1 byte (7 bits to be precise).</li>
If we also include the 8th bit, then we can represent another 127 characters.</li>

We can't represent all the characters from all the languages using one byte.</li> </ul>
Codepoint</h2>

We can use an integer instead, it has 32 bits (4 bytes).</li>
With that, we can assign a unique number to each character from all the languages and still have plenty left.</li>
This integer is called a unicode codepoint</code>, a unique number used to identify a character.</li> </ul> Unicode Transformation Format 8 (UTF-8)</h2> We can write these codepoints to a file and we are done! Hold on..</li> ASCII characters only need 1 byte, using 4 bytes for that would mean 3 wasted bytes</li> Because of these LLMs, ssd prices have trippled, storage isn't cheap anymore!</li> So we can't write the characters as 4 byte sequences to the file, what should we do then?</li> Codepoints are just positive integers.</li> All the codepoints from 0 to 127 can be represented by 1 byte, 8th bit is used to flag a 1 byte long codepoint.</li> Similarly all the codepoints can be encoded with varying number of bytes & we can use some bits to say how many.</li> There are other UTF encoding schemes (UTF-16, UTF-32) which use 2 & 4 bytes at least to represent a codepoint resp.</li> </ul> /* source: https://www.unicode.org/versions/Unicode17.0.0/core-spec/chapter-3/#G27506 */ 1 byte 2 bytes 3 bytes 4 bytes <---------------------|----------------------|-----------------------------------|-------------------------------> U+0000 .. U+007F U+0080 .. U+07FF U+0800 .. U+0FFF U+1000 .. U+CFFF U+D000 .. U+D7FF U+E000 .. U+FFFF U+10000 .. U+3FFFF U+40000 .. U+FFFFF U+100000 .. U+10FFFF</code></pre>UTF-8 Encoding/Decoding</h2> This RFC document</a> shows how codepoints are encoded with UTF-8.</li> Notice how the first few bits on each encoded byte are marked.</li> If the first byte of the sequence starts with 0</code>, we can read the next 7 bits to find the codepoint</li> If it's marked with 110</code>, 1110</code> or 11110</code>, we need to read the next 1,2,3 bytes resp to decode the codepoint.</li> Once we read all the bytes, we can put those x</code> bits into an integer from left to right.</li> The resulting number (max 21 bits) would be our codepoint.</li> </ul> /* source: https://www.rfc-editor.org/info/rfc3629/#ref-UNICODE */ Char. number range | UTF-8 octet sequence (hexadecimal) | (binary) --------------------+--------------------------------------------- 0000 0000-0000 007F | 0xxxxxxx 0000 0080-0000 07FF | 110xxxxx 10xxxxxx 0000 0800-0000 FFFF | 1110xxxx 10xxxxxx 10xxxxxx 0001 0000-0010 FFFF | 11110xxx 10xxxxxx 10xxxxxx 10xxxxxx</code></pre> Encoding is quite simple, we repeat the decoding steps in reverse</li> Use the codepoint to find out how many bytes it will take to encode it, see the numberline above.</li> Then starting from left to right, start filling in those x</code>s, see the table in the decoding section.</li> See the example below... you can read it in reverse for decoding.</li> </ul> character =>'世' codepoint => U+4E16 encodedLen => 3 bytes /* U+0800 <= 0x4E16 <= U+FFFF */ binary => '0b100111000010110' binary => ' 100 111000 010110' pattern => 1110|xxxx 10|xxxxxx 10|xxxxxx encoded => 1110|0100 10|111000 10|010110 ------------------------------------------------ UTF-8 => 11100100 10111000 10010110 </code></pre>Unicode Library</h1> I wrote a simple library for UTF-8 encoding/decoding, it lives on github, at printfdebugging/unicode</a>.</li> It has a ton of tests for encoding-decoding roundtrip integrity testing & for checking invalid byte sequences.</li> </ul>

printfdebugging - unicode

Unicode - A character is not a char!

Unicode</h1>

Unicode Library</h1> I wrote a simple library for UTF-8 encoding/decoding, it lives on github, at printfdebugging/unicode</a>.</li> It has a ton of tests for encoding-decoding roundtrip integrity testing & for checking invalid byte sequences.</li> </ul>

Unicode Library</h1>

I wrote a simple library for UTF-8 encoding/decoding, it lives on github, at printfdebugging/unicode</a>.</li>
It has a ton of tests for encoding-decoding roundtrip integrity testing & for checking invalid byte sequences.</li> </ul>