Unicode - A character is not a char!

Unicode

A character is not a char...

• ... unless you just write english.
• ASCII covers all the latin english characters and that at most takes 1 byte (7 bits to be precise).
• If we also include the 8th bit, then we can represent another 127 characters.
• We can't represent all the characters from all the languages using one byte.

Codepoint

• We can use an integer instead, it has 32 bits (4 bytes).
• With that, we can assign a unique number to each character from all the languages and still have plenty left.
• This integer is called a unicode codepoint, a unique number used to identify a character.

Unicode Transformation Format 8 (UTF-8)

• We can write these codepoints to a file and we are done! Hold on..
• ASCII characters only need 1 byte, using 4 bytes for that would mean 3 wasted bytes
• Because of these LLMs, ssd prices have trippled, storage isn't cheap anymore!
• So we can't write the characters as 4 byte sequences to the file, what should we do then?
• Codepoints are just positive integers.
• All the codepoints from 0 to 127 can be represented by 1 byte, 8th bit is used to flag a 1 byte long codepoint.
• Similarly all the codepoints can be encoded with varying number of bytes & we can use some bits to say how many.
• There are other UTF encoding schemes (UTF-16, UTF-32) which use 2 & 4 bytes at least to represent a codepoint resp.

/* source: https://www.unicode.org/versions/Unicode17.0.0/core-spec/chapter-3/#G27506 */

        1 byte                2 bytes                      3 bytes                             4 bytes
<---------------------|----------------------|-----------------------------------|------------------------------->
    U+0000 .. U+007F
                          U+0080 .. U+07FF
                                                         U+0800 .. U+0FFF
                                                         U+1000 .. U+CFFF
                                                         U+D000 .. U+D7FF
                                                         U+E000 .. U+FFFF
                                                                                         U+10000  .. U+3FFFF
                                                                                         U+40000  .. U+FFFFF
                                                                                         U+100000 .. U+10FFFF

UTF-8 Encoding/Decoding

• This RFC document shows how codepoints are encoded with UTF-8.
• Notice how the first few bits on each encoded byte are marked.
• If the first byte of the sequence starts with 0, we can read the next 7 bits to find the codepoint
• If it's marked with 110, 1110 or 11110, we need to read the next 1,2,3 bytes resp to decode the codepoint.
• Once we read all the bytes, we can put those x bits into an integer from left to right.
• The resulting number (max 21 bits) would be our codepoint.

/* source: https://www.rfc-editor.org/info/rfc3629/#ref-UNICODE */

Char. number range  |        UTF-8 octet sequence
   (hexadecimal)    |              (binary)
--------------------+---------------------------------------------
0000 0000-0000 007F | 0xxxxxxx
0000 0080-0000 07FF | 110xxxxx 10xxxxxx
0000 0800-0000 FFFF | 1110xxxx 10xxxxxx 10xxxxxx
0001 0000-0010 FFFF | 11110xxx 10xxxxxx 10xxxxxx 10xxxxxx

• Encoding is quite simple, we repeat the decoding steps in reverse
• Use the codepoint to find out how many bytes it will take to encode it, see the numberline above.
• Then starting from left to right, start filling in those xs, see the table in the decoding section.
• See the example below... you can read it in reverse for decoding.

character  =>'世'
codepoint  => U+4E16
encodedLen => 3 bytes /* U+0800 <= 0x4E16 <= U+FFFF */

binary  => '0b100111000010110'
binary  => '     100        111000        010110'
pattern => 1110|xxxx     10|xxxxxx     10|xxxxxx
encoded => 1110|0100     10|111000     10|010110
------------------------------------------------
UTF-8   =>  11100100      10111000      10010110

Unicode Library

• I wrote a simple library for UTF-8 encoding/decoding, it lives on github, at printfdebugging/unicode.
• It has a ton of tests for encoding-decoding roundtrip integrity testing & for checking invalid byte sequences.