This section provides a quick summary of features of UTF-8 encoding. UTF-8 is very efficient for Western language characters.
Features of UTF-8 encoding:
1.UTF-8 is very efficient in storage for characters in Western languages.
The code points for most of characters in Western languages are in the
U+0000...U+007F range, which will be encoded with 1 byte per character.
The rest of characters will be encoded by two bytes.
Some reports says that any text data in Western
language will be encoded with 1.1 bytes per character on average.
This is a big saving comparing with any other Unicode encodings.
2. UTF-8 is compatible with the single byte ASCII encoding. In another word,
ASCII encoding can be viewed as a sub set of the UTF-8 encoding.
Any ASCII byte stream is a valid UTF-8 byte stream.
3. UTF-8 is backward compatible with Unicode 3.0 character set, which only contains characters
in the U+0000...U+FFFF range.
4. UTF-8 is less efficient in storage for characters in CJK (Chinese, Japanese, and Korean)
languages comparing to UTF-16 encoding. The code points for most of characters in CJK languages are in the
U+000800...U+00FFFF range, which will be encoded in UTF-8 with 3 bytes per character.
UTF-16 will encode all characters 2 bytes per character.
5. Processing a UTF-8 encoded text files is relatively easy.
If the leading bit of the first byte is a 0, then 1 byte is used to encoding this character.
If the leading bit of the first byte is not a 0, then the number of non-zero leading bits is total
number bytes used to encoding this character.
6. Another nice nature of UTF-8 encoding is that all subsequent bytes in encoded multi-byte sequence
are having the pattern of 0b10xxxxxx in binary format.
If you are looking at one byte of an encoded character in the middle of the encoded stream,
and want to find out the first byte of this encoded character, you just need
to follow this simple logic:
while (current byte matches the bit pattern '10xxxxxx') {
Current byte = previous byte
}
