Unicode Tutorials - Herong's Tutorial Examples - v5.32, by Herong Yang
Examples of US-ASCII, UTF-8, UTF-16 and UTF-32 Encodings
This section provides examples of encoded byte sequences of US-ASCII, UTF-8, UTF-16, UTF-16BE, UTF-32BE encodings.
Let's continue to play with the testing program, EncodingSampler2.java, provided in the previous section. This time, I want to try the US-ASCII encoding:
C:\herong>java EncodingSampler2 US-ASCII US-ASCII encoding: Char, String, Writer, Charset, Encoder 00000000, 00, 00, 00, 00 0000003F, 3F, 3F, 3F, 3F 00000040, 40, 40, 40, 40 0000007F, 7F, 7F, 7F, 7F 00000080, 3F, 3F, 3F, 00 000000BF, 3F, 3F, 3F, 00 000000C0, 3F, 3F, 3F, 00 000000FF, 3F, 3F, 3F, 00 00000100, 3F, 3F, 3F, 00 00003FFF, 3F, 3F, 3F, 00 00004000, 3F, 3F, 3F, 00 00007FFF, 3F, 3F, 3F, 00 00008000, 3F, 3F, 3F, 00 0000BFFF, 3F, 3F, 3F, 00 0000C000, 3F, 3F, 3F, 00 0000EFFF, 3F, 3F, 3F, 00 0000F000, 3F, 3F, 3F, 00 0000FFFF, 3F, 3F, 3F, 00 0001F108, 3F, 3F, 3F, 00 0001F132, 3F, 3F, 3F, 00 0001F1A0, 3F, 3F, 3F, 00
It's obvious that US-ASCII works on a character set in the 0x0000 - 0x007F range.
Now I wan to try the UTF-8 encoding, which should be compatible with the US-ASCII encoding:
C:\herong>java EncodingSampler2 UTF-8 UTF-8 encoding: Char, String, Writer, Charset, Encoder 00000000, 00, 00, 00, 00 0000003F, 3F, 3F, 3F, 3F 00000040, 40, 40, 40, 40 0000007F, 7F, 7F, 7F, 7F 00000080, C2 80, C2 80, C2 80, C2 80 000000BF, C2 BF, C2 BF, C2 BF, C2 BF 000000C0, C3 80, C3 80, C3 80, C3 80 000000FF, C3 BF, C3 BF, C3 BF, C3 BF 00000100, C4 80, C4 80, C4 80, C4 80 00003FFF, E3 BF BF, E3 BF BF, E3 BF BF, E3 BF BF 00004000, E4 80 80, E4 80 80, E4 80 80, E4 80 80 00007FFF, E7 BF BF, E7 BF BF, E7 BF BF, E7 BF BF 00008000, E8 80 80, E8 80 80, E8 80 80, E8 80 80 0000BFFF, EB BF BF, EB BF BF, EB BF BF, EB BF BF 0000C000, EC 80 80, EC 80 80, EC 80 80, EC 80 80 0000EFFF, EE BF BF, EE BF BF, EE BF BF, EE BF BF 0000F000, EF 80 80, EF 80 80, EF 80 80, EF 80 80 0000FFFF, EF BF BF, EF BF BF, EF BF BF, EF BF BF 0001F108, F0 9F 84 88, F0 9F 84 88, F0 9F 84 88, F0 9F 84 88 0001F132, F0 9F 84 B2, F0 9F 84 B2, F0 9F 84 B2, F0 9F 84 B2 0001F1A0, F0 9F 86 A0, F0 9F 86 A0, F0 9F 86 A0, F0 9F 86 A0
The output matches my expectation.
Let's try another Unicode encoding, UTF-16:
C:\herong>java EncodingSampler2 UTF-16 UTF-16 encoding: Char, String, Writer, Charset, Encoder 00000000, FE FF 00 00, FE FF 00 00, FE FF 00 00, FE FF 00 00 0000003F, FE FF 00 3F, FE FF 00 3F, FE FF 00 3F, FE FF 00 3F 00000040, FE FF 00 40, FE FF 00 40, FE FF 00 40, FE FF 00 40 0000007F, FE FF 00 7F, FE FF 00 7F, FE FF 00 7F, FE FF 00 7F 00000080, FE FF 00 80, FE FF 00 80, FE FF 00 80, FE FF 00 80 000000BF, FE FF 00 BF, FE FF 00 BF, FE FF 00 BF, FE FF 00 BF 000000C0, FE FF 00 C0, FE FF 00 C0, FE FF 00 C0, FE FF 00 C0 000000FF, FE FF 00 FF, FE FF 00 FF, FE FF 00 FF, FE FF 00 FF 00000100, FE FF 01 00, FE FF 01 00, FE FF 01 00, FE FF 01 00 00003FFF, FE FF 3F FF, FE FF 3F FF, FE FF 3F FF, FE FF 3F FF 00004000, FE FF 40 00, FE FF 40 00, FE FF 40 00, FE FF 40 00 00007FFF, FE FF 7F FF, FE FF 7F FF, FE FF 7F FF, FE FF 7F FF 00008000, FE FF 80 00, FE FF 80 00, FE FF 80 00, FE FF 80 00 0000BFFF, FE FF BF FF, FE FF BF FF, FE FF BF FF, FE FF BF FF 0000C000, FE FF C0 00, FE FF C0 00, FE FF C0 00, FE FF C0 00 0000EFFF, FE FF EF FF, FE FF EF FF, FE FF EF FF, FE FF EF FF 0000F000, FE FF F0 00, FE FF F0 00, FE FF F0 00, FE FF F0 00 0000FFFF, FE FF FF FF, FE FF FF FF, FE FF FF FF, FE FF FF FF 0001F108, FE FF D8 3C DD 08, FE FF D8 3C DD 08, FE FF D8 3C DD 08, ... 0001F132, FE FF D8 3C DD 32, FE FF D8 3C DD 32, FE FF D8 3C DD 32, ... 0001F1A0, FE FF D8 3C DD A0, FE FF D8 3C DD A0, FE FF D8 3C DD A0, ...
Noticed that the first 2 bytes, 0xFEFF, of the encoding output is a BOM (Byte Order Mark) indicates that the following byte sequence is in Big Endian format. In other words, Java uses the "Big-Endian with BOM" format for UTF-16 encoding by default.
How about UTF-16BE encoding:
C:\herong>java EncodingSampler2 UTF-16BE UTF-16BE encoding: Char, String, Writer, Charset, Encoder 00000000, 00 00, 00 00, 00 00, 00 00 0000003F, 00 3F, 00 3F, 00 3F, 00 3F 00000040, 00 40, 00 40, 00 40, 00 40 0000007F, 00 7F, 00 7F, 00 7F, 00 7F 00000080, 00 80, 00 80, 00 80, 00 80 000000BF, 00 BF, 00 BF, 00 BF, 00 BF 000000C0, 00 C0, 00 C0, 00 C0, 00 C0 000000FF, 00 FF, 00 FF, 00 FF, 00 FF 00000100, 01 00, 01 00, 01 00, 01 00 00003FFF, 3F FF, 3F FF, 3F FF, 3F FF 00004000, 40 00, 40 00, 40 00, 40 00 00007FFF, 7F FF, 7F FF, 7F FF, 7F FF 00008000, 80 00, 80 00, 80 00, 80 00 0000BFFF, BF FF, BF FF, BF FF, BF FF 0000C000, C0 00, C0 00, C0 00, C0 00 0000EFFF, EF FF, EF FF, EF FF, EF FF 0000F000, F0 00, F0 00, F0 00, F0 00 0000FFFF, FF FF, FF FF, FF FF, FF FF 0001F108, D8 3C DD 08, D8 3C DD 08, D8 3C DD 08, D8 3C DD 08 0001F132, D8 3C DD 32, D8 3C DD 32, D8 3C DD 32, D8 3C DD 32 0001F1A0, D8 3C DD A0, D8 3C DD A0, D8 3C DD A0, D8 3C DD A0
Notice that how characters in the 0x10000 - 0x10FFFF range are encoded into 32 bits, or 16-bit pairs.
The last Unicode encoding to look at is the UTF-32BE encoding:
C:\herong>java EncodingSampler2 UTF-32BE UTF-32BE encoding: Char, String, Writer, Charset, Encoder 00000000, 00 00 00 00, 00 00 00 00, 00 00 00 00, 00 00 00 00 0000003F, 00 00 00 3F, 00 00 00 3F, 00 00 00 3F, 00 00 00 3F 00000040, 00 00 00 40, 00 00 00 40, 00 00 00 40, 00 00 00 40 0000007F, 00 00 00 7F, 00 00 00 7F, 00 00 00 7F, 00 00 00 7F 00000080, 00 00 00 80, 00 00 00 80, 00 00 00 80, 00 00 00 80 000000BF, 00 00 00 BF, 00 00 00 BF, 00 00 00 BF, 00 00 00 BF 000000C0, 00 00 00 C0, 00 00 00 C0, 00 00 00 C0, 00 00 00 C0 000000FF, 00 00 00 FF, 00 00 00 FF, 00 00 00 FF, 00 00 00 FF 00000100, 00 00 01 00, 00 00 01 00, 00 00 01 00, 00 00 01 00 00003FFF, 00 00 3F FF, 00 00 3F FF, 00 00 3F FF, 00 00 3F FF 00004000, 00 00 40 00, 00 00 40 00, 00 00 40 00, 00 00 40 00 00007FFF, 00 00 7F FF, 00 00 7F FF, 00 00 7F FF, 00 00 7F FF 00008000, 00 00 80 00, 00 00 80 00, 00 00 80 00, 00 00 80 00 0000BFFF, 00 00 BF FF, 00 00 BF FF, 00 00 BF FF, 00 00 BF FF 0000C000, 00 00 C0 00, 00 00 C0 00, 00 00 C0 00, 00 00 C0 00 0000EFFF, 00 00 EF FF, 00 00 EF FF, 00 00 EF FF, 00 00 EF FF 0000F000, 00 00 F0 00, 00 00 F0 00, 00 00 F0 00, 00 00 F0 00 0000FFFF, 00 00 FF FF, 00 00 FF FF, 00 00 FF FF, 00 00 FF FF 0001F108, 00 01 F1 08, 00 01 F1 08, 00 01 F1 08, 00 01 F1 08 0001F132, 00 01 F1 32, 00 01 F1 32, 00 01 F1 32, 00 01 F1 32 0001F1A0, 00 01 F1 A0, 00 01 F1 A0, 00 01 F1 A0, 00 01 F1 A0
Obviously, UTF-32BE encoding is much simpler and very easy to understand than UTF-16BE and UTF-8 encodings.
Table of Contents
ASCII Character Set and Encoding
GB2312 Character Set and Encoding
GB18030 Character Set and Encoding
JIS X0208 Character Set and Encodings
UTF-8 (Unicode Transformation Format - 8-Bit)
UTF-16, UTF-16BE and UTF-16LE Encodings
UTF-32, UTF-32BE and UTF-32LE Encodings
Python Language and Unicode Characters
Java Language and Unicode Characters
List of Supported Character Encodings in Java
EncodingSampler.java - Testing encode() Methods
Examples of CP1252 and ISO-8859-1 Encodings
►Examples of US-ASCII, UTF-8, UTF-16 and UTF-32 Encodings
Encoding Conversion Programs for Encoded Text Files
Using Notepad as a Unicode Text Editor
Using Microsoft Word as a Unicode Text Editor