Unicode Tutorials - Herong's Tutorial Examples

∟Character Encoding in Java

∟Examples of US-ASCII, UTF-8, UTF-16 and UTF-32 Encodings

This section provides examples of encoded byte sequences of US-ASCII, UTF-8, UTF-16, UTF-16BE, UTF-32BE encodings.

Let's continue to play with the testing program, EncodingSampler2.java, provided in the previous section. This time, I want to try the US-ASCII encoding:

C:\herong>java EncodingSampler2 US-ASCII
US-ASCII encoding:
Char,     String, Writer, Charset, Encoder
00000000, 00, 00, 00, 00
0000003F, 3F, 3F, 3F, 3F
00000040, 40, 40, 40, 40
0000007F, 7F, 7F, 7F, 7F
00000080, 3F, 3F, 3F, 00
000000BF, 3F, 3F, 3F, 00
000000C0, 3F, 3F, 3F, 00
000000FF, 3F, 3F, 3F, 00
00000100, 3F, 3F, 3F, 00
00003FFF, 3F, 3F, 3F, 00
00004000, 3F, 3F, 3F, 00
00007FFF, 3F, 3F, 3F, 00
00008000, 3F, 3F, 3F, 00
0000BFFF, 3F, 3F, 3F, 00
0000C000, 3F, 3F, 3F, 00
0000EFFF, 3F, 3F, 3F, 00
0000F000, 3F, 3F, 3F, 00
0000FFFF, 3F, 3F, 3F, 00
0001F108, 3F, 3F, 3F, 00
0001F132, 3F, 3F, 3F, 00
0001F1A0, 3F, 3F, 3F, 00

It's obvious that US-ASCII works on a character set in the 0x0000 - 0x007F range.

Now I wan to try the UTF-8 encoding, which should be compatible with the US-ASCII encoding:

C:\herong>java EncodingSampler2 UTF-8
UTF-8 encoding:
Char,     String, Writer, Charset, Encoder
00000000, 00, 00, 00, 00
0000003F, 3F, 3F, 3F, 3F
00000040, 40, 40, 40, 40
0000007F, 7F, 7F, 7F, 7F
00000080, C2 80, C2 80, C2 80, C2 80
000000BF, C2 BF, C2 BF, C2 BF, C2 BF
000000C0, C3 80, C3 80, C3 80, C3 80
000000FF, C3 BF, C3 BF, C3 BF, C3 BF
00000100, C4 80, C4 80, C4 80, C4 80
00003FFF, E3 BF BF, E3 BF BF, E3 BF BF, E3 BF BF
00004000, E4 80 80, E4 80 80, E4 80 80, E4 80 80
00007FFF, E7 BF BF, E7 BF BF, E7 BF BF, E7 BF BF
00008000, E8 80 80, E8 80 80, E8 80 80, E8 80 80
0000BFFF, EB BF BF, EB BF BF, EB BF BF, EB BF BF
0000C000, EC 80 80, EC 80 80, EC 80 80, EC 80 80
0000EFFF, EE BF BF, EE BF BF, EE BF BF, EE BF BF
0000F000, EF 80 80, EF 80 80, EF 80 80, EF 80 80
0000FFFF, EF BF BF, EF BF BF, EF BF BF, EF BF BF
0001F108, F0 9F 84 88, F0 9F 84 88, F0 9F 84 88, F0 9F 84 88
0001F132, F0 9F 84 B2, F0 9F 84 B2, F0 9F 84 B2, F0 9F 84 B2
0001F1A0, F0 9F 86 A0, F0 9F 86 A0, F0 9F 86 A0, F0 9F 86 A0

The output matches my expectation.

Let's try another Unicode encoding, UTF-16:

C:\herong>java EncodingSampler2 UTF-16
UTF-16 encoding:
Char,     String, Writer, Charset, Encoder
00000000, FE FF 00 00, FE FF 00 00, FE FF 00 00, FE FF 00 00
0000003F, FE FF 00 3F, FE FF 00 3F, FE FF 00 3F, FE FF 00 3F
00000040, FE FF 00 40, FE FF 00 40, FE FF 00 40, FE FF 00 40
0000007F, FE FF 00 7F, FE FF 00 7F, FE FF 00 7F, FE FF 00 7F
00000080, FE FF 00 80, FE FF 00 80, FE FF 00 80, FE FF 00 80
000000BF, FE FF 00 BF, FE FF 00 BF, FE FF 00 BF, FE FF 00 BF
000000C0, FE FF 00 C0, FE FF 00 C0, FE FF 00 C0, FE FF 00 C0
000000FF, FE FF 00 FF, FE FF 00 FF, FE FF 00 FF, FE FF 00 FF
00000100, FE FF 01 00, FE FF 01 00, FE FF 01 00, FE FF 01 00
00003FFF, FE FF 3F FF, FE FF 3F FF, FE FF 3F FF, FE FF 3F FF
00004000, FE FF 40 00, FE FF 40 00, FE FF 40 00, FE FF 40 00
00007FFF, FE FF 7F FF, FE FF 7F FF, FE FF 7F FF, FE FF 7F FF
00008000, FE FF 80 00, FE FF 80 00, FE FF 80 00, FE FF 80 00
0000BFFF, FE FF BF FF, FE FF BF FF, FE FF BF FF, FE FF BF FF
0000C000, FE FF C0 00, FE FF C0 00, FE FF C0 00, FE FF C0 00
0000EFFF, FE FF EF FF, FE FF EF FF, FE FF EF FF, FE FF EF FF
0000F000, FE FF F0 00, FE FF F0 00, FE FF F0 00, FE FF F0 00
0000FFFF, FE FF FF FF, FE FF FF FF, FE FF FF FF, FE FF FF FF
0001F108, FE FF D8 3C DD 08, FE FF D8 3C DD 08, FE FF D8 3C DD 08, ...
0001F132, FE FF D8 3C DD 32, FE FF D8 3C DD 32, FE FF D8 3C DD 32, ...
0001F1A0, FE FF D8 3C DD A0, FE FF D8 3C DD A0, FE FF D8 3C DD A0, ...

Noticed that the first 2 bytes, 0xFEFF, of the encoding output is a BOM (Byte Order Mark) indicates that the following byte sequence is in Big Endian format. In other words, Java uses the "Big-Endian with BOM" format for UTF-16 encoding by default.

How about UTF-16BE encoding:

C:\herong>java EncodingSampler2 UTF-16BE
UTF-16BE encoding:
Char,     String, Writer, Charset, Encoder
00000000, 00 00, 00 00, 00 00, 00 00
0000003F, 00 3F, 00 3F, 00 3F, 00 3F
00000040, 00 40, 00 40, 00 40, 00 40
0000007F, 00 7F, 00 7F, 00 7F, 00 7F
00000080, 00 80, 00 80, 00 80, 00 80
000000BF, 00 BF, 00 BF, 00 BF, 00 BF
000000C0, 00 C0, 00 C0, 00 C0, 00 C0
000000FF, 00 FF, 00 FF, 00 FF, 00 FF
00000100, 01 00, 01 00, 01 00, 01 00
00003FFF, 3F FF, 3F FF, 3F FF, 3F FF
00004000, 40 00, 40 00, 40 00, 40 00
00007FFF, 7F FF, 7F FF, 7F FF, 7F FF
00008000, 80 00, 80 00, 80 00, 80 00
0000BFFF, BF FF, BF FF, BF FF, BF FF
0000C000, C0 00, C0 00, C0 00, C0 00
0000EFFF, EF FF, EF FF, EF FF, EF FF
0000F000, F0 00, F0 00, F0 00, F0 00
0000FFFF, FF FF, FF FF, FF FF, FF FF
0001F108, D8 3C DD 08, D8 3C DD 08, D8 3C DD 08, D8 3C DD 08
0001F132, D8 3C DD 32, D8 3C DD 32, D8 3C DD 32, D8 3C DD 32
0001F1A0, D8 3C DD A0, D8 3C DD A0, D8 3C DD A0, D8 3C DD A0

Notice that how characters in the 0x10000 - 0x10FFFF range are encoded into 32 bits, or 16-bit pairs.

The last Unicode encoding to look at is the UTF-32BE encoding:

C:\herong>java EncodingSampler2 UTF-32BE
UTF-32BE encoding:
Char,     String, Writer, Charset, Encoder
00000000, 00 00 00 00, 00 00 00 00, 00 00 00 00, 00 00 00 00
0000003F, 00 00 00 3F, 00 00 00 3F, 00 00 00 3F, 00 00 00 3F
00000040, 00 00 00 40, 00 00 00 40, 00 00 00 40, 00 00 00 40
0000007F, 00 00 00 7F, 00 00 00 7F, 00 00 00 7F, 00 00 00 7F
00000080, 00 00 00 80, 00 00 00 80, 00 00 00 80, 00 00 00 80
000000BF, 00 00 00 BF, 00 00 00 BF, 00 00 00 BF, 00 00 00 BF
000000C0, 00 00 00 C0, 00 00 00 C0, 00 00 00 C0, 00 00 00 C0
000000FF, 00 00 00 FF, 00 00 00 FF, 00 00 00 FF, 00 00 00 FF
00000100, 00 00 01 00, 00 00 01 00, 00 00 01 00, 00 00 01 00
00003FFF, 00 00 3F FF, 00 00 3F FF, 00 00 3F FF, 00 00 3F FF
00004000, 00 00 40 00, 00 00 40 00, 00 00 40 00, 00 00 40 00
00007FFF, 00 00 7F FF, 00 00 7F FF, 00 00 7F FF, 00 00 7F FF
00008000, 00 00 80 00, 00 00 80 00, 00 00 80 00, 00 00 80 00
0000BFFF, 00 00 BF FF, 00 00 BF FF, 00 00 BF FF, 00 00 BF FF
0000C000, 00 00 C0 00, 00 00 C0 00, 00 00 C0 00, 00 00 C0 00
0000EFFF, 00 00 EF FF, 00 00 EF FF, 00 00 EF FF, 00 00 EF FF
0000F000, 00 00 F0 00, 00 00 F0 00, 00 00 F0 00, 00 00 F0 00
0000FFFF, 00 00 FF FF, 00 00 FF FF, 00 00 FF FF, 00 00 FF FF
0001F108, 00 01 F1 08, 00 01 F1 08, 00 01 F1 08, 00 01 F1 08
0001F132, 00 01 F1 32, 00 01 F1 32, 00 01 F1 32, 00 01 F1 32
0001F1A0, 00 01 F1 A0, 00 01 F1 A0, 00 01 F1 A0, 00 01 F1 A0

Obviously, UTF-32BE encoding is much simpler and very easy to understand than UTF-16BE and UTF-8 encodings.

Table of Contents

 About This Book

 Character Sets and Encodings

 ASCII Character Set and Encoding

 GB2312 Character Set and Encoding

 GB18030 Character Set and Encoding

 JIS X0208 Character Set and Encodings

 Unicode Character Set

 UTF-8 (Unicode Transformation Format - 8-Bit)

 UTF-16, UTF-16BE and UTF-16LE Encodings

 UTF-32, UTF-32BE and UTF-32LE Encodings

 Python Language and Unicode Characters

 Java Language and Unicode Characters

►Character Encoding in Java

 What Is Character Encoding

 List of Supported Character Encodings in Java

 EncodingSampler.java - Testing encode() Methods

 Examples of CP1252 and ISO-8859-1 Encodings

►Examples of US-ASCII, UTF-8, UTF-16 and UTF-32 Encodings

 Examples of GB18030 Encoding

 Testing decode() Methods

 Character Set Encoding Maps

 Encoding Conversion Programs for Encoded Text Files

 Using Notepad as a Unicode Text Editor

 Using Microsoft Word as a Unicode Text Editor

 Using Microsoft Excel as a Unicode Text Editor

 Unicode Fonts

 Archived Tutorials

 References

 Full Version in PDF/EPUB

Examples of US-ASCII, UTF-8, UTF-16 and UTF-32 Encodings - Updated in 2024, by Herong Yang