Unicode Tutorials - Herong's Tutorial Examples - v5.32, by Herong Yang
Character Set Encoding Map Analyzer
This section provides a tutorial example on how to write a simple program to analyze and print out the encoding map showing relations between character code points and their encoded byte sequences of a given encoding.
As mentioned in the previous chapter, Java 11 supports 171 built-in character set encodings.
In order to figure out the encoding map (relations between character code points and their encoded byte sequences) of a specific supported encoding, I wrote the following program to analyze a given encoding and print a map between code points (from 0x0000 to 0x10FFFF) and their encoded byte sequences:
/* EncodingAnalyzer2.java * Copyright (c) 2019 HerongYang.com. All Rights Reserved. */ import java.io.*; class EncodingAnalyzer2 { static char hexDigit[] = {'0', '1', '2', '3', '4', '5', '6', '7', '8', '9', 'A', 'B', 'C', 'D', 'E', 'F'}; public static void main(String[] a) { String charset = null; if (a.length>0) charset = a[0]; if (charset==null) System.out.println("Default encoding:"); else System.out.println(charset+" encoding:"); int lastByte = 0; int lastLength = 0; byte[] startSequence = null; int startChar = 0; byte[] endSequence = null; int endChar = 0; boolean isFirstChar = true; for (int i=0; i<=0x10FFFF; i++) { int c = i; String s = new String(Character.toChars(c)); byte[] b = null; if (charset==null) { b = s.getBytes(); } else { try { b = s.getBytes(charset); } catch (UnsupportedEncodingException e) { System.out.println(e.toString()); break; } } int l = b.length; int lb = ((int) b[l-1]) & 0x00FF; if (isFirstChar==true) { isFirstChar = false; startSequence = b; startChar = c; lastByte = lb - 1; lastLength = l; } if (!(l==lastLength && (lb==lastByte+1 || lb==lastByte))) { System.out.print(intToHex(startChar)+" >"); printBytes(startSequence); System.out.print(" - "+intToHex(endChar)+" >"); printBytes(endSequence); if (lastByte==0x3F) System.out.print(": Invalid range"); System.out.println(""); startSequence = b; startChar = c; } endSequence = b; endChar = c; lastLength = l; lastByte = lb; } System.out.print(intToHex(startChar)+" >"); printBytes(startSequence); System.out.print(" - "+intToHex(endChar)+" >"); printBytes(endSequence); if (lastByte==0x3F) System.out.print(": Invalid range"); System.out.println(""); } public static void printBytes(byte[] b) { for (int j=0; j<b.length; j++) System.out.print(" "+byteToHex(b[j])); } public static String byteToHex(byte b) { char[] a = { hexDigit[(b >> 4) & 0x0f], hexDigit[b & 0x0f] }; return new String(a); } public static String charToHex(char c) { byte hi = (byte) (c >>> 8); byte lo = (byte) (c & 0xff); return byteToHex(hi) + byteToHex(lo); } public static String intToHex(int i) { char hi = (char) (i >>> 16); char lo = (char) (i & 0xffff); return charToHex(hi) + charToHex(lo); } }
Note that:
The output of this program will be discussed in sections bellow.
Table of Contents
ASCII Character Set and Encoding
GB2312 Character Set and Encoding
GB18030 Character Set and Encoding
JIS X0208 Character Set and Encodings
UTF-8 (Unicode Transformation Format - 8-Bit)
UTF-16, UTF-16BE and UTF-16LE Encodings
UTF-32, UTF-32BE and UTF-32LE Encodings
Python Language and Unicode Characters
Java Language and Unicode Characters
►Character Set Encoding Map Analyzer
Character Set Encoding Maps - US-ASCII and ISO-8859-1/Latin 1
Character Set Encoding Maps - CP1252/Windows-1252
Character Set Encoding Maps - Unicode UTF-8
Character Set Encoding Maps - Unicode UTF-16, UTF-16BE, UTF-16LE
Character Set Encoding Maps - Unicode UTF-32, UTF-32BE, UTF-32LE
Character Counter Program for Any Given Encoding
Character Set Encoding Comparison
Encoding Conversion Programs for Encoded Text Files
Using Notepad as a Unicode Text Editor
Using Microsoft Word as a Unicode Text Editor