Unicode Support on "str" Data Type

Unicode Tutorials - Herong's Tutorial Examples

∟Unicode Support on "str" Data Type

This section provides tutorial example on how to use the built-in data type 'str' to store Unicode characters as a list of code points.

When using Python built-in data type "str" to store Unicode characters, you need to remember that:

"str" represents a Unicode character string as a list of Unicode code points.
Each code point is stored as a byte sequence using the UTF-8 encoding.
"str" literals in Python source code are decoded using the default UTF-8 encoding.
"coding: <encoding-name>" or "coding=<encoding-name>" in comment lines can be used to change the source code default encoding.
Escape sequence of "\N{<code_name>}" in "str" literal can be used to specify a Unicode code point by its name.
Escape sequence of "\x.." in "str" literal can be used to specify a small Unicode code point by its 8-bit hexsdecimal value.
Escape sequence of "\u...." in "str" literal can be used to specify a medium Unicode code point by its 16-bit hexsdecimal value.
Escape sequence of "\U........" in "str" literal can be used to specify a large Unicode code point by its 32-bit hexsdecimal value.
len(str) built-in function returns the number Unicode code points of a given Unicode character strings.
The number of characters in a string may be less than the number of code points, because a single Unicode character may be represented with a combination of 2 code points.
ord(char) built-in function returns the decimal value of the Unicode code point of a single character string.
chr(int) built-in function returns a single character string represented by the given decimal value of a Unicode code point.

Here is a Python script that demonstrates how to use data type "str" to store Unicode character strings:

# str-Data-Type-on-Unicode.py
# Copyright 2019 (c) HerongYang.com. All Rights Reserved.
#
# "str" literals are decoded as UTF-8 encoded bytes by default
str = "Français"

# len(str) returns the number of code points in a "str"
num = len(str)

# a "str" can be accessed as a list of Unicode code points 
for i in range(num):
  char = str[i]
  code = ord(char)
  print("{0} - {1}".format(hex(code), code))

# \N{}, \x, \u and \U escape sequences can be used in "str" literals 
print("Français")
print("Fran\N{LATIN SMALL LETTER C WITH CEDILLA}ais")
print("Fran\xe7ais")
print("Fran\u00e7ais")
print("Fran\U000000e7ais")

# one character can be constructed with 2 combining code points
one = "\N{LATIN SMALL LETTER C WITH CEDILLA}"
two = "\N{LATIN SMALL LETTER C}\N{COMBINING CEDILLA}"
print("{0} with {1} code point".format(one, len(one)))
print("{0} with {1} code points".format(two, len(two)))

Run the above script, it will print the following output:

herong$ python3 str-Data-Type-on-Unicode.py 

0x46 - 70
0x72 - 114
0x61 - 97
0x6e - 110
0xe7 - 231
0x61 - 97
0x69 - 105
0x73 - 115
Français
Français
Français
Français
Français
ç with 1 code points
ç with 2 code points