UTF-8

UTF-8 is a variable-width character encoding used for electronic communication. Defined by the Unicode Standard, the name is derived from Unicode (or Universal Coded Character Set) Transformation Format – 8-bit.^[1]

UTF-8 is capable of encoding all 1,112,064^{[nb 1]} valid character code points in Unicode using one to four one-byte (8-bit) code units. Code points with lower numerical values, which tend to occur more frequently, are encoded using fewer bytes. It was designed for backward compatibility with ASCII: the first 128 characters of Unicode, which correspond one-to-one with ASCII, are encoded using a single byte with the same binary value as ASCII, so that valid ASCII text is valid UTF-8-encoded Unicode as well. Since ASCII bytes do not occur when encoding non-ASCII code points into UTF-8, UTF-8 is safe to use within most programming and document languages that interpret certain ASCII characters in a special way, such as / (slash) in filenames, \ (backslash) in escape sequences, and % in printf.

UTF-8 was designed as a superior alternative to UTF-1, a proposed variable-width encoding with partial ASCII compatibility which lacked some features including self-synchronization and fully ASCII-compatible handling of characters such as slashes. Ken Thompson and Rob Pike produced the first implementation for the Plan 9 operating system in September 1992.^[2][3] This led to its adoption by X/Open as its specification for FSS-UTF, which would first be officially presented at USENIX in January 1993 and subsequently adopted by the Internet Engineering Task Force (IETF) in RFC 2277 (BCP 18) for future Internet standards work, replacing Single Byte Character Sets such as Latin-1 in older RFCs.

UTF-8 is by far the most common encoding for the World Wide Web, accounting for over 97% of all web pages, and up to 100% for some languages, as of 2021.^[4]

Naming[]

The official Internet Assigned Numbers Authority (IANA) code for the encoding is "UTF-8".^[5] All letters are upper-case, and the name is hyphenated. This spelling is used in all the Unicode Consortium documents relating to the encoding.

Alternatively, the name "utf-8" may be used by all standards conforming to the IANA list (which include CSS, HTML, XML, and HTTP headers),^[6] as the declaration is case insensitive.^[5]

Other variants, such as those that omit the hyphen or replace it with a space, i.e. "utf8" or "UTF 8", are not accepted as correct by the governing standards.^[7] Despite this, most web browsers can understand them, and so standards intended to describe existing practice (such as HTML5) may effectively require their recognition.^[8]

Unofficially, UTF-8-BOM and UTF-8-NOBOM are sometimes used for text files which contain or don't contain a byte order mark (BOM), respectively.^{[citation needed]} In Japan especially, UTF-8 encoding without a BOM is sometimes called "UTF-8N".^[9][10]

Windows XP and later, including all supported Windows versions, have codepage 65001, as a synonym for UTF-8 (since Windows 7 support for UTF-8 is better),^[11] and Microsoft has a script for Windows 10, to enable it by default for its program Microsoft Notepad.^[12]

In PCL, UTF-8 is called Symbol-ID "18N" (PCL supports 183 character encodings, called Symbol Sets, which potentially could be reduced to one, 18N, that is UTF-8).^[13]

Encoding[]

Since the restriction of the Unicode code-space to 21-bit values in 2003, UTF-8 is defined to encode code points in one to four bytes, depending on the number of significant bits in the numerical value of the code point. The following table shows the structure of the encoding. The x characters are replaced by the bits of the code point.

The first 128 characters (US-ASCII) need one byte. The next 1,920 characters need two bytes to encode, which covers the remainder of almost all Latin-script alphabets, and also IPA extensions, Greek, Cyrillic, Coptic, Armenian, Hebrew, Arabic, Syriac, Thaana and N'Ko alphabets, as well as Combining Diacritical Marks. Three bytes are needed for characters in the rest of the Basic Multilingual Plane, which contains virtually all characters in common use,^[14] including most Chinese, Japanese and Korean characters. Four bytes are needed for characters in the other planes of Unicode, which include less common CJK characters, various historic scripts, mathematical symbols, and emoji (pictographic symbols).

A "character" can actually take more than 4 bytes, e.g. an emoji flag character takes 8 bytes since it's "constructed from a pair of Unicode scalar values".^[15]

Examples[]

Consider the encoding of the Euro sign, €:

1. The Unicode code point for "€" is U+20AC.
2. As this code point lies between U+0800 and U+FFFF, this will take three bytes to encode.
3. Hexadecimal 20AC is binary 0010 0000 1010 1100. The two leading zeros are added because a three-byte encoding needs exactly sixteen bits from the code point.
4. Because the encoding will be three bytes long, its leading byte starts with three 1s, then a 0 (1110...)
5. The four most significant bits of the code point are stored in the remaining low order four bits of this byte (11100010), leaving 12 bits of the code point yet to be encoded (...0000 1010 1100).
6. All continuation bytes contain exactly six bits from the code point. So the next six bits of the code point are stored in the low order six bits of the next byte, and 10 is stored in the high order two bits to mark it as a continuation byte (so 10000010).
7. Finally the last six bits of the code point are stored in the low order six bits of the final byte, and again 10 is stored in the high order two bits (10101100).

The three bytes 11100010 10000010 10101100 can be more concisely written in hexadecimal, as E2 82 AC.

The following table summarises this conversion, as well as others with different lengths in UTF-8. The colors indicate how bits from the code point are distributed among the UTF-8 bytes. Additional bits added by the UTF-8 encoding process are shown in black.