UTF-8 vs UTF-16 vs ASCII: Character Encoding Explained for Developers

Introduction

Computers do not store text. They store numbers. A character encoding is the agreed-upon contract that says which numbers stand for which characters, and how those numbers get packed into bytes. When the writer and the reader use the same contract, text round-trips perfectly. When they disagree by even a little, you get the broken output every developer has cursed at.

The confusing part is that there are two separate questions hiding inside the word "encoding," and most explanations smush them together. The first question is which number represents a character — that is what Unicode answers. The second is how that number is laid out as bytes — that is what UTF-8, UTF-16, and UTF-32 answer. Keep those two layers separate in your head and the whole topic stops being scary.

Encoding is a sibling of Base64 encoding, which solves a related-but-different problem: getting binary data safely through text-only channels. If Base64 is "binary to text," character encoding is "text to binary." Understanding both is what stops you from guessing.

Where It Started: ASCII

ASCII — the American Standard Code for Information Interchange, standardized in 1963 — is where everything began. It defines exactly 128 characters and assigns each one a number from 0 to 127. The uppercase letter A is 65. The digit 0 is 48. A space is 32. The first 32 values are non-printing control codes — newline, tab, carriage return, the bell character that used to make terminals beep.

Because the highest ASCII value is 127, every character fits in 7 bits. Computers work in 8-bit bytes, so ASCII text wastes one bit per character — the high bit is always zero. That spare bit turned out to matter enormously later, but in 1963 nobody minded.

ASCII has one quality worth admiring: it is dead simple. One character, one byte, no ambiguity. Sorting strings is just sorting numbers. strlen counts bytes and that is also the character count. For decades, English-speaking programmers could treat "byte" and "character" as the same word and never get hurt. That assumption is the original sin behind half the encoding bugs that exist today.

The Code Page Chaos

ASCII covers unaccented English and nothing else. No é, no ñ, no ü, no ß, and certainly no Greek, Cyrillic, Arabic, Hebrew, or Chinese. The rest of the world needed those characters, and the spare high bit gave them 128 extra slots (values 128–255) to play with.

The problem: 128 slots is nowhere near enough for every language, so the industry invented dozens of incompatible mappings for that upper range. These were called code pages or extended ASCII. ISO-8859-1 (Latin-1) put Western European accents up there. ISO-8859-5 used the same byte values for Cyrillic. Windows-1252 was Microsoft's slightly different take on Latin-1. The DOS code page 437 had box-drawing characters.

Byte value 233 meant é in Latin-1, щ in ISO-8859-5, and something else again in a Greek code page. A document was only readable if you happened to know which code page it was written in, and that information was almost never stored with the file. Open a Russian text file with a Western code page and you got a screen of nonsense. This is the original mojibake — a Japanese word for garbled text that became the universal term for it.

The world needed one numbering scheme big enough for every character in every language, used by everyone. That is the job Unicode was created to do.

Unicode: Characters vs Code Points

Here is the single most important sentence in this article: Unicode is not an encoding. Unicode is a giant catalog. It assigns every character a unique number called a code point, and that is all it does. How that number becomes bytes is a separate decision — and that decision is UTF-8, UTF-16, or UTF-32.

Code points are written as U+ followed by hexadecimal. The letter A is U+0041 (65, same as ASCII — Unicode deliberately kept the first 128 code points identical to ASCII). The euro sign € is U+20AC. The grinning face emoji 😀 is U+1F600. Today Unicode defines well over 150,000 characters and has room for 1,114,112 code points in total, numbered U+0000 through U+10FFFF.

Two pieces of vocabulary save you from a lot of confusion later:

• The Basic Multilingual Plane (BMP). Code points U+0000 to U+FFFF — the first 65,536. This holds almost every character in modern living languages. Most text you will ever process lives entirely in the BMP.
• Supplementary characters. Code points U+10000 and above — emoji, historic scripts, rare CJK ideographs, mathematical symbols. They are the reason encodings need an "escape hatch" for big numbers, and the reason emoji break naive code.

So when you save a file, two things happen. First, each character is mapped to its code point — that is Unicode's job, and it is fixed. Second, each code point is serialized into actual bytes — that is the encoding's job, and you get to choose. Everything below is about that second step.

How UTF-8 Encodes

UTF-8 is a variable-width encoding: a code point becomes 1, 2, 3, or 4 bytes depending on how big it is. It was designed by Ken Thompson and Rob Pike in 1992, and its cleverness is hard to overstate.

The x bits hold the actual code point; the fixed bits are routing information. Look at what the leading bits do. A byte starting with 0 is a standalone single-byte character. A byte starting with 11 is the start of a multi-byte sequence, and the count of leading ones tells you how many bytes follow. A byte starting with 10 is a continuation byte — it is never the start of anything.

That design has two beautiful consequences. First, the first 128 code points encode as a single byte identical to ASCII. Every valid ASCII file is already a valid UTF-8 file, byte for byte. That backward compatibility is why adopting UTF-8 cost the world almost nothing. Second, the encoding is self-synchronizing: if you drop into the middle of a UTF-8 stream, you can find the next character boundary by skipping forward until you see a byte that does not start with 10. A corrupted byte damages one character, not the entire rest of the file.

Let's encode the euro sign €, code point U+20AC. In binary that is 0010 0000 1010 1100. It falls in the 3-byte range, so we slot those 16 bits into the 1110xxxx 10xxxxxx 10xxxxxx template — taking 4 bits, then 6, then 6:

Code point:  U+20AC  =  0010 0000 1010 1100
Template:    1110xxxx  10xxxxxx  10xxxxxx
Filled in:   1110 0010 10 000010 10 101100
Bytes:       0xE2      0x82      0xAC

So € is the three bytes E2 82 AC in UTF-8. That is exactly what you would see if you ran the bytes through a hex viewer.

How UTF-16 and Surrogate Pairs Work

UTF-16 is also variable-width, but its unit is a 16-bit code unit rather than an 8-bit byte. Every BMP character (U+0000 to U+FFFF) fits in exactly one 16-bit code unit. Simple — the code point is the code unit.

The trouble starts above U+FFFF, because those code points do not fit in 16 bits. UTF-16 handles them with a mechanism called surrogate pairs. Unicode reserves a block of code points, U+D800 to U+DFFF, that are deliberately not assigned to any character. They exist only to be used in pairs: a high surrogate (U+D800–U+DBFF) followed by a low surrogate (U+DC00–U+DFFF). Together a pair encodes one supplementary character.

Encoding the grinning face 😀 (U+1F600) shows the algorithm:

1. Subtract 0x10000:   0x1F600 - 0x10000 = 0xF600
2. 0xF600 as 20 bits:  0000111101 1000000000
3. High 10 bits (0x3D) -> 0xD800 + 0x3D = 0xD83D
4. Low  10 bits (0x200) -> 0xDC00 + 0x200 = 0xDE00
Result: the pair  D83D DE00

So in UTF-16 the single emoji 😀 is the two code units D83D DE00 — four bytes. This is the heart of the "emoji breaks my code" problem we will get to shortly. It is also why the surrogate range can never represent a real character: those code points are permanently spoken for.

UTF-16 has one more wrinkle: byte order. A 16-bit value can be stored most-significant-byte-first (big-endian) or least-significant-byte-first (little-endian). The code unit 0x0041 is the bytes 00 41 in big-endian and 41 00 in little-endian. That ambiguity does not exist in UTF-8, where the byte order is fixed by the spec. UTF-16 files resolve it with a Byte Order Mark, covered next.

UTF-32 and the BOM

UTF-32 is the brute-force option: every code point gets exactly 4 bytes, always. No variable width, no surrogate pairs, no special cases. The code point is the stored value, zero-padded to 32 bits.

The appeal is that indexing is trivial — the Nth character is at byte offset N × 4, full stop. The cost is brutal: a plain English document in UTF-32 is four times larger than in ASCII or UTF-8, since every character carries three padding bytes. That is why UTF-32 almost never appears in files or on the wire. It shows up only as an in-memory representation inside some language runtimes, where the constant-time indexing is worth the memory.

The Byte Order Mark (BOM) is a special code point, U+FEFF, optionally placed at the very start of a file to declare its encoding and byte order. Each encoding has a distinctive BOM byte sequence:

For UTF-16 the BOM is genuinely useful — it tells the reader which byte order to expect. For UTF-8 the BOM is almost always a mistake. UTF-8 has no byte-order ambiguity, so the BOM carries no information, and those three stray bytes routinely break things: a shell script with a UTF-8 BOM fails because the #! shebang is no longer the first byte, a PHP file emits whitespace before its headers, a CSV shows a phantom  in the first cell. If a tool offers "UTF-8" and "UTF-8 with BOM," pick plain UTF-8 unless something specifically demands the BOM.

The Same String, Three Encodings

Theory clicks when you see the bytes. Take a deliberately mixed three-character string: A€😀 — one plain ASCII letter, one BMP symbol, and one supplementary emoji. Here is exactly what each encoding produces, byte for byte.

Total size for this three-character string: 8 bytes in UTF-8, 8 bytes in UTF-16, 12 bytes in UTF-32. Three takeaways jump out of that table:

• The A is one byte in UTF-8 and four bytes in UTF-32. For ASCII-heavy text — code, JSON, English prose, HTML markup — UTF-8 is dramatically more compact.
• The emoji is four bytes in every encoding. There is no free lunch above the BMP.
• In UTF-16 LE the emoji's bytes are 3D D8 00 DE — the surrogate pair D83D DE00 with each code unit's bytes swapped for little-endian. Read it without knowing the byte order and you would decode the wrong code points entirely.

Three characters do not become "three of anything" in a predictable way. They become 8 or 12 bytes, or — as the next section shows — a wildly variable number of "string units" depending on what your language thinks a character is.

Why Emoji Break String Length

Ask three programming languages for the length of the string "😀" and you can get three different answers. None of them is wrong; they are measuring different things.

// JavaScript — strings are UTF-16 code units
"😀".length              // 2   (the surrogate pair counts as two)
[..."😀"].length         // 1   (the spread operator iterates code points)

# Python 3 — strings are sequences of code points
len("😀")                # 1

// Go — strings are UTF-8 byte sequences
len("😀")                // 4   (the byte count)
utf8.RuneCountInString("😀") // 1

JavaScript says 2 because its strings are sequences of UTF-16 code units, and the emoji is a surrogate pair. Python 3 says 1 because its strings are sequences of code points. Go says 4 because a Go string is a sequence of bytes and the emoji is four UTF-8 bytes. The lesson is not "Go is wrong" — it is that "length of a string" is an ambiguous question until you say which unit you mean: bytes, code units, or code points.

This is exactly how a "max 20 characters" username field gets exploited or accidentally broken. If the limit is enforced as a JavaScript .length check, a user can fit only ten emoji into it — or, worse, a string truncated at 20 code units can be sliced straight through the middle of a surrogate pair, leaving a lone half-character that some downstream system rejects as invalid.

And it gets one layer deeper. What a human calls "one character" is a grapheme cluster, which can be several code points. The flag emoji 🇯🇵 is two code points (regional indicators J and P). A family emoji can be five or more code points joined by invisible zero-width joiners. Skin-tone variants add a modifier code point. So even "count the code points" is not the same as "count what the user sees." For anything user-facing — truncating a display name, validating input length — use a library that segments by grapheme cluster (the Intl.Segmenter API in modern JavaScript, the regex module in Python). Counting bytes or code units will eventually mangle someone's name.

Why UTF-8 Won the Web

As of today UTF-8 is used by well over 98% of all web pages. That dominance is not an accident or a popularity contest — it is the direct result of design decisions.

• ASCII compatibility. Every existing ASCII document, every ASCII protocol, every ASCII config file was already valid UTF-8. Migration required no rewrite. UTF-16 has no such property — an ASCII file is not valid UTF-16.
• Compactness for real-world text. HTML tags, JSON keys, URLs, source code, and English prose are overwhelmingly ASCII, and UTF-8 stores each of those characters in a single byte. UTF-16 would double the size of all of it for no benefit.
• No byte-order problem. UTF-8's byte order is fixed by the specification. There is no big-endian vs little-endian UTF-8, so there is no BOM to negotiate and no way to misread the stream.
• Robustness. Because UTF-8 is self-synchronizing, a single corrupted or lost byte damages exactly one character. In UTF-16, losing one byte shifts the parity of every following code unit and turns the rest of the file into garbage.
• No illegal-byte ambiguity. UTF-8 has byte values and sequences that are simply invalid, which makes it easy to detect when data is not UTF-8. That detectability is a quiet but real advantage.

UTF-16 still has a home: it is the in-memory string representation in JavaScript engines, the JVM, the .NET runtime, and the Windows API, mostly for historical reasons dating to when Unicode fit in 16 bits and surrogate pairs did not exist yet. But for files, network protocols, databases, and APIs, the answer in 2026 is simple: use UTF-8 everywhere, declare it explicitly, and stop thinking about it.

The Gotchas That Bite in Production

1. Mojibake — the classic mismatch

Mojibake is what you get when bytes are decoded with the wrong encoding. The textbook case: text saved as UTF-8 but read as Windows-1252. The name José contains é, which is the two UTF-8 bytes C3 A9. Read those two bytes through Windows-1252 and you get two separate characters — Ã (byte C3) and © (byte A9) — so José displays as José. If you ever see a stray Ã in front of accented letters, you are looking at UTF-8 bytes decoded as Latin-1. The fix is never to "replace the bad characters" — it is to decode with the correct encoding in the first place.

2. Double encoding

Double encoding happens when already-encoded text is encoded a second time. A string is converted to UTF-8 bytes, then those bytes are mistakenly treated as Latin-1 characters and converted to UTF-8 again. Now é has become é stored as four bytes. It often survives one display cycle and looks merely "a bit off," then gets worse every time the data passes through another buggy layer. The cause is almost always a pipeline where one stage assumes input is raw text when it is actually already-encoded bytes. The fix is to pin down, at every boundary, whether you are holding decoded text or encoded bytes — and never to call an encode function on something that is already encoded.

3. MySQL's fake utf8

This one has burned a generation of developers. For years, MySQL's character set named utf8 was not real UTF-8. It silently capped each character at three bytes, which means it could store the entire BMP but nothing above it. The moment a user typed an emoji, the insert failed with an "incorrect string value" error, or the data was truncated. Real, four-byte-capable UTF-8 in MySQL is the character set named utf8mb4 ("mb4" = maximum four bytes). If you are on MySQL or MariaDB, your tables, columns, and connection should all use utf8mb4. Treat plain utf8 as a legacy trap.

4. Missing or wrong charset declarations

Bytes do not carry their own encoding. A file is just bytes; the encoding is metadata that must be communicated separately. On the web that means an HTTP Content-Type: text/html; charset=utf-8 header and a <meta charset="utf-8"> tag. For a database it means the connection charset, not just the column charset. For an API it means the Content-Type on both request and response. When you omit the declaration, every consumer falls back to a guess, and guesses differ by browser, operating system, and locale. Always declare the encoding explicitly — the single most effective habit for never seeing mojibake again.

5. Truncating by bytes instead of characters

Slicing a UTF-8 string at an arbitrary byte offset — to fit a database column, a fixed-width field, or a preview snippet — can cut straight through the middle of a multi-byte character. The result is an incomplete byte sequence that renders as a replacement character � or throws a decode error downstream. If you must truncate, truncate on character or grapheme boundaries, or size the limit in bytes and back off to the nearest valid boundary. The same caution applies when chunking a stream: a character can straddle a chunk boundary, so a chunk-by-chunk decoder must buffer the incomplete tail.

A practical way to build intuition here: take a string with accented characters and emoji, run it through a tool that shows you the raw bytes, and watch the byte count diverge from the character count. The same instinct that helps you reason about a JSON payload's real size — try the JSON editor with some emoji in a string value — applies directly to encoding.

Common Questions

Is UTF-8 the same as Unicode?

No. Unicode is the catalog that assigns every character a code point number. UTF-8 is one of several ways to turn those code point numbers into bytes. UTF-16 and UTF-32 are the other two standard ways. "Unicode" answers which number; "UTF-8" answers which bytes.

Is ASCII still relevant?

Yes, in the sense that ASCII is a perfect subset of UTF-8. The first 128 Unicode code points are identical to ASCII, and they encode as a single identical byte in UTF-8. You rarely choose "ASCII" as an encoding deliberately anymore, but every UTF-8 system handles ASCII text natively because the two overlap exactly.

Should I ever use UTF-16 or UTF-32 for files?

Almost never. For files, network payloads, databases, and APIs, UTF-8 is the right default in essentially every case. UTF-16 mostly survives as an in-memory representation inside language runtimes (JavaScript, Java, .NET) and the Windows API. UTF-32 is occasionally used in memory when constant-time indexing by code point is worth quadrupling the memory cost. Neither belongs in a file you control.

Why does my emoji show up as two characters?

Because the language or tool is counting UTF-16 code units, and emoji above U+FFFF are stored as a surrogate pair — two code units. JavaScript's String.prototype.length does exactly this. To count what a user perceives as one character, iterate code points (the spread operator, or Intl.Segmenter for full grapheme clusters) instead of reading .length.

What is that � replacement character?

U+FFFD is the official Unicode "replacement character." A decoder emits it when it hits a byte sequence that is invalid for the encoding it was told to use — corrupted data, a wrong-encoding mismatch, or a string truncated mid-character. Seeing it means the bytes and the declared encoding disagree. It is a symptom; the cure is upstream, where the wrong encoding was assumed.

How do I detect a file's encoding?

You cannot do it with certainty — encoding is metadata, not something stored inside the bytes. A BOM, if present, is a strong hint. Heuristic libraries (chardet and its relatives) make educated guesses based on byte patterns and are often right but never guaranteed. The real answer is not to detect: communicate the encoding explicitly through headers, metadata, or a documented convention, and default everything to UTF-8.

Wrapping Up

Character encoding stops being mysterious the moment you separate the two layers. Unicode assigns every character a code point — one fixed number, no choices to make. UTF-8, UTF-16, and UTF-32 are three different ways to serialize those numbers into bytes, and for almost everything you will ever build, UTF-8 is the correct choice: ASCII-compatible, compact for real text, byte-order-free, and self-synchronizing.

Every encoding bug in the introduction — José, the broken CSV, the emoji that crashed a length check, the boxes in the PDF — comes down to one of two failures: text encoded with one contract and decoded with another, or code that confused bytes with code units with characters. Declare your encoding explicitly at every boundary, use utf8mb4 in MySQL, count the right unit when you measure or truncate strings, and those bugs simply stop happening.

If you found this useful, the natural next reads are the Base64 encoding guide for the "binary to text" side of the story, and MD5 vs SHA-256 vs SHA-512 for how raw bytes get fingerprinted. And if you want to watch byte counts diverge from character counts in real time, paste some accented text and emoji into the JSON editor — it runs entirely in your browser.