str.lower() looks too ordinary to be dangerous. It takes text, returns lowercase text, and appears in codebases everywhere. Yet inside a standards implementation, even a familiar string method can cross a security boundary.

That happened in Python’s implementation of Stringprep, the text-preparation framework used by the original internationalized domain-name standard. The protocol was built from Unicode 3.2 data published in 2002. Python kept the old database needed by the protocol, but one fallback in the case-folding path called the ordinary str.lower() method. That method follows the Unicode database bundled with the running interpreter.

The result was a split-brain algorithm: most of the implementation lived in 2003, while one operation silently moved forward whenever Python adopted a newer Unicode release.

This is not the usual Unicode warning about characters that merely look alike. The deeper problem is version drift inside a supposedly deterministic protocol. If a stored identifier, certificate rule, allowlist, cache key, or network destination is prepared differently on two machines, those machines may disagree about what resource a user meant.

Domain names need a bridge between Unicode and ASCII

The Domain Name System was designed around a narrow set of ASCII characters. Human language is not. Internationalized Domain Names in Applications, or IDNA, bridge that gap by transforming a Unicode label into an ASCII form that starts with xn--. The reversible encoding at the end is Punycode.

Punycode alone is not enough. Before encoding, an application must decide which inputs should compare as equivalent, which characters should disappear, which are forbidden, and how bidirectional text should behave. IDNA 2003 delegated that preparation to Nameprep, a profile of the more general Stringprep framework in RFC 3454.

Stringprep describes a pipeline:

1. Map selected characters, including case mappings.
2. Normalize the result with Unicode NFKC when the profile requires it.
3. Reject prohibited output.
4. Apply bidirectional-text rules.
5. Pass the prepared label to the protocol-specific encoder.

Every stage is part of the meaning of the resulting identifier. Changing one is not a harmless implementation improvement. It changes the protocol.

The standard deliberately froze Unicode at version 3.2

Unicode evolves. New scripts are added, previously unassigned code points acquire characters, and properties can be refined. Casing is data-driven too. A character that had no lowercase partner in one release can gain one later.

That evolution is healthy for general text processing. It is dangerous when a wire protocol expects a fixed answer.

RFC 3454 therefore did not say “use whatever Unicode knows today.” Its tables were generated from Unicode 3.2. Appendix B records the mappings required for case folding, while other appendices record unassigned and prohibited code points. Two independent implementations using those tables should produce the same prepared string even if one runs decades after the other.

Python supports this requirement in an unusually explicit way. The standard library exposes the interpreter’s current Unicode database through unicodedata, but it also ships unicodedata.ucd_3_2_0. The legacy database exists specifically for protocols such as Stringprep and IDNA 2003.

You can see the two clocks in one interpreter:

import unicodedata

print(unicodedata.unidata_version)       # for example, "15.1.0"
print(unicodedata.ucd_3_2_0.unidata_version)  # "3.2.0"

Both databases are correct. They answer different questions:

• The current database answers, “What does this character mean under the Unicode version supported by this Python runtime?”
• The legacy database answers, “What did Unicode 3.2 say, as required by this old protocol?”

The vulnerability appeared because one code path asked the first question when the protocol required the second.

A convenient fallback escaped the frozen database

Stringprep’s B.3 table contains case-folding exceptions. Python represented those exceptions in a lookup and used a fallback for every code point not present in the table. In simplified form, the code looked like this:

def map_table_b3(character):
    replacement = b3_exceptions.get(ord(character))
    if replacement is not None:
        return replacement
    return character.lower()

The lookup was frozen. The fallback was not.

At the time the table was created, calling lowercase for an ordinary character looked equivalent to storing every identity or simple lowercase mapping explicitly. Years later, character.lower() had learned about characters and casing relationships that Unicode 3.2 did not contain. The shortcut stopped being equivalent to the table it was meant to implement.

Cherokee makes the failure concrete. Unicode originally encoded Cherokee letters in an uppercase-like form. Lowercase Cherokee letters were added much later. Under a modern Unicode database, the Cherokee letter Ꭰ (U+13A0) lowercases to ꭰ (U+AB70). Under Unicode 3.2, U+AB70 was not the lowercase counterpart used by Stringprep; it was outside the frozen repertoire.

On an affected Python runtime, this happens:

label = "ᎠᎠ"

print(label.lower())
# ꭰꭰ

print(label.encode("idna"))
# b'xn--kz9aa'   # affected behavior

The RFC-bound result preserves the Unicode 3.2 interpretation and encodes the original pair as:

xn--58da

The difference is not cosmetic. Those ASCII labels name different DNS locations. A runtime upgrade can therefore change which hostname an application contacts even though the source Unicode string is unchanged.

Why this crosses a security boundary

Not every standards mismatch is exploitable. The risk appears when software makes a security decision on one representation and performs an action on another.

Consider a service that accepts a Unicode hostname and applies an allowlist before making an outbound request. The validator runs in one process or container; the network client runs in another. If their Python versions carry different Unicode data, they can derive different ASCII labels from the same input.

The same pattern can affect:

• URL allowlists and blocklists;
• certificate hostname checks;
• proxy routing and egress policy;
• cookie and origin boundaries;
• caches indexed by normalized hostnames;
• database uniqueness constraints;
• signed records that are normalized again during verification.

The dangerous shape is a check/use split:

Unicode input
    ├── validator with mapping version A → identifier A → allowed
    └── connector with mapping version B → identifier B → contacted

Even when every component is locally consistent, the system is not. The bug lives in the relationship between them.

Stored data adds a second failure mode. Suppose an application canonicalizes a username or hostname, stores only the canonical value, and later recomputes that value after an upgrade. A new mapping can create collisions, orphan old records, or cause lookups to reach a different row. Unicode upgrades then behave like schema migrations, except the schema change is hidden inside a library call.

This is different from homoglyph spoofing

Unicode security discussions often begin with confusables: a Cyrillic character that resembles a Latin one, mixed scripts in an identifier, or invisible formatting characters. Those are real concerns, and Unicode Technical Standard #39 provides restriction levels and confusable-skeleton mechanisms to help detect them.

Version drift is a different class of failure:

• Confusable text gives different code points similar visual appearances.
• Normalization differences give canonically or compatibly related sequences inconsistent binary forms.
• Case-folding differences change which strings compare without case distinctions.
• Version drift makes any of those transformations depend on the library or runtime version.

Solving only the visual problem does not fix the protocol problem. A string can contain no deceptive glyphs and still map differently because two components use different Unicode data. Likewise, casefolding is not a universal security sanitizer. Python’s str.casefold() is designed for caseless matching under the runtime’s current Unicode version. It is more aggressive than lower()—for example, it maps German ß to ss—but it is still the wrong tool when a protocol mandates an older, fixed mapping table.

The correct operation depends on the contract, not on which string method sounds strongest.

The repair makes the protocol data complete again

The CPython fix did not freeze str.lower() globally. General Python code should continue receiving current Unicode behavior. Instead, the repair regenerated Stringprep’s data so that every code point whose modern lowercase behavior differs from Unicode 3.2 gets an explicit protocol-specific mapping.

Conceptually, the corrected function becomes:

def map_table_b3(character):
    replacement = protocol_exceptions.get(ord(character))
    if replacement is not None:
        return replacement
    return character.lower()  # safe only after all version differences are covered

The important part is how protocol_exceptions is built. CPython’s generation tooling walks the Unicode code-point space, compares the modern behavior with the Unicode 3.2 database and the RFC tables, and records the differences. Regression tests then pin known examples including Cherokee, Georgian, Cyrillic, and Roman numeral characters.

The merged CPython change also tightened property handling for code points outside RFC 3454. The accompanying security note states the intended invariant plainly: stringprep and the built-in encodings.idna codec must not consider Unicode attributes beyond those defined by the RFC.

This repair pattern is broadly useful. When an old specification and a living database disagree, isolate the compatibility behavior in generated, reviewable data. Do not weaken the modern API for every caller.

IDNA 2003 and IDNA 2008 are different protocols

It is tempting to respond by replacing every call to Python’s built-in idna codec with a newer method. For new applications, moving away from IDNA 2003 is usually the right direction—but migration must be explicit.

Python’s built-in:

hostname.encode("idna")

implements IDNA 2003. The third-party idna package implements the newer IDNA 2008 family and Unicode Technical Standard #46 processing. The standards differ in character validity, mappings, and compatibility behavior. Swapping implementations can change accepted input and output, so it deserves tests and a data-migration plan rather than a blind dependency edit.

IDNA 2008 largely moved away from Stringprep’s model of embedding a snapshot of Unicode mappings. Its rules are designed to work with evolving Unicode properties while preserving stronger stability constraints. That makes the newer family a better basis for contemporary internationalized domains, but it does not erase deployed IDNA 2003 data or interoperability requirements.

Applications sometimes need both:

• compatibility with stored or remote IDNA 2003 identifiers;
• IDNA 2008 or UTS #46 for new user input;
• an explicit boundary that says which version owns each value.

The version must travel with the data or be unambiguous from context. A bare string is not enough when two standards can interpret it differently.

A safer design for security-sensitive text

This incident suggests a practical engineering checklist.

1. Name the exact transformation

Avoid helpers called normalize_name() or clean_host(). Use names that expose the contract, such as to_idna2008_ascii() or nfkc_casefold_identifier_v17(). Precise names make accidental substitutions easier to spot in review.

2. Canonicalize once, then pass the canonical value

Do not let an allowlist, proxy, HTTP client, logger, and database each normalize the same raw input. Convert at one trusted boundary, validate the canonical result, and pass that exact result to the operation. This collapses the check/use split.

3. Record algorithm and data versions

If canonical forms are stored, include the transformation version in the schema or metadata. Treat upgrades as migrations. Recompute in a staging job, look for new collisions and changed keys, and decide how old records will be addressed.

4. Test non-ASCII edge cases across supported runtimes

ASCII-only fixtures will never expose Unicode drift. Build a corpus from specification tables, historical regressions, newly assigned code points, multi-code-point folds, normalization boundaries, and bidirectional cases. Run it against every supported runtime and architecture.

5. Compare components, not just functions

An implementation can pass its own unit tests while disagreeing with another service. Add end-to-end tests that send the canonical identifier from policy enforcement to the network or storage layer. Verify the exact bytes at both ends.

6. Prefer generated tables with provenance

Hand-maintained exception lists decay. Generate them from pinned upstream data, store checksums or version identifiers, and make diffs reviewable. A table may be large, but it is often safer than a clever fallback whose behavior is inherited from the host runtime.

7. Separate display text from identity keys

Users should see the text they entered, with appropriate spoofing warnings. Internal equality and routing should use a clearly defined canonical key. A confusable skeleton is a detection aid, not display text and not a general-purpose normalized identifier.

The larger lesson: standards code has a time dimension

Developers usually review a function by asking what inputs it accepts and what outputs it returns. Versioned text processing adds a third dimension: when the function’s data was defined.

str.lower() was not inherently unsafe. It was unsafe because its contract was “current Unicode lowercase,” while its caller’s contract was “the case mapping frozen into RFC 3454.” Both behaved as designed. The composition did not.

That pattern reaches far beyond domain names. Cryptographic suites, MIME registries, time-zone data, certificate roots, locale rules, parsers, and database collations all mix living software with versioned external knowledge. A convenient platform API can silently import today’s answer into a protocol that requires yesterday’s.

The safest question in standards code is therefore not only “Is this function correct?” It is:

Correct according to which data version, and will every component use the same one?

Once text influences identity, authorization, routing, or signatures, Unicode data is no longer a presentation detail. It is part of the security protocol.

Further reading

• When str.lower() is a security vulnerability in Python, Seth Larson’s account of the discovery and repair.
• RFC 3454: Preparation of Internationalized Strings, the Stringprep framework and its Unicode 3.2 tables.
• CPython pull request #155293, the generated compatibility data, regression tests, and security note.
• Python unicodedata documentation, including the legacy ucd_3_2_0 database.
• Unicode Technical Standard #39, identifier restrictions and confusable detection.
• RFC 5890: Internationalized Domain Names for Applications, the terminology and architecture of IDNA 2008.