URL Slugs: Diacritics, Unicode, and What Gets Stripped

A slug is the readable, URL-safe identifier in a path: the my-first-post in /blog/my-first-post. It gets typed, copy-pasted, dropped into an href, and stored as a filename or database key, by browsers and servers that disagree on edge cases. The subset that works everywhere without special handling is lowercase ASCII letters, digits, and one separator, nearly always a hyphen. A slugifier's job is to squeeze arbitrary title text into that subset while losing as little meaning as possible.

Spaces and punctuation

A literal space is legal in some parts of a URL and not in others. Depending on where it sits and which library encodes it, it becomes %20 or a +. It also breaks anything that splits on whitespace: shell commands, some older link parsers, chat clients that auto-link pasted text.

Punctuation has the same problem to a lesser degree. Some characters are part of URL syntax (?, #, &, /). The rest add visual noise and encode inconsistently. Turning every non-alphanumeric run into one predictable separator avoids all of it. A title like What's new in 2.0? comes out as what-s-new-in-2-0: the apostrophe, the space, the dot, and the question mark all become separators, and the trailing one is trimmed.

How diacritic stripping works

é looks like one character, but Unicode can store it two ways. It can be a single precomposed codepoint, U+00E9 ("LATIN SMALL LETTER E WITH ACUTE"). Or it can be two codepoints: a plain e (U+0065) followed by COMBINING ACUTE ACCENT (U+0301), a mark from Unicode's combining diacritical marks block that attaches to the character before it.

NFD normalization (Normalization Form Decomposed) converts every precomposed character in a string into that base-plus-mark form. After that, stripping diacritics means deleting every codepoint in the combining-marks range. The plain base letters are what's left.

"café résumé".normalize("NFD")
// "café résumé"  — the é has become e + U+0301

"cafe\u0301 re\u0301sume\u0301".replace(/[\u0300-\u036f]/g, "")
// "cafe resume" — combining marks gone, plain ASCII left

The slug generator on this site works this way. It normalizes to NFD, strips the combining-marks range, lowercases, collapses everything non-alphanumeric into one separator, and trims the ends. é → e, ñ → n, and ü → u all come out of the same regex once the string is decomposed. You don't need a per-letter mapping table.

Examples

The third row may surprise you. German ß ("eszett") has no diacritic to strip. It's a letter of its own, with no NFD decomposition, so the normalize step leaves it alone. The next step replaces every remaining non-a-zA-Z0-9 character with a separator, and it treats ß like a space or a comma: it becomes a hyphen.

So Straße turns into Stra, a separator, and e. The word is split in two instead of becoming strasse. Handling it properly needs an explicit ß → ss substitution before the generic strip-and-collapse pass. Without it, the generic approach breaks the word apart as well as losing information.

Where this stops working

NFD only helps when the accented character has a plain Latin letter underneath with a mark on top. French, Spanish, German, Portuguese, and similar Latin-script languages fit that. Scripts that aren't built on Latin letters don't. Cyrillic Москва, Greek Αθήνα, and CJK text such as 東京 don't decompose into a Latin letter plus a mark, because there's no Latin letter in them.

Running NFD and mark-stripping over that text accomplishes nothing. What you get depends on the filter that keeps alphanumeric characters afterward. A Unicode-aware filter leaves the original characters as they were. A filter that assumes only ASCII survives leaves you with an empty or garbled string, and some slugifiers ship with that bug. Diacritic stripping handles Latin-alphabet accents, a common case. It isn't transliteration, and it doesn't handle Unicode in general.

For non-Latin titles you have two options. You can transliterate first: romanize Cyrillic or Greek phonetically, or map CJK to pinyin or romaji. That's a separate, language-specific process, not a normalization trick. Or you can keep a non-ASCII slug and percent-encode it in the URL, which modern browsers and most frameworks handle correctly. Guard against the empty slug in particular. Falling back to something (a transliteration, a hash, or the original text) beats a blank path segment.

Case and separators

URLs are case-sensitive, so lowercasing is a convention and not a requirement. Mixed-case slugs invite the same content at two paths and look out of place next to every other lowercase URL on the site, so nearly every slugifier lowercases by default. My Post and MY POST should both land on my-post, not on two different URLs.

Separator runs matter too. Replacing each disallowed character with a hyphen, one for one, turns "foo!! -- bar" into foo----bar instead of foo-bar. The implementation has to collapse consecutive separators into one, and then trim any left at the start or end, where they only produce an ugly leading or trailing hyphen.

Two titles, one slug

Slugifying is many-to-one. "Café Notes", "Cafe Notes", and "CAFE, Notes!" all produce cafe-notes. That follows from throwing away case, punctuation, and diacritics on purpose, so it isn't a bug in the slugifier. It does mean the system that stores slugs needs a rule for when a new item's slug already exists. The usual ones, from least to most change to the URL scheme:

• Append a counter on collision: cafe-notes, then cafe-notes-2, cafe-notes-3. These read well, but you have to check existing slugs before assigning one, and the numbers mean nothing in particular. They can look odd after items are deleted or reordered.
• Append a short opaque suffix, often taken from the item's database ID or a few characters of a random or hash-based ID (cafe-notes-4f2a). The suffix is unique by construction, so you skip the existence check. The URL looks a little less clean.
• Don't keep slugs globally unique. Scope them to another key already in the URL. A blog with per-author paths like /blog/alice/cafe-notes and /blog/bob/cafe-notes never has a collision, because the two slugs were never after the same URL.