7 ms·
RFC 9839 and Bad Unicode
- evrennetwork 1y ago[dead]
- ape4 1y agoSeems like libraries that serialize to JSON should have an option to filter out these bad characters.
- Manfred 1y agoMy experience writing Unicode related libraries is that people don't use features when you have to explain why and when to use them. I assume that's why Tim puts the emphasis on "working on something new".
- CharlesW 1y agoThis RFC and Go-language reference library is designed to be used by existing libraries that do serialization/sanitation/validation. This is hot off the press, so I'm sure Tim would appreciate it if you'd let your favorite library know it exists.
- nikolayasdf123 1y agointeresting. isn't in Go it is just `unicode.IsPrint(r rune)`? https://pkg.go.dev/unicode#IsPrint https://pkg.go.dev/unicode#IsPrint
- xdennis 1y agoHow is Unicode in any way related to JSON? JSON should just encode whatever dumb data someone wants to transport. Unicode validation/cleanup should be done separately because it's needed in multiple places, not just JSON.
- layer8 1y agoThe contents of JSON strings doesn’t admit random binary data. You need to use an encoding like Base64 for that purpose.
- recursive 1y agoJSON is text. If you're not going to use unicode in the representation of your text, you'll need some other way.
- dcrazy 1y agoThe current JSON spec mandates UTF-8, but practically speaking encoding is a higher-level concept. I suspect there are many server implementations that will respect the Content-Encoding header in a POST request containing JSON.
- ninkendo 1y agoSo? All the letters in this string are “just text”: "\u0000\u0089\uDEAD\uD9BF\uDFFF" JSON itself allows putting sequences of escape characters in the string that don’t unescape to valid Unicode. That’s fine, because the strings aren’t required to represent any particular encoding: it’s up to a layer higher than JSON to be opinionated about that. I wouldn’t want my shell’s pipeline buffers to reject data it doesn’t like, why should a JSON serializer?
- recursive 1y agoI actually agree, now that I understand what you're talking about.
- zzo38computer 1y agoJSON (unfortunately) requires strings to be Unicode. (JSON has other problems too, but Unicode is one of them.)
- layer8 1y agoNo. As the RFC notes: “Silently deleting an ill-formed part of a string is a known security risk. Responding to that risk, Section 3.2 of [UNICODE] recommends dealing with ill-formed byte sequences by signaling an error or replacing problematic code points, ideally with "�" (U+FFFD, REPLACEMENT CHARACTER).” I would almost always go for “signaling an error”.
- Waterluvian 1y agoI’m frustrated by things like Unicode where it’s “good” except… you need to know to exclude some of them. Unicode feels like a wild jungle of complexity. An understandable consequence of trying to formalize so many ways to write language. But it really sucks to have to reason about some characters being special compared to others. The only sanity I’ve found is to treat Unicode strings as if they’re some proprietary data unit format. You can accept them, store them, render them, and compare them with each other for (data, not semantic) equality. But you just don’t ever try to reason about their content. Heck I’m not even comfortable trying to concatenate them or anything like that.
- Etheryte 1y agoAs a simple example off the top of my head, if the first string ends in an orphaned emoji modifier and the second one starts with a modifiable emoji, you're already going to have trouble. It's only downhill from there with more exotic stuff.
- kps 1y agoUnicode combining/modifying/joining characters should have been prefix rather than suffix/infix, in blocks by arity.
- layer8 1y agoOne benefit of the suffix convention is that strings sort more usefully that way by default, without requiring special handling for those characters. Unicode 1.0 also explains: “The convention used by the Unicode standard is consistent with the logical order of other non-spacing marks in Semitic and Indic scripts, the great majority of which follow the base characters with respect to which they are positioned. To avoid the complication of defining and implementing non-spacing marks on both sides of base characters, the Unicode standard specifies that all non-spacing marks must follow their base characters. This convention conforms to the way modern font technology handles the rendering of non-spacing graphical forms, so that mapping from character store to font rendering is simplified.”
- o11c 1y agoI have had real-world programs broken by blind assumption of "does not deliberately contain controls" (form feed is particularly common for things intended to be paginated, escape is common for things designed for a terminal, etc.) and even "is fully UTF-8" (there are lots of old data files and logs that are never going away). If you aren't doing something useful with the text, you're best off passing a byte-sequence through unchanged. Unfortunately, Microsoft Windows exists, so sometimes you have to pass `char16_t` sequences through instead. The worst part about UTF-16 is that invalid UTF-16 is fundamentally different than invalid UTF-8. When translating between them (really: when transforming external data into an internal form for processing), the former can use WTF-8 whereas the latter can use Python-style surrogateescape, but you can't mix these.
- integralid 1y agoI'm not certain... On one hand I agree that some characters are problematic (or invalid) - like unpaired surrogates. But the worst case scenario is imo when people designing data structures and protocols start to feel the need to disallow arbitrary classes of characters, even properly escaped. In the example, username validation is a job of another layer. For example I want to make sure username is shorter than 60 characters, has no emojis or zalgo text, and yes, no null bytes, and return a proper error from the API. I don't want my JSON parsing to fail on completely different layer pre-validation. And for username some classes are obviously bad - like explained. But what if I send text files that actually use those weird tabs. I expect things that work in my language utf8 "string" type to be encodable. Even more importantly, I see plenty of use cases for null byte, and it is in fact often seen in JSON in the wild. On the other hand, if we have to use a restricted set of "normal" Unicode characters, having a standard feels useful - better than everyone creating their own mini standard. So I think I like the idea, just don't buy the argumentation or examples in the blog post.
- csande17 1y agoYeah, I feel like the only really defensible choices you can make for string representation in a low-level wire protocol in 2025 are: - "Unicode Scalars", aka "well-formed UTF-16", aka "the Python string type" - "Potentially ill-formed UTF-16", aka "WTF-8", aka "the JavaScript string type" - "Potentially ill-formed UTF-8", aka "an array of bytes", aka "the Go string type" - Any of the above, plus "no U+0000", if you have to interface with a language/library that was designed before people knew what buffer overflow exploits were
- stuartjohnson12 1y ago> "WTF-8", aka "the JavaScript string type" This sequence of characters is a work of art.
- wging 1y agoFor more details: https://simonsapin.github.io/wtf-8/ https://simonsapin.github.io/wtf-8/
- baybal2 1y ago[dead]
- JimDabell 1y ago> PRECISion · You may find yourself wondering why the IETF waited until 2025 to provide help with Bad Unicode. It didn’t; here’s RFC 8264: PRECIS Framework: Preparation, Enforcement, and Comparison of Internationalized Strings in Application Protocols; the first PRECIS predecessor was published in 2002. 8264 is 43 pages long, containing a very thorough discussion of many more potential Bad Unicode issues than 9839 does. I’d also suggest people check out the accompanying RFCs 8265 and 8266: PRECIS Framework: Preparation, Enforcement, and Comparison of Internationalized Strings in Application Protocols: — https://www.rfc-editor.org/rfc/rfc8264 https://www.rfc-editor.org/rfc/rfc8264 Preparation, Enforcement, and Comparison of Internationalized Strings: Representing Usernames and Passwords — https://www.rfc-editor.org/rfc/rfc8265 https://www.rfc-editor.org/rfc/rfc8265 Preparation, Enforcement, and Comparison of Internationalized Strings Representing Nicknames: — https://www.rfc-editor.org/rfc/rfc8266 https://www.rfc-editor.org/rfc/rfc8266 Generally speaking, you don’t want usernames being displayed that can change the text direction, or passwords that have different byte representations depending on the device that was used to type it in. These RFCs have specific profiles to avoid that. I think for these kinds of purposes, failing closed is more secure than failing open. I’d rather disallow whatever the latest emoji to hit the streets is from usernames than potentially allow it to screw up every page that displays usernames.
- singpolyma3 1y agoThe problem with failing closed is that you end up 20 years later still not supporting emoji from 20 years ago and users get annoyed...
- JimDabell 1y agoThat’s a problem that can happen if you don’t perform any maintenance for 20 years, and the outcome is mild annoyance from a small fraction of users. The problem with failing open can manifest tomorrow, and the outcome can cause your site to become unreadable.
- afiori 1y agoThat is a problem for future us, I sure don't envy them
- weinzierl 1y agoI think there should be a restriction in the standard on how many Unicode scalar values a graphical unit can have. Last time I checked (a couple of years ago admittedly) there was no such restriction in the standard. There was however a recommendation to restrict a graphical unit to 128 bytes for "streaming applications". Bringing this or at least a limit on the scalar units into the standard would make implementation and processing so much easier without restricting sensible applications.
- ks2048 1y agoIt's worth noting that Unicode already defines a "General Category" for all code points that categorizes some of these types of "weird" characters. https://en.wikipedia.org/wiki/Unicode_character_property#General_Category https://en.wikipedia.org/wiki/Unicode_character_property#Gen... e.g. in Python, import unicodedata print(unicodedata.category(chr(0))) print(unicodedata.category(chr(0xdead))) Shows "Cc" (control) and "Cs" (surrogate).
- develatio 1y agoI was not able to understand why these code points are bad. The post states that they are bad, but why? Any examples? Any actual situations and PoC that might help me understand how will that break "my code"?
- JimDabell 1y agoSuppose, when you were registering your username `develatio`, you decided to put U+202E RIGHT-TO-LEFT OVERRIDE in there as well. Now when somebody is reading this page and their browser gets to your username, it switches the text direction to render it right-to-left.
- develatio 1y agoand "that's it"? I mean, it does sound like it might introduce unexpected UI behaviour, but are there any other more serious / dangerous consequences?
- JimDabell 1y agoMaking any page that mentions you – including admin pages that might be used to disable your account – become unreadable is bad enough. Another comment linked to this: https://trojansource.codes https://trojansource.codes
- yencabulator 1y agoOne of my pet peeves is when UIs don't clearly constrain and delineate the extent of user-controlled text. Plenty of phishing attacks have relied on having attacker-controlled input seem authoritative, e.g. getting gmail to repeat back something to the victim.
- LikesPwsh 1y agoRTL lets you obfuscate file extensions. E.g. Annexe.txt (that you might assume would be safely opened by a text editor) could actually be Ann\u202Etxt.exe, a dangerous executable.
- 1y ago
- arp242 1y agoExcluding all of "legacy controls" not just as literals but also escaped strings (e.g. "\u0027") seems too much. C1 is essentially unused AFAIK and that's okay, but a number of C0 characters do see real-world use (escape, EOF, NUL). IMHO there are valid and reasonable use cases to use some of them.
- NelsonMinar 1y agoI've made good use of unusual C0 characters like U+001E (Record Separator). I think it makes sense to exclude them from documents but they can be useful in text data streams.
- senderista 1y agoAgreed, I would be very annoyed to see separator characters arbitrarily rejected by software I don't control. I think these characters are seriously underused, considering all the issues with in-band separators.
- msgodel 1y agoI've seen program source code with form feeds (U+000C) in it. Apparently Emacs has built in support for using them for navigation so adjacent things occasionally contain them.
- nikolayasdf123 1y agohow does this compare to Go `unicode.IsPrint(r rune)`? https://pkg.go.dev/unicode#IsPrint https://pkg.go.dev/unicode#IsPrint what does bad/dangerous this code catch that `unicode.IsPrint` is not catching? or other way, what good/useful does `unicode.IsPrint`removing, that this code keeps?
- mort96 1y agoI don't know all the details of `unicode.IsPrint` function, but one major issue is: it's Go-specific. If you're defining a protocol, you probably don't want the spec to include text such as, "the username field must only contain Unicode code points which are conidered printable by the Go programming language's 'unicode.IsPrint' function". You would rather want to write, "the username field must not contain Unicode code points which are considered problematic by RFC 9839".
- nikolayasdf123 1y agointeresting. so Go seem to be using unicode categories, which is part of unicode spec/standard. so it is fairly language agnostic. > IsPrint == .. categories L, M, N, P, S and the ASCII space character. how does that compare to this standard (RFC 9839)? (don't mind that this is Go. just consider same unicode categories).
- ninkendo 1y agoIt seems like most of these are handled by just rejecting invalid UTF-8 byte sequences (ideally, erroring out altogether) when interpreting a string as UTF-8. I mean, unpaired surrogates, or any surrogate for that matter, is already illegal as a UTF-8 byte sequence. Any competent language that uses UTF-8 for strings should already be returning errors when given such sequences. The list of code points which are problematic (non-printing, etc) are IMO much more useful and nontrivial. But it’d be useful to treat those as a separate concept from plain-old illegal UTF-8 byte sequences.
- doug_durham 1y agoThat seems reasonable. It should be up to the application implementer to make that choice and not a lower level more general purpose library. I haven't run into any JSON parsers for usernames only code.
- account42 1y ago> Any competent language that uses UTF-8 for strings should already be returning errors when given such sequences. No they shouldn't because that's how you get file managers that can't manage files.
- ninkendo 1y agoThe file manager wouldn’t use the “string” type to hold file names, if it’s written properly. Languages like Rust have things like OsString as separate from String for just this reason. If you have a type that says “my contents are valid UTF-8”, then you should reject invalid UTF-8 when populating it, obviously. Why would it work any other way? If you need a type that can hold arbitrary byte sequences, use a type that can hold arbitrary byte sequences.
- account42 1y agoThis is an unrealistic expectation. Local file names are just one example of many where you need to deal with UTF-8ish data that you should interpret as UTF-8 for display but pass along unmangled to other systems. Storing all that data twice and duplicating all relevant operations is both inefficient and will introduce more bugs as the two strings get out of sync. The gains from enforcing strict UTF-8 validation are minimal while the downsides are many - not the least of which is intentionally breaking forward compatibility with future Unicode versions that may extend what is valid. It's is also not what happens in practice. File managers that cannot rename or delete some files because they are unnecessarily "smart" about interpreting strings passed to them is very much how things have worked out in reality.
- yencabulator 1y agoI am torn on one decision: Whether to control inputs, or to wrap untrusted input in a datatype that displays it safely (web+log+debug).
- djoldman 1y agoI don't understand how this helps. Defining a subset of unicode to accept does not obviate the need to check that values conform to type definitions.
- skybrian 1y agoYes, that's true. It's for generic, low-level parsing code that doesn't know what a username is. There will also need to be field-specific validation.
- Grimeton 1y agoImagine the possibilities: PS C:\> mkdir x Directory: C:\ Mode LastWriteTime Length Name ---- ------------- ------ ---- d---- 23.08.2025 21:43 x PS C:\> cd x PS C:\x> new-item -ItemType File 'C꞉⧵Windows⧵System32⧵rundll32.exe' Directory: C:\x Mode LastWriteTime Length Name ---- ------------- ------ ---- -a--- 23.08.2025 21:43 0 C꞉⧵Windows⧵System32⧵rundll32.exe PS C:\x> $f=gi .\C꞉⧵Windows⧵System32⧵rundll32.exe PS C:\x> $f | fl -Property * -Force PSPath : Microsoft.PowerShell.Core\FileSystem::C:\x\C꞉⧵Windows⧵System32⧵rundll32.exe PSParentPath : Microsoft.PowerShell.Core\FileSystem::C:\x PSChildName : C꞉⧵Windows⧵System32⧵rundll32.exe PSDrive : C PSProvider : Microsoft.PowerShell.Core\FileSystem PSIsContainer : False Mode : -a--- ModeWithoutHardLink : -a--- VersionInfo : File: C:\x\C꞉⧵Windows⧵System32⧵rundll32.exe InternalName: OriginalFilename: FileVersion: FileDescription: Product: ProductVersion: Debug: False Patched: False PreRelease: False PrivateBuild: False SpecialBuild: False Language: BaseName : C꞉⧵Windows⧵System32⧵rundll32 ResolvedTarget : C:\x\C꞉⧵Windows⧵System32⧵rundll32.exe Target : LinkType : Name : C꞉⧵Windows⧵System32⧵rundll32.exe Length : 0 DirectoryName : C:\x Directory : C:\x IsReadOnly : False Exists : True FullName : C:\x\C꞉⧵Windows⧵System32⧵rundll32.exe Extension : .exe CreationTime : 23.08.2025 21:43:35 CreationTimeUtc : 23.08.2025 19:43:35 LastAccessTime : 23.08.2025 21:43:35 LastAccessTimeUtc : 23.08.2025 19:43:35 LastWriteTime : 23.08.2025 21:43:35 LastWriteTimeUtc : 23.08.2025 19:43:35 LinkTarget : UnixFileMode : -1 Attributes : Archive PS C:\x>
- 1oooqooq 1y agoit's almost like every language need another standards: safeunicode. which does away with control flow and specially positioning garbage. and doesn't consider as valid unknown (or missing) surrogates. why do i want very specialized text tabulation and positioning chars in my string? it's as if they tried to solve text encoding AND solve CSV and pagination and desktop publishing all in one go.
- Dwedit 1y agoBut then what would safe Unicode allow and reject? Carriage Returns? Horizontal Tabs? Form Feeds?
- unit149 1y ago[dead]
- RustyRussell 1y agoDid anyone else find the use if ABNF annoying? unicode-assignable = %x9 / %xA / %xD / ; useful controls %x20-7E / ; exclude C1 controls and DEL %xA0-D7FF / ; exclude surrogates %xE000-FDCF / ; exclude FDD0 nonchars %xFDF0-FFFD / ; exclude FFFE and FFFF nonchars %x10000-1FFFD / %x20000-2FFFD / ; (repeat per plane) %x30000-3FFFD / %x40000-4FFFD / %x50000-5FFFD / %x60000-6FFFD / %x70000-7FFFD / %x80000-8FFFD / %x90000-9FFFD / %xA0000-AFFFD / %xB0000-BFFFD / %xC0000-CFFFD / %xD0000-DFFFD / %xE0000-EFFFD / %xF0000-FFFFD / %x100000-10FFFD I mean, just define ranges. Also, where are the test vectors? Because when I implement this, that's the first thing I have to write, and you could save me a lot of work here. Bonus points if it's in JSON and UTF-8 already, though the invalid UTF-8 in an RFC might really gum things up: hex encode maybe?
- timbray 1y agoThe tests for the go code at https://github.com/timbray/RFC9839 https://github.com/timbray/RFC9839 are in effect test vectors.
- RustyRussell 1y agoI want to implement this. My code is in C. How does this help me check my implementation? I guess I could ask ChatGPT to convert your tests to my code, but that seems the long way around.
- djoldman 1y agohttps://github.com/timbray/RFC9839/blob/main/unichars.go https://github.com/timbray/RFC9839/blob/main/unichars.go I don't know rust at all but I can pretty quickly understand: var unicodeAssignables = []runePair{ {0x20, 0x7E}, // ASCII {0xA, 0xA}, // newline {0xA0, 0xD7FF}, // most of the BMP {0xE000, 0xFDCF}, // BMP after surrogates {0xFDF0, 0xFFFD}, // BMP after noncharacters block {0x9, 0x9}, // Tab {0xD, 0xD}, // CR {0x10000, 0x1FFFD}, // astral planes from here down {0x20000, 0x2FFFD}, {0x30000, 0x3FFFD}, {0x40000, 0x4FFFD}, {0x50000, 0x5FFFD}, {0x60000, 0x6FFFD}, {0x70000, 0x7FFFD}, {0x80000, 0x8FFFD}, {0x90000, 0x9FFFD}, {0xA0000, 0xAFFFD}, {0xB0000, 0xBFFFD}, {0xC0000, 0xCFFFD}, {0xD0000, 0xDFFFD}, {0xE0000, 0xEFFFD}, {0xF0000, 0xFFFFD}, {0x100000, 0x10FFFD}, }
- zzo38computer 1y agoI do not agree that Unicode is good. I think that Unicode is not good. I also think that, regardless of the character set, what to include (control characters, graphic characters, maximum length, etc) will have to depend on the specific application anyways, so trying to include/exclude in JSON doesn't work as well. Giving a name to a specific subset (or sometimes a superset, but usually subset) of Unicode (or any other character sets, such as ASCII or ISO 2022 or TRON code) can be useful, but don't assume it is good for everyone or even is good for most things, because it isn't. RFC 9839 does give names to some subsets of Unicode, which may sometimes be useful, but should not automatically assume that is right for what you will be making. My opinion is to consider to not use or require Unicode.
- msgodel 1y agoIt's really the combining characters that are the problem. That changed it from a character set to a DSL for describing characters.
- msgodel 1y agoThe form feed character (U+000C) should probably have been included in useful controls. That's used in a fair amount of program source code.
- OCTAGRAM 1y ago> The first code point is zero, in Unicode jargon U+0000. In human-readable text it has no meaning, but it will interfere with the operation of certain programming languages. This part encourages more active usage of U+0000, so that programmers of certain programming languages get a message that they are not welcome
- account42 1y agoNo, please do in fact allow unpaired surrogates and other weird characters so people can round trip "bad" data like file names through your protocol.