12 ms·

I love Unicode, but I'm more and more coming to the conclusion that strings are evil and should be treated as opaque byte arrays, whose only available operation

I love Unicode, but I'm more and more coming to the conclusion that strings are evil and should be treated as opaque byte arrays, whose only available operation is rendering into a bounded area. I now see any other string operation as code smell. It's scary how much of our infrastructure relies on strings, given how few guarantees string operations actually give. Take files names, for example. Two visually identical file names may map to different files (because confusables[1]), or two different names map to the same file (because normalization[2]), or the ".jpg" at the end may not actually be the extension (because right-to-left override[3]), not to mention names with newlines or backspaces in them, and inconsistencies between operating systems. I would go as far as blaming our overreliance on strings for all the injection attacks we see (XSS, SQL, command, etc). [1] https://unicode.org/cldr/utility/confusables.jsp https://unicode.org/cldr/utility/confusables.jsp [2] https://developer.apple.com/library/archive/qa/qa1173/_index.html https://developer.apple.com/library/archive/qa/qa1173/_index... [3] https://krebsonsecurity.com/2011/09/right-to-left-override-aids-email-attacks/ https://krebsonsecurity.com/2011/09/right-to-left-override-a...

kortex 7y ago
How do you do the rendering? Who (process-wise) is responsible for converting bytes to pixels? How do users on social media put in their name? How is it stored? How do users get urls with specific usernames? Now take all that and multiply by the complexity of world languages, many which don't even map to one glyph == one morpheme. The ol' apple message crash bug was due to the property of some Arabic not being monotonic in rendering space vs string length. I think we could have skipped utf8 and just gone to 4byte runes. But even then, that would not have avoided the above bug. Utf16 is a hot mess though, worst of all worlds.
- BoppreH 7y ago
  The problem is now deeply entrenched, so I don't have perfect answers. But here are my guesses: > Who (process-wise) is responsible for converting bytes to pixels? The operating system, with minor exceptions (word processors, for example). Rendering logic is too complicated to be embedded into every application. And you get better accessibility and consistency. > How do users on social media put in their name? How is it stored? Keyboards, or their preferred input methods, and stored in UTF-8. I'm not saying to get rid of all strings, just don't use it for infrastructure. > How do users get urls with specific usernames? You don't, because that's how you get little Bobby FRACTION-SLASH. Also, if you have a valuable namespace like URLs, people will hack each other to get valuable names, but that's only tangentially related. > Now take all that and multiply by the complexity of world languages, many which don't even map to one glyph == one morpheme. The ol' apple message crash bug was due to the property of some Arabic not being monotonic in rendering space vs string length. That's exactly my point! You get this multiplied complexity when people try to peek into string contents instead of treating them like black boxes. Stop with the dangerous string operations and you now support usernames with zalgo-ed hieroglyphs if that's what users want. > I think we could have skipped utf8 and just gone to 4byte runes. But even then, that would not have avoided the above bug. > Utf16 is a hot mess though, worst of all worlds. Agreed with UTF-16. I like UTF-8, and I honestly think it solved our encoding problems for non-legacy applications. Everyone should be using it, as long as the contents are for human consumption only.
  - camgunz 7y ago
    UTF-16 is way better for many Asian languages. You know, languages billions of people use.
    - BoppreH 7y ago
      Space-wise, yes, but the size of our user-entered strings is rarely a concern. On the other hand, UTF-16 is often mixed with UCS-2, which is not really Unicode, and lulls developers into a false sense of security by almost never needing two code units for a given code point. It's a partial fix that makes bugs harder to catch. There's also the BOM issue, and not being ASCII compatible...
      - camgunz 7y ago
        Space is always a concern, and encodings should largely be transparent to your application unless you're writing something like a rendering engine. Consider things like databases (relational/document/key-value/etc.), JSON payloads, compression algorithms, battery life, etc. etc. There is a problem with lots of developers assuming UTF-16 is fixed-width (just look at this thread), but that's a mistake for any UTF encoding, and it's almost always a mistake in any encoding--you shouldn't be indexing into strings. People say, "just use UTF-8, you never have to worry about it and it's ASCII compatible", but you shouldn't be using the ASCII compatibility (arguably this is an anti-feature) and you shouldn't have to worry about the details like BOM because you're using your language's built-in string support or a library right?
    - jhanschoo 7y ago
      What's your reasoning? Where there is significant markup, UTF-16 encoding is rarely more space-efficient than UTF-8. When compression is involved, there is rarely a difference.
      - thaumasiotes 7y ago
        UTF-16 may not be great, but UTF-8 is an obviously bad choice for Asian languages. UTF-8 is a variable-width encoding on the following system: ASCII: 1 byte European characters: 2 bytes Asian characters: 3 bytes There is no reason you'd want this in Asia. China uses GB2312, which flips it around: ASCII: 1 byte Asian characters: 2 bytes European characters: 3 bytes
- swiley 7y ago
  That harfbuzz guy, he’s the only one allowed to do it.
vsareto 7y ago
>I now see any other string operation as code smell. Make string ops compile to byte array ops?
mc3 7y ago
> whose only available operation is rendering into a bounded area how would Google work? how would a web browser work, e.g. if you type in a URL?
- BoppreH 7y ago
  There are exceptions, of course. Fuzzy search is one of them. Image editing software and programming language parsers are another two. And what's the problem of letting the user type a URL? You take the resulting string (given to you by the OS subsystem responsible for keyboard input), and stuff that into your favorite HTTP library. Don't get me wrong, strings are absolutely necessary. I'm not suggesting we switch to pictograms. But string processing should be treated like the dangerous operation it is, like pointer math and cryptography.
  - mc3 7y ago
    I just write a lot of code that looks inside strings, in simple 'crud' apps, so I can't imagine avoiding these kinds of operations. Of course if you can use highly trusted libraries and just pass 'em around it's OK. But the Github issue was sort of subtle. In hindsight it seems like an obvious one, but it's a mistake I typically see in a lot of code reviews I've done where people just reach for the convenient variable without thinking behind the intent and 'meaning' behind it. The problem is you would probably have code like this: sendChangePassword(string email) { userDetails = getUserDetails(email); if (userDetails !== null) MailerHelper.sendChangePasswordEmail(email, userDetails.name); } How would the don't look at strings thing apply? Well you could not use .toLowerCase inside the getUserDetails, then you are treating the email as a stream of bytes. But it would allow someone to sign up multiple times as fred@gmail.com, FRED@gmail.com. This might be bad for you (spammers) and bad for the user (they don't realise they actually have 2 accounts). The alternative could be 'well use a library written by experts for dealing with email duplication issues', i.e. detecting that string A and string B map to the same email or not. But we criticize ourselves for the NPM dependency madness, right? So some balance is needed. It is tricky to not deal with strings in some way! One pattern I am keen on, and will probably use in a refactor I am doing is using types to add meaning to strings. So in Typescript for example, don't pass a string, but create a wrapper class called, say EmailString, than on construction does some validation (or maybe none), but as a minimum you have documentation of what the string means. You can shift-F12 the constructor to check the hopefully few places it is constructed to make sure they do the write thing, then in the 100's of consuming places you know you are dealing with an email.
Iv 7y ago
Disclaimer: as a heavy user of unicode, using it for both French and Japanese, I love it and see how important it is in the world. I just ranted about why ASCII is important to programmers: https://news.ycombinator.com/item?id=21760540 https://news.ycombinator.com/item?id=21760540 and this is a perfect example. ASCII has almost a 1-to-1 mapping between screen representation and byte representation. Once you know your font will differentiate between 1 i L | l and 0 O you still need to know a bit about control characters and you are good to go. Unicode has tons of pages, control characters, diacritics and rendering oddities that make it hard to use like a tool. I think your approach of considering like bytes to render is spot on. I consider these strings as something like SVG fragments: you can do text operations on them but you have to be confident that you know what you are doing.
- Avamander 7y ago
  I'm personally incredibly annoyed by just the idea of "Unicode is hard, let's do ASCII", most of the world is non-ASCII, it's just annoying and sad to still see systems that fail when people try to use their native languages. UTF-8 should be the default pretty much everywhere, there are quite easy ways to avoid homograph attacks, those attacks are a poor excuse to discriminate against non-anglosphere.
  - antris 7y ago
    Yes, all user-facing text should be Unicode, always. But ASCII has its use cases as well, as the parent comment mentioned in programming. I am natively non-ASCII compatible, but I am glad to program in ASCII and English. The only things that should not be English in written code are domain-specific terms that do not have an official unambiguous English translation. That's the only case to be made for non-ASCII characters in programming that I can think of, but I think romanization can take care of it with most languages.
    - ksec 7y ago
      This, I think people are arguing for ASCII in programming, not going back to ASCII in general.
amenghra 7y ago
Mr. dot is evil is the first thing we taught new engineers at Facebook during the security engineering on boarding. There’s a whole slide deck (filled with real code snippets) with examples where a string was used instead of a better suited object representation and lead to a security flaw. We eventually build xhp/jsx to get rid of strings-holding-html data, but that was just scratching the surface of bugs caused by user-supplied strings.
- lookdangerous 7y ago
  Mr. Dot as in the php stir concatenation operator?
  - paulddraper 7y ago
    That's my guess as well. Speaking of "evil" broken abstractions...
scythe 7y ago
So here's the thing. When you display a word written entirely in Cyrillic characters, it would be wrong for it not to display as normal text. But when you display a word that contains eight Latin characters and one Cyrillic character, couldn't the display create some sort of warning? Highlight the section-mismatched character with a box? It's not normal in any language to mix alphabets.
- BoppreH 7y ago
  That's a good heuristic, but you are still left with spoofing using 100% Cyrillic, and even old school approaches like "app1e.com" (1 instead of l). When you consider distracted users, or people with bad eyesight, you are back to square l.
  - silon42 7y ago
    We (I) really would like system where I would enable the languages/characters that I support/recognise and the rest would be displayed as a number in a box.
- ComputerGuru 7y ago
  > it’s not normal in any language to mix alphabets Decidedly not true. Languages like French and Turkish and German use their own alphabet but they look like the typical “American” Latin alphabet but aren’t. How do you decide if it’s English with non-American letters thrown in or not? Languages like Arabic are transliterated with English letters and Arabic numerals. Other Transliteration dialects use random Arabic characters with Latin others.
  - pwdisswordfish2 7y ago
    By the ‘script’ property, which every Unicode code point has? It's hardly bulletproof, but it can certainly stop some of the more obvious shenanigans.
- Liquid_Fire 7y ago
  > It's not normal in any language to mix alphabets. This is not true. For example, my native language has suffix-based articles (there is no separate word "the"; instead there is a suffix attached to the noun). When including a foreign word in the middle of a sentence, some people will attach the article suffix directly to the foreign word, or at best with a hyphen (this is likely not correct "by the book", but some people write like this anyway). With your suggestion, this will show a warning when it is perfectly normal usage. Furthermore, some Asian languages like Japanese do not use spaces, so when they mix in foreign words or names written in their original script how do you tell that it's intended to be a separate word?
jkaptur 7y ago
You’ve read my mind! I’ve been meaning to give a talk where I just go down the list of string methods and point out how each one is a hideous source of bugs.
- BoppreH 7y ago
  Please, go forth and spread the word. Drop me an email (on my profile) if you want ideas. I don't have a blog yet, but I already have a draft for a post titled "Strings are evil". There's a lot wrong with the current way we develop software, but I'm convinced this is a big one.
rgj 7y ago
Strings are not evil - user supplied input should be treated as evil. The fact that most user input is supplied as a string is a mere coincidence.
- koboll 7y ago
  Yep. Github's mistake in the parent article wasn't overlooking a character casing collision -- it was sending password reset emails to the email provided by the resetter rather than the saved email for that user.
simias 7y ago
The problem is that your method works well if the string is either: - Only meant to be used in a machine-to-machine interface (like a JSON key for instance). - Only meant to be used in a machine-to-human interface (like the text letting you know that you used a wrong password). For strings that have a special significance to both humans and machines you can still run into problems, mainly because of the many unicode strings that look similar (or sometimes even identical) visually but are encoded with a different byte sequence. Take usernames or URLs for instance. But in the end Unicode is messy because human languages are messy. As programmers we like well-bounded problems with elegant generic solutions. This clearly isn't possible here and we have to deal with it.
founderling 7y ago
opaque byte arrays, only available operation is rendering into a bounded area Goodbye web then. Because the whole request you get from a client is nothing but strings. So when you run an onlineshop and a customer orders "7" screwdrivers - then you are screwed. Because what does an "opaque byte array" of "opaque byte arrays" cost? How much shipping will that be? But at least you have a brand new customer: Henry@gmail.com. Since you do not lowercase the email and do not recognize him as henry@gmail.com which he used in his last order.
- mnw21cam 7y ago
  > But at least you have a brand new customer: Henry@gmail.com. Since you do not lowercase the email and do not recognize him as henry@gmail.com which he used in his last order. Which would be correct behaviour. From RFC 5321, part 2.4, "The local-part of a mailbox MUST BE treated as case sensitive."
- 2038AD 7y ago
  >But at least you have a brand new customer: Henry@gmail.com. Since you do not lowercase the email and do not recognise him as henry@gmail.com which he used in his last order You may not like it but this is fully standards-compliant behaviour. RFC 5321 states: >The local-part of a mailbox MUST BE treated as case sensitive I'm not saying this is a good behaviour and the RFC also discourages exploiting it further. There are more ways an email address can be equivalent though :-). Since we know it's gmail in this case we know the email address is both case insensitive and dots don't matter. Comments are also allowed in email addresses (in both the local part and the domain part). Here's a couple examples from RFC 2822: pete(his account)@silly.test(his host) c@(Chris's host.)public.example [0] https://tools.ietf.org/html/rfc5321 https://tools.ietf.org/html/rfc5321 [1] https://support.google.com/mail/answer/7436150 https://support.google.com/mail/answer/7436150 [2] https://www.ietf.org/rfc/rfc2822.html#appendix-A.5 https://www.ietf.org/rfc/rfc2822.html#appendix-A.5
- BoppreH 7y ago
  That's a good point, these problems are not easy to solve. But they should be solved. For example, the user types "7" into the UI control. The UI control knows it's supposed to hold a number, so it has methods that return the parsed number (done by the OS, very carefully), or an error to the user. Similarly, you don't send the product name, but the ID of the product the user selected from their fuzzy search. I would go as far as using this technique for the email too.
thaumasiotes 7y ago
I want to note a separate issue of defensive coding that comes up in the writeup: > GitHub's forgot password feature could be compromised because the system lowercased the provided email address and compared it to the email address stored in the user database. If there was a match, GitHub would send the reset password link to the email address provided by the attacker The logical flow is: 1. Get the email address from the forgot-password request. 2. Get the email address from the database for the same account. 3. Check whether they match. 4a. If not, we're under attack -- refuse the request. 4b. If so, all is well -- send a password reset to the email address. Of course, we know the email address twice -- we asked the user for it during the password reset process (step 1), but we never needed to do that because we already had an email address on file for the account. We retrieved that email address in step 2. We know that the two addresses are the same, but, if you look at the semantics behind the variables, in step 4b we're choosing one of these two "equivalent" options, depending on which variable we use for the email address: 1. Send the account password to the account owner. 2. Send the account password to a guy who doesn't know what the password is. And these have very different risk profiles. Choosing the first option instead of the second would have prevented this attack without needing to worry about unicode case-translation issues. You never want to trust information you just received from an unknown user when you already have the same information from a more authoritative source.
- gingerlime 7y ago
  Yes, you're absolutely right. The real bug was sending to the "wrong" matching email. But this is what makes this bug so hard to find. You're looking at "equal" strings, so why should it make a difference if you pick A or A if A === A ? :)
  - thaumasiotes 7y ago
    Hard to find, but it's the kind of thing that will hopefully come up in code review. Consider this pseudocode with helpful pseudo-hungarian notation: username = request.post_params('username') evil_email = request.post_params('email_address') user = get_user_by_name(username) good_email = user.email_address if good_email != evil_email: # Hackers! else: reset_password(user.id, evil_email) # it's fine; it's the same as user.email_address You can see the problem in the call to reset_password. Sure, it's the same as user.email_address, but if you just used user.email_address you'd be doing the right thing and you wouldn't need the comment about "don't worry; it's fine". Then you start to wonder if that line shouldn't look more like reset_password(user) and from a security perspective, it should. (I assume the reason Github allows the code to specify the email address in the first place is that an account may be associated with multiple addresses.) If it makes no difference which email address you pick, why not pick the one that represents doing something safe instead of the one that represents doing something dangerous?
giancarlostoro 7y ago
I usually store files inside UUID sub directories with UUID names. Then keep the original name saved elsewhere in a db to avoid filename collision.
simion314 7y ago
But why non english users should be forced to name their files with english characters, libraries and code that work with files should always use tests with unicode in file names.
merb 7y ago
> but I'm more and more coming to the conclusion that strings are evil and should be treated as opaque byte arrays, whose only available operation is rendering into a bounded area nope. they are not byte arrays. if treating them as such, you run into many nasty bugs. today I encountered one (C#): if (string.Length > 60) string.Remove(60) this has a fucking serious bug, especially if you try to show the string afterwards or worse save it inside a database. strings are arrays of codepoints. NEVER TREAT THEM AS BYTE/CHAR ARRAYS.
- BoppreH 7y ago
  I meant "byte array" as in "this is a bunch of binary data you should not look into". Like people already do with images. You wouldn't remove a byte from the middle of a JPEG file, and you shouldn't do it with strings either.

kortex 7y ago

How do you do the rendering? Who (process-wise) is responsible for converting bytes to pixels? How do users on social media put in their name? How is it stored? How do users get urls with specific usernames? Now take all that and multiply by the complexity of world languages, many which don't even map to one glyph == one morpheme. The ol' apple message crash bug was due to the property of some Arabic not being monotonic in rendering space vs string length. I think we could have skipped utf8 and just gone to 4byte runes. But even then, that would not have avoided the above bug. Utf16 is a hot mess though, worst of all worlds.

BoppreH 7y ago
The problem is now deeply entrenched, so I don't have perfect answers. But here are my guesses: > Who (process-wise) is responsible for converting bytes to pixels? The operating system, with minor exceptions (word processors, for example). Rendering logic is too complicated to be embedded into every application. And you get better accessibility and consistency. > How do users on social media put in their name? How is it stored? Keyboards, or their preferred input methods, and stored in UTF-8. I'm not saying to get rid of all strings, just don't use it for infrastructure. > How do users get urls with specific usernames? You don't, because that's how you get little Bobby FRACTION-SLASH. Also, if you have a valuable namespace like URLs, people will hack each other to get valuable names, but that's only tangentially related. > Now take all that and multiply by the complexity of world languages, many which don't even map to one glyph == one morpheme. The ol' apple message crash bug was due to the property of some Arabic not being monotonic in rendering space vs string length. That's exactly my point! You get this multiplied complexity when people try to peek into string contents instead of treating them like black boxes. Stop with the dangerous string operations and you now support usernames with zalgo-ed hieroglyphs if that's what users want. > I think we could have skipped utf8 and just gone to 4byte runes. But even then, that would not have avoided the above bug. > Utf16 is a hot mess though, worst of all worlds. Agreed with UTF-16. I like UTF-8, and I honestly think it solved our encoding problems for non-legacy applications. Everyone should be using it, as long as the contents are for human consumption only.
- camgunz 7y ago
  UTF-16 is way better for many Asian languages. You know, languages billions of people use.
  - BoppreH 7y ago
    Space-wise, yes, but the size of our user-entered strings is rarely a concern. On the other hand, UTF-16 is often mixed with UCS-2, which is not really Unicode, and lulls developers into a false sense of security by almost never needing two code units for a given code point. It's a partial fix that makes bugs harder to catch. There's also the BOM issue, and not being ASCII compatible...
    - camgunz 7y ago
      Space is always a concern, and encodings should largely be transparent to your application unless you're writing something like a rendering engine. Consider things like databases (relational/document/key-value/etc.), JSON payloads, compression algorithms, battery life, etc. etc. There is a problem with lots of developers assuming UTF-16 is fixed-width (just look at this thread), but that's a mistake for any UTF encoding, and it's almost always a mistake in any encoding--you shouldn't be indexing into strings. People say, "just use UTF-8, you never have to worry about it and it's ASCII compatible", but you shouldn't be using the ASCII compatibility (arguably this is an anti-feature) and you shouldn't have to worry about the details like BOM because you're using your language's built-in string support or a library right?
  - jhanschoo 7y ago
    What's your reasoning? Where there is significant markup, UTF-16 encoding is rarely more space-efficient than UTF-8. When compression is involved, there is rarely a difference.
    - thaumasiotes 7y ago
      UTF-16 may not be great, but UTF-8 is an obviously bad choice for Asian languages. UTF-8 is a variable-width encoding on the following system: ASCII: 1 byte European characters: 2 bytes Asian characters: 3 bytes There is no reason you'd want this in Asia. China uses GB2312, which flips it around: ASCII: 1 byte Asian characters: 2 bytes European characters: 3 bytes
swiley 7y ago
That harfbuzz guy, he’s the only one allowed to do it.

vsareto 7y ago

>I now see any other string operation as code smell. Make string ops compile to byte array ops?

mc3 7y ago

> whose only available operation is rendering into a bounded area how would Google work? how would a web browser work, e.g. if you type in a URL?

BoppreH 7y ago
There are exceptions, of course. Fuzzy search is one of them. Image editing software and programming language parsers are another two. And what's the problem of letting the user type a URL? You take the resulting string (given to you by the OS subsystem responsible for keyboard input), and stuff that into your favorite HTTP library. Don't get me wrong, strings are absolutely necessary. I'm not suggesting we switch to pictograms. But string processing should be treated like the dangerous operation it is, like pointer math and cryptography.
- mc3 7y ago
  I just write a lot of code that looks inside strings, in simple 'crud' apps, so I can't imagine avoiding these kinds of operations. Of course if you can use highly trusted libraries and just pass 'em around it's OK. But the Github issue was sort of subtle. In hindsight it seems like an obvious one, but it's a mistake I typically see in a lot of code reviews I've done where people just reach for the convenient variable without thinking behind the intent and 'meaning' behind it. The problem is you would probably have code like this: sendChangePassword(string email) { userDetails = getUserDetails(email); if (userDetails !== null) MailerHelper.sendChangePasswordEmail(email, userDetails.name); } How would the don't look at strings thing apply? Well you could not use .toLowerCase inside the getUserDetails, then you are treating the email as a stream of bytes. But it would allow someone to sign up multiple times as fred@gmail.com, FRED@gmail.com. This might be bad for you (spammers) and bad for the user (they don't realise they actually have 2 accounts). The alternative could be 'well use a library written by experts for dealing with email duplication issues', i.e. detecting that string A and string B map to the same email or not. But we criticize ourselves for the NPM dependency madness, right? So some balance is needed. It is tricky to not deal with strings in some way! One pattern I am keen on, and will probably use in a refactor I am doing is using types to add meaning to strings. So in Typescript for example, don't pass a string, but create a wrapper class called, say EmailString, than on construction does some validation (or maybe none), but as a minimum you have documentation of what the string means. You can shift-F12 the constructor to check the hopefully few places it is constructed to make sure they do the write thing, then in the 100's of consuming places you know you are dealing with an email.

Iv 7y ago

Disclaimer: as a heavy user of unicode, using it for both French and Japanese, I love it and see how important it is in the world. I just ranted about why ASCII is important to programmers: https://news.ycombinator.com/item?id=21760540 https://news.ycombinator.com/item?id=21760540 and this is a perfect example. ASCII has almost a 1-to-1 mapping between screen representation and byte representation. Once you know your font will differentiate between 1 i L | l and 0 O you still need to know a bit about control characters and you are good to go. Unicode has tons of pages, control characters, diacritics and rendering oddities that make it hard to use like a tool. I think your approach of considering like bytes to render is spot on. I consider these strings as something like SVG fragments: you can do text operations on them but you have to be confident that you know what you are doing.

Avamander 7y ago
I'm personally incredibly annoyed by just the idea of "Unicode is hard, let's do ASCII", most of the world is non-ASCII, it's just annoying and sad to still see systems that fail when people try to use their native languages. UTF-8 should be the default pretty much everywhere, there are quite easy ways to avoid homograph attacks, those attacks are a poor excuse to discriminate against non-anglosphere.
- antris 7y ago
  Yes, all user-facing text should be Unicode, always. But ASCII has its use cases as well, as the parent comment mentioned in programming. I am natively non-ASCII compatible, but I am glad to program in ASCII and English. The only things that should not be English in written code are domain-specific terms that do not have an official unambiguous English translation. That's the only case to be made for non-ASCII characters in programming that I can think of, but I think romanization can take care of it with most languages.
  - ksec 7y ago
    This, I think people are arguing for ASCII in programming, not going back to ASCII in general.

amenghra 7y ago

Mr. dot is evil is the first thing we taught new engineers at Facebook during the security engineering on boarding. There’s a whole slide deck (filled with real code snippets) with examples where a string was used instead of a better suited object representation and lead to a security flaw. We eventually build xhp/jsx to get rid of strings-holding-html data, but that was just scratching the surface of bugs caused by user-supplied strings.

lookdangerous 7y ago
Mr. Dot as in the php stir concatenation operator?
- paulddraper 7y ago
  That's my guess as well. Speaking of "evil" broken abstractions...

scythe 7y ago

So here's the thing. When you display a word written entirely in Cyrillic characters, it would be wrong for it not to display as normal text. But when you display a word that contains eight Latin characters and one Cyrillic character, couldn't the display create some sort of warning? Highlight the section-mismatched character with a box? It's not normal in any language to mix alphabets.

BoppreH 7y ago
That's a good heuristic, but you are still left with spoofing using 100% Cyrillic, and even old school approaches like "app1e.com" (1 instead of l). When you consider distracted users, or people with bad eyesight, you are back to square l.
- silon42 7y ago
  We (I) really would like system where I would enable the languages/characters that I support/recognise and the rest would be displayed as a number in a box.
ComputerGuru 7y ago
> it’s not normal in any language to mix alphabets Decidedly not true. Languages like French and Turkish and German use their own alphabet but they look like the typical “American” Latin alphabet but aren’t. How do you decide if it’s English with non-American letters thrown in or not? Languages like Arabic are transliterated with English letters and Arabic numerals. Other Transliteration dialects use random Arabic characters with Latin others.
- pwdisswordfish2 7y ago
  By the ‘script’ property, which every Unicode code point has? It's hardly bulletproof, but it can certainly stop some of the more obvious shenanigans.
Liquid_Fire 7y ago
> It's not normal in any language to mix alphabets. This is not true. For example, my native language has suffix-based articles (there is no separate word "the"; instead there is a suffix attached to the noun). When including a foreign word in the middle of a sentence, some people will attach the article suffix directly to the foreign word, or at best with a hyphen (this is likely not correct "by the book", but some people write like this anyway). With your suggestion, this will show a warning when it is perfectly normal usage. Furthermore, some Asian languages like Japanese do not use spaces, so when they mix in foreign words or names written in their original script how do you tell that it's intended to be a separate word?

jkaptur 7y ago

You’ve read my mind! I’ve been meaning to give a talk where I just go down the list of string methods and point out how each one is a hideous source of bugs.

BoppreH 7y ago
Please, go forth and spread the word. Drop me an email (on my profile) if you want ideas. I don't have a blog yet, but I already have a draft for a post titled "Strings are evil". There's a lot wrong with the current way we develop software, but I'm convinced this is a big one.

rgj 7y ago

Strings are not evil - user supplied input should be treated as evil. The fact that most user input is supplied as a string is a mere coincidence.

koboll 7y ago
Yep. Github's mistake in the parent article wasn't overlooking a character casing collision -- it was sending password reset emails to the email provided by the resetter rather than the saved email for that user.

simias 7y ago

The problem is that your method works well if the string is either: - Only meant to be used in a machine-to-machine interface (like a JSON key for instance). - Only meant to be used in a machine-to-human interface (like the text letting you know that you used a wrong password). For strings that have a special significance to both humans and machines you can still run into problems, mainly because of the many unicode strings that look similar (or sometimes even identical) visually but are encoded with a different byte sequence. Take usernames or URLs for instance. But in the end Unicode is messy because human languages are messy. As programmers we like well-bounded problems with elegant generic solutions. This clearly isn't possible here and we have to deal with it.

founderling 7y ago

opaque byte arrays, only available operation is rendering into a bounded area Goodbye web then. Because the whole request you get from a client is nothing but strings. So when you run an onlineshop and a customer orders "7" screwdrivers - then you are screwed. Because what does an "opaque byte array" of "opaque byte arrays" cost? How much shipping will that be? But at least you have a brand new customer: Henry@gmail.com. Since you do not lowercase the email and do not recognize him as henry@gmail.com which he used in his last order.

mnw21cam 7y ago
> But at least you have a brand new customer: Henry@gmail.com. Since you do not lowercase the email and do not recognize him as henry@gmail.com which he used in his last order. Which would be correct behaviour. From RFC 5321, part 2.4, "The local-part of a mailbox MUST BE treated as case sensitive."
2038AD 7y ago
>But at least you have a brand new customer: Henry@gmail.com. Since you do not lowercase the email and do not recognise him as henry@gmail.com which he used in his last order You may not like it but this is fully standards-compliant behaviour. RFC 5321 states: >The local-part of a mailbox MUST BE treated as case sensitive I'm not saying this is a good behaviour and the RFC also discourages exploiting it further. There are more ways an email address can be equivalent though :-). Since we know it's gmail in this case we know the email address is both case insensitive and dots don't matter. Comments are also allowed in email addresses (in both the local part and the domain part). Here's a couple examples from RFC 2822: pete(his account)@silly.test(his host) c@(Chris's host.)public.example [0] https://tools.ietf.org/html/rfc5321 https://tools.ietf.org/html/rfc5321 [1] https://support.google.com/mail/answer/7436150 https://support.google.com/mail/answer/7436150 [2] https://www.ietf.org/rfc/rfc2822.html#appendix-A.5 https://www.ietf.org/rfc/rfc2822.html#appendix-A.5
BoppreH 7y ago
That's a good point, these problems are not easy to solve. But they should be solved. For example, the user types "7" into the UI control. The UI control knows it's supposed to hold a number, so it has methods that return the parsed number (done by the OS, very carefully), or an error to the user. Similarly, you don't send the product name, but the ID of the product the user selected from their fuzzy search. I would go as far as using this technique for the email too.

thaumasiotes 7y ago

I want to note a separate issue of defensive coding that comes up in the writeup: > GitHub's forgot password feature could be compromised because the system lowercased the provided email address and compared it to the email address stored in the user database. If there was a match, GitHub would send the reset password link to the email address provided by the attacker The logical flow is: 1. Get the email address from the forgot-password request. 2. Get the email address from the database for the same account. 3. Check whether they match. 4a. If not, we're under attack -- refuse the request. 4b. If so, all is well -- send a password reset to the email address. Of course, we know the email address twice -- we asked the user for it during the password reset process (step 1), but we never needed to do that because we already had an email address on file for the account. We retrieved that email address in step 2. We know that the two addresses are the same, but, if you look at the semantics behind the variables, in step 4b we're choosing one of these two "equivalent" options, depending on which variable we use for the email address: 1. Send the account password to the account owner. 2. Send the account password to a guy who doesn't know what the password is. And these have very different risk profiles. Choosing the first option instead of the second would have prevented this attack without needing to worry about unicode case-translation issues. You never want to trust information you just received from an unknown user when you already have the same information from a more authoritative source.

gingerlime 7y ago
Yes, you're absolutely right. The real bug was sending to the "wrong" matching email. But this is what makes this bug so hard to find. You're looking at "equal" strings, so why should it make a difference if you pick A or A if A === A ? :)
- thaumasiotes 7y ago
  Hard to find, but it's the kind of thing that will hopefully come up in code review. Consider this pseudocode with helpful pseudo-hungarian notation: username = request.post_params('username') evil_email = request.post_params('email_address') user = get_user_by_name(username) good_email = user.email_address if good_email != evil_email: # Hackers! else: reset_password(user.id, evil_email) # it's fine; it's the same as user.email_address You can see the problem in the call to reset_password. Sure, it's the same as user.email_address, but if you just used user.email_address you'd be doing the right thing and you wouldn't need the comment about "don't worry; it's fine". Then you start to wonder if that line shouldn't look more like reset_password(user) and from a security perspective, it should. (I assume the reason Github allows the code to specify the email address in the first place is that an account may be associated with multiple addresses.) If it makes no difference which email address you pick, why not pick the one that represents doing something safe instead of the one that represents doing something dangerous?

giancarlostoro 7y ago

I usually store files inside UUID sub directories with UUID names. Then keep the original name saved elsewhere in a db to avoid filename collision.

simion314 7y ago

But why non english users should be forced to name their files with english characters, libraries and code that work with files should always use tests with unicode in file names.

merb 7y ago

> but I'm more and more coming to the conclusion that strings are evil and should be treated as opaque byte arrays, whose only available operation is rendering into a bounded area nope. they are not byte arrays. if treating them as such, you run into many nasty bugs. today I encountered one (C#): if (string.Length > 60) string.Remove(60) this has a fucking serious bug, especially if you try to show the string afterwards or worse save it inside a database. strings are arrays of codepoints. NEVER TREAT THEM AS BYTE/CHAR ARRAYS.

BoppreH 7y ago
I meant "byte array" as in "this is a bunch of binary data you should not look into". Like people already do with images. You wouldn't remove a byte from the middle of a JPEG file, and you shouldn't do it with strings either.