5 ms·

The first time I saw perfectly valid Unicode take down an RSS reader (in ~2003), it was an ü that did it. I have taken to calling that the kaboomlaut in unit te

The first time I saw perfectly valid Unicode take down an RSS reader (in ~2003), it was an ü that did it. I have taken to calling that the kaboomlaut in unit tests.

azalemeth 4y ago
The are quite a lot of "gotchas" with passwords being Unicode though. One of my favourite tricks is to include a null byte in passwords and see what happens. Sudo treats it as the end of the string. Passwd gives "token manipulation error" and nopes out. Many websites seem to fail in interesting ways...
- deleted 4y ago
  [deleted]
- Dylan16807 4y ago
  I wouldn't call that "being Unicode" though, since null byte is in ASCII.
  - msla 4y ago
    It depends on the encoding: In UTF-16, a lot of characters have 0x00 as part of their encoding. That's the null byte, even if the software isn't technically supposed to see it that way, so it's a valid test. (Easy, too: Any character that's also in ASCII will be represented with a 0x00 byte in UTF-16.) Of course, UTF-8 solves that problem, plus the problem of surrogate pairs.
    - fsckboy 4y ago
      > That's the null byte, even if the software isn't technically supposed to see it that way that's a null byte, even if the spec says it's not the string terminator. i just think it's worth referring to things by using words that more precise semantics if possible
- cmeacham98 4y ago
  This has nothing to do with Unicode (assuming that by "Unicode" you mean "UTF-8"). The null byte has the exact same meaning in UTF-8 as ASCII: https://stackoverflow.com/questions/6907297/can-utf-8-contain-zero-byte https://stackoverflow.com/questions/6907297/can-utf-8-contai...
  - aidenn0 4y ago
    If by Unicode you mean UCS-2 or UTF-16 though it applies.
  - lelanthran 4y ago
    > The null byte has the exact same meaning in UTF-8 as ASCII: https://stackoverflow.com/questions/6907297/can-utf-8-contai https://stackoverflow.com/questions/6907297/can-utf-8-contai... And what meaning is that? That link doesn't refer to an authority on the meaning of the nul byte.
    - bouke 4y ago
      In C, strings are nul terminated. Thus when at some point your password is passed along a tool written in C (or any language with the same behaviour) it’ll be truncated at that position.
- Eleison23 4y ago
  [dead]
- chungy 4y ago
  A bigger deal happens with ambiguities concerning composed versus decomposed characters. Linux and Windows tend to prefer writing composed characters, whereas Mac OS will prefer writing decomposed characters. The letter é will have two distinct byte sequences to represent the same character (more precisely, the same grapheme).
  - Gare 4y ago
    Shouldn't normalization functions handle that?
    - chungy 4y ago
      Yes, now see if any random service actually uses said functions...
      - still_grokking 4y ago
        If Unicode weren't broken it wouldn't need any "normalization functions". And no, I don't have anything better to propose at the moment. Unicode is a mess, but currently without realistic alternatives. Still I would like to live long enough to see that we finally manage to create some proper text format without all kinds of crazy gotchas and deficiencies.
        sshine 4y ago
        Care to elaborate? Are you suggesting that a better version of Unicode does not allow combining characters? [1] Some languages have an explosive amount of characters if you regard each possible modifier in combination with each base character, some of which aren't practically used. By allowing the composition of modifiers and base letters, all the historically used ones are available, and all the odd ones are technically expressible. I'm not sure how you encode all the world's languages, live and dead ones, without a few gotchas and deficiencies. What would an always-normalized Unicode look like, if not either having really, really many characters, or having a non-trivial syntax? [1]: https://www.unicode.org/versions/Unicode15.0.0/ch03.pdf#G30602 https://www.unicode.org/versions/Unicode15.0.0/ch03.pdf#G306...
- doublepg23 4y ago
  I've run into a service where having a password longer than 15 characters _truncates_ it but the password change dialog does not so it's stuck in limbo.
  - aidenn0 4y ago
    I ran into a service where the password entry input enforced a shorter length than the change password. If I pasted my password it failed, but if I auto-filled from bitwarden it succeeded. Took me weeks to figure out why...
  - alisonatwork 4y ago
    One site I went to lowercased my password before storing it unencrypted in the DB, which I only discovered after failing to log in and asking for a password reset mail, so it helpfully emailed me my clear text lowercased password.
    - cdaringe 4y ago
      Incredible. Please tell us who did this, not to shame them, bit to help us!
      - alisonatwork 4y ago
        It was an online store I bought something from quite a few years back. I would hope most stores are using more modern back ends nowadays that doesn't allow this to happen.
    - skim_milk 4y ago
      I worked at a place that did this, case-insensitive passwords and all. The worst part? They had unencrypted production database backups on all of their dev laptops, the majority of which left the premises after every night. I couldn't get a figure of how many laptops were lost but statistically speaking there had to have been at least 3 lost laptops with unencrypted production database backups per year using industry statistics. Eventually I convinced leadership to invest in basic security after conservative but still embarrassingly high 6-to-7-figure estimates of annual loss expectancy that only took a measly 5 figures a year to eliminate 75% of the risk, but the company only went around to it a long while after I left the place.
itslennysfault 4y ago
When AOL 4.0 came out they added the ability to embed images in emails, and there was a super fun bug where you could put a certain unicode char into the file name of an image and it would blue screen the recipient if they opened the email. Fun times.
mr_toad 4y ago
Umlauts are in Latin1, so it’s pretty shoddy software that couldn’t deal with that.
- dmitriid 4y ago
  As late as 2020 I encountered a European shop that couldn't handle my Swedish address that had an å in it. That is, it accepted the address on the website, but then it got screwed in the delivery. I'm still in awe.
  - 0ld 4y ago
    you'd be even more impressed to find out how often polish characters (ąęłćśżźńó) get totally screwed in... poland itself
  - cerved 4y ago
    UPS sent me a parcel and replaced ä with ae
    - dgellow 4y ago
      That’s a valid and common replacement for German umlauts
      - cerved 4y ago
        That was not a German umlaut
  - tharkun__ 4y ago
    Sounds 'normal' if the website was built in a recent stack in recent years. The shipping backend might run on a mainframe, almost if not fully unchanged from 40-50 years ago and blissfully unaware of anything but the mainframe quivalent of your standard ASCII. You can see large stores run all their inventory and such on mainframes. If you ever look over someone's shoulder. My insurance also does but it looks like they don't because it has a nice standard Windows look on the 'inside' of the window. Looked at the title bar and yep, it's just a 3270 terminal emulator that renders fields as nice standard windows input fields that follow whatever theme your computer is set to instead of your telltale black background "DOS" look.
xena 4y ago
dümlaut

The first time I saw perfectly valid Unicode take down an RSS reader (in ~2003), it was an ü that did it. I have taken to calling that the kaboomlaut in unit tests.

azalemeth 4y ago

The are quite a lot of "gotchas" with passwords being Unicode though. One of my favourite tricks is to include a null byte in passwords and see what happens. Sudo treats it as the end of the string. Passwd gives "token manipulation error" and nopes out. Many websites seem to fail in interesting ways...

deleted 4y ago
[deleted]
Dylan16807 4y ago
I wouldn't call that "being Unicode" though, since null byte is in ASCII.
- msla 4y ago
  It depends on the encoding: In UTF-16, a lot of characters have 0x00 as part of their encoding. That's the null byte, even if the software isn't technically supposed to see it that way, so it's a valid test. (Easy, too: Any character that's also in ASCII will be represented with a 0x00 byte in UTF-16.) Of course, UTF-8 solves that problem, plus the problem of surrogate pairs.
  - fsckboy 4y ago
    > That's the null byte, even if the software isn't technically supposed to see it that way that's a null byte, even if the spec says it's not the string terminator. i just think it's worth referring to things by using words that more precise semantics if possible
cmeacham98 4y ago
This has nothing to do with Unicode (assuming that by "Unicode" you mean "UTF-8"). The null byte has the exact same meaning in UTF-8 as ASCII: https://stackoverflow.com/questions/6907297/can-utf-8-contain-zero-byte https://stackoverflow.com/questions/6907297/can-utf-8-contai...
- aidenn0 4y ago
  If by Unicode you mean UCS-2 or UTF-16 though it applies.
- lelanthran 4y ago
  > The null byte has the exact same meaning in UTF-8 as ASCII: https://stackoverflow.com/questions/6907297/can-utf-8-contai https://stackoverflow.com/questions/6907297/can-utf-8-contai... And what meaning is that? That link doesn't refer to an authority on the meaning of the nul byte.
  - bouke 4y ago
    In C, strings are nul terminated. Thus when at some point your password is passed along a tool written in C (or any language with the same behaviour) it’ll be truncated at that position.
Eleison23 4y ago
[dead]
chungy 4y ago
A bigger deal happens with ambiguities concerning composed versus decomposed characters. Linux and Windows tend to prefer writing composed characters, whereas Mac OS will prefer writing decomposed characters. The letter é will have two distinct byte sequences to represent the same character (more precisely, the same grapheme).
- Gare 4y ago
  Shouldn't normalization functions handle that?
  - chungy 4y ago
    Yes, now see if any random service actually uses said functions...
    - still_grokking 4y ago
      If Unicode weren't broken it wouldn't need any "normalization functions". And no, I don't have anything better to propose at the moment. Unicode is a mess, but currently without realistic alternatives. Still I would like to live long enough to see that we finally manage to create some proper text format without all kinds of crazy gotchas and deficiencies.
      - sshine 4y ago
        Care to elaborate? Are you suggesting that a better version of Unicode does not allow combining characters? [1] Some languages have an explosive amount of characters if you regard each possible modifier in combination with each base character, some of which aren't practically used. By allowing the composition of modifiers and base letters, all the historically used ones are available, and all the odd ones are technically expressible. I'm not sure how you encode all the world's languages, live and dead ones, without a few gotchas and deficiencies. What would an always-normalized Unicode look like, if not either having really, really many characters, or having a non-trivial syntax? [1]: https://www.unicode.org/versions/Unicode15.0.0/ch03.pdf#G30602 https://www.unicode.org/versions/Unicode15.0.0/ch03.pdf#G306...
doublepg23 4y ago
I've run into a service where having a password longer than 15 characters _truncates_ it but the password change dialog does not so it's stuck in limbo.
- aidenn0 4y ago
  I ran into a service where the password entry input enforced a shorter length than the change password. If I pasted my password it failed, but if I auto-filled from bitwarden it succeeded. Took me weeks to figure out why...
- alisonatwork 4y ago
  One site I went to lowercased my password before storing it unencrypted in the DB, which I only discovered after failing to log in and asking for a password reset mail, so it helpfully emailed me my clear text lowercased password.
  - cdaringe 4y ago
    Incredible. Please tell us who did this, not to shame them, bit to help us!
    - alisonatwork 4y ago
      It was an online store I bought something from quite a few years back. I would hope most stores are using more modern back ends nowadays that doesn't allow this to happen.
  - skim_milk 4y ago
    I worked at a place that did this, case-insensitive passwords and all. The worst part? They had unencrypted production database backups on all of their dev laptops, the majority of which left the premises after every night. I couldn't get a figure of how many laptops were lost but statistically speaking there had to have been at least 3 lost laptops with unencrypted production database backups per year using industry statistics. Eventually I convinced leadership to invest in basic security after conservative but still embarrassingly high 6-to-7-figure estimates of annual loss expectancy that only took a measly 5 figures a year to eliminate 75% of the risk, but the company only went around to it a long while after I left the place.

itslennysfault 4y ago

When AOL 4.0 came out they added the ability to embed images in emails, and there was a super fun bug where you could put a certain unicode char into the file name of an image and it would blue screen the recipient if they opened the email. Fun times.

mr_toad 4y ago

Umlauts are in Latin1, so it’s pretty shoddy software that couldn’t deal with that.

dmitriid 4y ago
As late as 2020 I encountered a European shop that couldn't handle my Swedish address that had an å in it. That is, it accepted the address on the website, but then it got screwed in the delivery. I'm still in awe.
- 0ld 4y ago
  you'd be even more impressed to find out how often polish characters (ąęłćśżźńó) get totally screwed in... poland itself
- cerved 4y ago
  UPS sent me a parcel and replaced ä with ae
  - dgellow 4y ago
    That’s a valid and common replacement for German umlauts
    - cerved 4y ago
      That was not a German umlaut
- tharkun__ 4y ago
  Sounds 'normal' if the website was built in a recent stack in recent years. The shipping backend might run on a mainframe, almost if not fully unchanged from 40-50 years ago and blissfully unaware of anything but the mainframe quivalent of your standard ASCII. You can see large stores run all their inventory and such on mainframes. If you ever look over someone's shoulder. My insurance also does but it looks like they don't because it has a nice standard Windows look on the 'inside' of the window. Looked at the title bar and yep, it's just a 3270 terminal emulator that renders fields as nice standard windows input fields that follow whatever theme your computer is set to instead of your telltale black background "DOS" look.

xena 4y ago

dümlaut