6 ms·

The are quite a lot of "gotchas" with passwords being Unicode though. One of my favourite tricks is to include a null byte in passwords and see what happens. Su

The are quite a lot of "gotchas" with passwords being Unicode though. One of my favourite tricks is to include a null byte in passwords and see what happens. Sudo treats it as the end of the string. Passwd gives "token manipulation error" and nopes out. Many websites seem to fail in interesting ways...

deleted 4y ago
[deleted]
Dylan16807 4y ago
I wouldn't call that "being Unicode" though, since null byte is in ASCII.
- msla 4y ago
  It depends on the encoding: In UTF-16, a lot of characters have 0x00 as part of their encoding. That's the null byte, even if the software isn't technically supposed to see it that way, so it's a valid test. (Easy, too: Any character that's also in ASCII will be represented with a 0x00 byte in UTF-16.) Of course, UTF-8 solves that problem, plus the problem of surrogate pairs.
  - fsckboy 4y ago
    > That's the null byte, even if the software isn't technically supposed to see it that way that's a null byte, even if the spec says it's not the string terminator. i just think it's worth referring to things by using words that more precise semantics if possible
cmeacham98 4y ago
This has nothing to do with Unicode (assuming that by "Unicode" you mean "UTF-8"). The null byte has the exact same meaning in UTF-8 as ASCII: https://stackoverflow.com/questions/6907297/can-utf-8-contain-zero-byte https://stackoverflow.com/questions/6907297/can-utf-8-contai...
- aidenn0 4y ago
  If by Unicode you mean UCS-2 or UTF-16 though it applies.
- lelanthran 4y ago
  > The null byte has the exact same meaning in UTF-8 as ASCII: https://stackoverflow.com/questions/6907297/can-utf-8-contai https://stackoverflow.com/questions/6907297/can-utf-8-contai... And what meaning is that? That link doesn't refer to an authority on the meaning of the nul byte.
  - bouke 4y ago
    In C, strings are nul terminated. Thus when at some point your password is passed along a tool written in C (or any language with the same behaviour) it’ll be truncated at that position.
Eleison23 4y ago
[dead]
chungy 4y ago
A bigger deal happens with ambiguities concerning composed versus decomposed characters. Linux and Windows tend to prefer writing composed characters, whereas Mac OS will prefer writing decomposed characters. The letter é will have two distinct byte sequences to represent the same character (more precisely, the same grapheme).
- Gare 4y ago
  Shouldn't normalization functions handle that?
  - chungy 4y ago
    Yes, now see if any random service actually uses said functions...
    - still_grokking 4y ago
      If Unicode weren't broken it wouldn't need any "normalization functions". And no, I don't have anything better to propose at the moment. Unicode is a mess, but currently without realistic alternatives. Still I would like to live long enough to see that we finally manage to create some proper text format without all kinds of crazy gotchas and deficiencies.
      - sshine 4y ago
        Care to elaborate? Are you suggesting that a better version of Unicode does not allow combining characters? [1] Some languages have an explosive amount of characters if you regard each possible modifier in combination with each base character, some of which aren't practically used. By allowing the composition of modifiers and base letters, all the historically used ones are available, and all the odd ones are technically expressible. I'm not sure how you encode all the world's languages, live and dead ones, without a few gotchas and deficiencies. What would an always-normalized Unicode look like, if not either having really, really many characters, or having a non-trivial syntax? [1]: https://www.unicode.org/versions/Unicode15.0.0/ch03.pdf#G30602 https://www.unicode.org/versions/Unicode15.0.0/ch03.pdf#G306...
doublepg23 4y ago
I've run into a service where having a password longer than 15 characters _truncates_ it but the password change dialog does not so it's stuck in limbo.
- aidenn0 4y ago
  I ran into a service where the password entry input enforced a shorter length than the change password. If I pasted my password it failed, but if I auto-filled from bitwarden it succeeded. Took me weeks to figure out why...
- alisonatwork 4y ago
  One site I went to lowercased my password before storing it unencrypted in the DB, which I only discovered after failing to log in and asking for a password reset mail, so it helpfully emailed me my clear text lowercased password.
  - cdaringe 4y ago
    Incredible. Please tell us who did this, not to shame them, bit to help us!
    - alisonatwork 4y ago
      It was an online store I bought something from quite a few years back. I would hope most stores are using more modern back ends nowadays that doesn't allow this to happen.
  - skim_milk 4y ago
    I worked at a place that did this, case-insensitive passwords and all. The worst part? They had unencrypted production database backups on all of their dev laptops, the majority of which left the premises after every night. I couldn't get a figure of how many laptops were lost but statistically speaking there had to have been at least 3 lost laptops with unencrypted production database backups per year using industry statistics. Eventually I convinced leadership to invest in basic security after conservative but still embarrassingly high 6-to-7-figure estimates of annual loss expectancy that only took a measly 5 figures a year to eliminate 75% of the risk, but the company only went around to it a long while after I left the place.

deleted 4y ago

[deleted]

Dylan16807 4y ago

I wouldn't call that "being Unicode" though, since null byte is in ASCII.

msla 4y ago
It depends on the encoding: In UTF-16, a lot of characters have 0x00 as part of their encoding. That's the null byte, even if the software isn't technically supposed to see it that way, so it's a valid test. (Easy, too: Any character that's also in ASCII will be represented with a 0x00 byte in UTF-16.) Of course, UTF-8 solves that problem, plus the problem of surrogate pairs.
- fsckboy 4y ago
  > That's the null byte, even if the software isn't technically supposed to see it that way that's a null byte, even if the spec says it's not the string terminator. i just think it's worth referring to things by using words that more precise semantics if possible

cmeacham98 4y ago

This has nothing to do with Unicode (assuming that by "Unicode" you mean "UTF-8"). The null byte has the exact same meaning in UTF-8 as ASCII: https://stackoverflow.com/questions/6907297/can-utf-8-contain-zero-byte https://stackoverflow.com/questions/6907297/can-utf-8-contai...

aidenn0 4y ago
If by Unicode you mean UCS-2 or UTF-16 though it applies.
lelanthran 4y ago
> The null byte has the exact same meaning in UTF-8 as ASCII: https://stackoverflow.com/questions/6907297/can-utf-8-contai https://stackoverflow.com/questions/6907297/can-utf-8-contai... And what meaning is that? That link doesn't refer to an authority on the meaning of the nul byte.
- bouke 4y ago
  In C, strings are nul terminated. Thus when at some point your password is passed along a tool written in C (or any language with the same behaviour) it’ll be truncated at that position.

Eleison23 4y ago

[dead]

chungy 4y ago

A bigger deal happens with ambiguities concerning composed versus decomposed characters. Linux and Windows tend to prefer writing composed characters, whereas Mac OS will prefer writing decomposed characters. The letter é will have two distinct byte sequences to represent the same character (more precisely, the same grapheme).

Gare 4y ago
Shouldn't normalization functions handle that?
- chungy 4y ago
  Yes, now see if any random service actually uses said functions...
  - still_grokking 4y ago
    If Unicode weren't broken it wouldn't need any "normalization functions". And no, I don't have anything better to propose at the moment. Unicode is a mess, but currently without realistic alternatives. Still I would like to live long enough to see that we finally manage to create some proper text format without all kinds of crazy gotchas and deficiencies.
    - sshine 4y ago
      Care to elaborate? Are you suggesting that a better version of Unicode does not allow combining characters? [1] Some languages have an explosive amount of characters if you regard each possible modifier in combination with each base character, some of which aren't practically used. By allowing the composition of modifiers and base letters, all the historically used ones are available, and all the odd ones are technically expressible. I'm not sure how you encode all the world's languages, live and dead ones, without a few gotchas and deficiencies. What would an always-normalized Unicode look like, if not either having really, really many characters, or having a non-trivial syntax? [1]: https://www.unicode.org/versions/Unicode15.0.0/ch03.pdf#G30602 https://www.unicode.org/versions/Unicode15.0.0/ch03.pdf#G306...

doublepg23 4y ago

I've run into a service where having a password longer than 15 characters _truncates_ it but the password change dialog does not so it's stuck in limbo.

aidenn0 4y ago
I ran into a service where the password entry input enforced a shorter length than the change password. If I pasted my password it failed, but if I auto-filled from bitwarden it succeeded. Took me weeks to figure out why...
alisonatwork 4y ago
One site I went to lowercased my password before storing it unencrypted in the DB, which I only discovered after failing to log in and asking for a password reset mail, so it helpfully emailed me my clear text lowercased password.
- cdaringe 4y ago
  Incredible. Please tell us who did this, not to shame them, bit to help us!
  - alisonatwork 4y ago
    It was an online store I bought something from quite a few years back. I would hope most stores are using more modern back ends nowadays that doesn't allow this to happen.
- skim_milk 4y ago
  I worked at a place that did this, case-insensitive passwords and all. The worst part? They had unencrypted production database backups on all of their dev laptops, the majority of which left the premises after every night. I couldn't get a figure of how many laptops were lost but statistically speaking there had to have been at least 3 lost laptops with unencrypted production database backups per year using industry statistics. Eventually I convinced leadership to invest in basic security after conservative but still embarrassingly high 6-to-7-figure estimates of annual loss expectancy that only took a measly 5 figures a year to eliminate 75% of the risk, but the company only went around to it a long while after I left the place.