4 ms·

Is there any reasonably popular encoding that is "hole free / complete" in the sense any sequence of bytes has a valid decoding ? I use B85 or B64 when needed b

by srean 10y ago

Is there any reasonably popular encoding that is "hole free / complete" in the sense any sequence of bytes has a valid decoding ? I use B85 or B64 when needed but was wondering if there is a Unicode encoding that will do the job.

deleted 10y ago
[deleted]
masklinn 10y ago
> Is there any reasonably popular encoding that is "hole free / complete" in the sense any sequence of bytes has a valid decoding ? That's a property of the ISO-8859 encodings, it turns out to be pretty terrible as you have no way of distinguishing utter garbage/nonsense from actual text. > I use B85 or B64 when needed but was wondering if there is a Unicode encoding that will do the job. Base64 and 85 are pretty much the opposite of Unicode encodings. Encodings turn text (abstract unicode codepoints) into actual bytes you can store, Base64 and Base85 turn bytes into ASCII or ASCII-compatible text.
- srean 10y ago
  > Base64 and 85 are pretty much the opposite I know and that's the purpose I use it for: to smuggle in binary data as text, but no reason why an encoding scheme should not be dual purpose. Thanks for the tip about ISO-8859
  - edblarney 10y ago
    In b64, you're going to want to pick a short set of characters that can be encoded clearly and simply, i.e. ascii chars. And the shorter the better, i.e. even in 6 bits. You take the first six bits of your binary data and convert to some ascii char mapping. And then the the next 6 bits and so on. You can't do that with unicode, and it wouldn't make sense for any other encoding standard. They are completely different things.
    - Moru 10y ago
      Mabe srean wants a more efficient way of base64 a binary file into some text-only media like email. This depends totally on where you want to put it. If it's an email you are pretty stuck with base64, if it's a string that nothing else touches, you can use the binary data directly (eg: iso-8859) :)
      - srean 10y ago
        That's indeed right. B85 is already a little more efficient than B64, but was wondering if one could abuse Unicode for this. Its a really silly, stupid situation I need this for, exchanging data with Python and I can only use Unicode strings.
        masklinn 10y ago
        > That's indeed right. B85 is already a little more efficient than B64, but was wondering if one could abuse Unicode for this. Sure, kinda: https://github.com/pfrazee/base-emoji https://github.com/pfrazee/base-emoji (it's a base256 using emoji), but then you still need to encode that text, which is going to require 4 bytes per symbol, so I'm not sure you're going to get any actual gain over B64/B85 in the end. There's also the option of using a subset of the U+0100~U+07FF range (though it contains a diacticial block which may not be ideal) as it encodes to 2 bytes in both UTF-8 and UTF-16 (though there are diacritics in these blocks, and some of the codepoints are reserved but not allocated so…).
jcranmer 10y ago
Every 1-byte charset that uses high bits (i.e., not UTF-7, ASCII, or EBCDIC) has this property. Depending on how you feel about whether unpaired surrogates constitute a valid decoding, UTF-16 has that property as well. It's not a very good property to have. UTF-8's invalid decodings means that it's very easy to detect if you're not using UTF-8: iconv's charset detector will rule certainty of UTF-8 if it sees just 3 valid UTF-8 multibyte sequences and no invalid sequences, and that's pretty much the only reliable detector.
Manishearth 10y ago
Javascript. It uses bad UTF-16, where unpaired surrogates are allowed. They're meaningless, but allowed.
doubleunplussed 10y ago
Python hit this issue in the last few years after the transition to unicode-everything. Since the filesystem is supposed to have an encoding, but doesn't enforce it, Python needed to choose what to do with a filename that contained bytes that were invalid for the encoding. In order to make a round trip of decoding and encoding result in the same bytes, they decided to use certain surrogates to represent the invalid bytes: https://www.python.org/dev/peps/pep-0383/ https://www.python.org/dev/peps/pep-0383/ So with most systems being UTF8, this means that the encoding is "UTF8, except with bytes that would be invalid UTF8 mapping to this range of surrogates". So you've got an encoding with no holes, which is compatible with all valid UTF8 (since those surrogates are invalid UTF8). It's not common, so it doesn't match that criterion of yours,but I think it's a great solution and hope it catches on. We need a solution for "text, except with graceful fallback when someone decided to put arbitrary bytes in there"
- srean 10y ago
  Aha ! thanks for the anecdote. My situation is indeed very similar, and yeah Python is involved, but that's not where I would put the blame.
  - doubleunplussed 10y ago
    Oh, I'm not saying Python is to blame in any sense, rather, that the core devs had the same problem as you and hit on a neat solution.
    - srean 10y ago
      Oh ! sorry never thought you were saying Python was the source of the problem. Since Python happened to be a common feature common between our two scenarios, wanted to caution other HN'ers not to put blame on Python. In my case Python was indeed doing the right thing.

by srean 10y ago

deleted 10y ago

[deleted]

masklinn 10y ago

> Is there any reasonably popular encoding that is "hole free / complete" in the sense any sequence of bytes has a valid decoding ? That's a property of the ISO-8859 encodings, it turns out to be pretty terrible as you have no way of distinguishing utter garbage/nonsense from actual text. > I use B85 or B64 when needed but was wondering if there is a Unicode encoding that will do the job. Base64 and 85 are pretty much the opposite of Unicode encodings. Encodings turn text (abstract unicode codepoints) into actual bytes you can store, Base64 and Base85 turn bytes into ASCII or ASCII-compatible text.

srean 10y ago
> Base64 and 85 are pretty much the opposite I know and that's the purpose I use it for: to smuggle in binary data as text, but no reason why an encoding scheme should not be dual purpose. Thanks for the tip about ISO-8859
- edblarney 10y ago
  In b64, you're going to want to pick a short set of characters that can be encoded clearly and simply, i.e. ascii chars. And the shorter the better, i.e. even in 6 bits. You take the first six bits of your binary data and convert to some ascii char mapping. And then the the next 6 bits and so on. You can't do that with unicode, and it wouldn't make sense for any other encoding standard. They are completely different things.
  - Moru 10y ago
    Mabe srean wants a more efficient way of base64 a binary file into some text-only media like email. This depends totally on where you want to put it. If it's an email you are pretty stuck with base64, if it's a string that nothing else touches, you can use the binary data directly (eg: iso-8859) :)
    - srean 10y ago
      That's indeed right. B85 is already a little more efficient than B64, but was wondering if one could abuse Unicode for this. Its a really silly, stupid situation I need this for, exchanging data with Python and I can only use Unicode strings.
      - masklinn 10y ago
        > That's indeed right. B85 is already a little more efficient than B64, but was wondering if one could abuse Unicode for this. Sure, kinda: https://github.com/pfrazee/base-emoji https://github.com/pfrazee/base-emoji (it's a base256 using emoji), but then you still need to encode that text, which is going to require 4 bytes per symbol, so I'm not sure you're going to get any actual gain over B64/B85 in the end. There's also the option of using a subset of the U+0100~U+07FF range (though it contains a diacticial block which may not be ideal) as it encodes to 2 bytes in both UTF-8 and UTF-16 (though there are diacritics in these blocks, and some of the codepoints are reserved but not allocated so…).

jcranmer 10y ago

Every 1-byte charset that uses high bits (i.e., not UTF-7, ASCII, or EBCDIC) has this property. Depending on how you feel about whether unpaired surrogates constitute a valid decoding, UTF-16 has that property as well. It's not a very good property to have. UTF-8's invalid decodings means that it's very easy to detect if you're not using UTF-8: iconv's charset detector will rule certainty of UTF-8 if it sees just 3 valid UTF-8 multibyte sequences and no invalid sequences, and that's pretty much the only reliable detector.

Manishearth 10y ago

Javascript. It uses bad UTF-16, where unpaired surrogates are allowed. They're meaningless, but allowed.

doubleunplussed 10y ago

Python hit this issue in the last few years after the transition to unicode-everything. Since the filesystem is supposed to have an encoding, but doesn't enforce it, Python needed to choose what to do with a filename that contained bytes that were invalid for the encoding. In order to make a round trip of decoding and encoding result in the same bytes, they decided to use certain surrogates to represent the invalid bytes: https://www.python.org/dev/peps/pep-0383/ https://www.python.org/dev/peps/pep-0383/ So with most systems being UTF8, this means that the encoding is "UTF8, except with bytes that would be invalid UTF8 mapping to this range of surrogates". So you've got an encoding with no holes, which is compatible with all valid UTF8 (since those surrogates are invalid UTF8). It's not common, so it doesn't match that criterion of yours,but I think it's a great solution and hope it catches on. We need a solution for "text, except with graceful fallback when someone decided to put arbitrary bytes in there"

srean 10y ago
Aha ! thanks for the anecdote. My situation is indeed very similar, and yeah Python is involved, but that's not where I would put the blame.
- doubleunplussed 10y ago
  Oh, I'm not saying Python is to blame in any sense, rather, that the core devs had the same problem as you and hit on a neat solution.
  - srean 10y ago
    Oh ! sorry never thought you were saying Python was the source of the problem. Since Python happened to be a common feature common between our two scenarios, wanted to caution other HN'ers not to put blame on Python. In my case Python was indeed doing the right thing.