5 ms·

Given that interoperability is one of the explicitly mentioned design criteria, it seems a bit of an oversight that they forgot to specify how the encoding actu

Given that interoperability is one of the explicitly mentioned design criteria, it seems a bit of an oversight that they forgot to specify how the encoding actually works. It says nothing about how you encode the 32 bit integer into the five characters and the description of the decoding operation is also pretty vague. The five characters SHALL each be converted into a value 0 to 84, and accumulated by multiplication by 85, from most to least significant. There is of course the example and the reference implementation and everyone knows how this kind of encoding works, so you can certainly figure out how you are supposed to implement it.

drpixie 3y ago
And the author specifies Z85 only for 4 byte chunks of data - not all data comes in 32 bit chunks. If the authors want Z85 to be widely used, specifying padding would be helpful. Anyway, it all seems a bit https://xkcd.com/927/ https://xkcd.com/927/
- dullcrisp 3y ago
  It looks like they’re counting on most data being padded to 32 bits. The simplicity due to lack of padding rules seems to be the only real advantage this encoding has over base 64. Otherwise you’re right, offering up a new standard because there are too many competing base 64 standards makes no sense.
  - nabla9 3y ago
    >The simplicity due to lack of padding rules seems to be the only real advantage Because there is no padding rule given, it's impossible to use the standard to encode most binary strings. For example, how would you encode/decode these? 0x0000000000000000 0x00000000000000 0x000000000000 0x0000000000 0x00000000 0x000000
    - fwip 3y ago
      If the situation requires decoding strings of unknown length, you simply prefix the length.
      - nabla9 3y ago
        It was rhetorical question. What you describe is not defined in the format. You can't use the format to exchange strings of unknown length. If people would start using this encoding, different users would adopt different solutions padding, length prefixes etc. and it becomes mess.
        fwip 3y ago
        It's not defined in the format, sure. That doesn't mean the format is bad. The format also doesn't specify how to interpret the binary data, or how to encode multiple strings, or automatically checksum it for you. But you can obviously use the format to exchange strings of variable length, or encode both integers and images, send multiple strings, or add your own checksum. It's not "impossible to encode" that content just because you need to decide how to represent it. It's not "a mess" if some people use fixed-length strings and some use length-prefixed strings. It's just reality for any encoding scheme - you build layers around it, according to how you want to use it. The same way the context of your software determines whether this binary string is supposed to represent a float, a name, a hash, or a pixel-art masterpiece, it will also determine the appropriate serialization.
    - dullcrisp 3y ago
      You could only encode the ones whose length was a multiple of 8. 0x00000000 would be 00000. 0x0000000000000000 would be 00000 00000. How would you encode binary 000000 in hex?
      - yau8edq12i 3y ago
        "Hex(adecimal)" is not an encoding scheme, it's a number basis. "Binary 00000" as a number is just zero, which is written equivalently as "0", "00"... in hexadecimal. This is different from a chunk of data consisting of six bits that all happen to be zero. And indeed, you need to decide on how to convey that, and simply saying "hex" - or "binary" or "decimal" for that matter - is not enough, that is correct.
  - karmakaze 3y ago
    All the issues mentioned in these comments is a good indication why this encoding isn't more widely known/used. I myself typically binary compress then base64 encode.
    - bcaxis 3y ago
      For your trouble, you save some extra bytes with 85 vs 64.
      - petre 3y ago
        With the added bonus of funny characters which are a pain to encode in JSON, XML and URLs. I guess I'll keep using using URL safe Base64, thank you.
        bcaxis 3y ago
        Modifying the base alphabet is simple if there are a couple of problem characters. I use 85 in json just fine without escaping overhead. Never tried for URLs or XML.
    - Varriount 3y ago
      Good to know I'm not the only one who has done this! I once wrote a template generation tool that embedded its input configuration into its output files as base64-encoded gzip'd JSON. Worked quite well at allowing template regeneration.
    - jgalt212 3y ago
      and then the webserver probably recompresses again via gzip.
- grodriguez100 3y ago
  That xkcd comic was the first thing that came to mind as well. The article explains how Base64 is “problematic because it has more than a dozen variants”. So instead of picking one, let’s invent a new standard.
- mnw21cam 3y ago
  That XKCD is so old now that the alt-text: > Fortunately, the charging one has been solved now that we've all standardized on mini-USB. Or is it micro-USB? Shit. is hilariously even more true.
dullcrisp 3y ago
The mapping seems fairly clear, even if an implementation is not provided. Just write the 32-bit integer in base 85, using the provided numerals. Most significant numeral first. Since 2^32 < 85^5 you can do it in five characters, though some sequences cannot be achieved. Just like you would do with any other numeric base.
- danbruc 3y ago
  But where does it say that it has to be done like that? There are a lot of functions that map [0, 2³² - 1] to [0, 84]⁵ or to [0, 85⁵ - 1] which they at least imply by mentioning the base-85 representation. There is of course the canonical mapping, the identity function, but it never says that we are supposed to use it.
gary_0 3y ago
Given that 85^5 > 2^32, it's a little strange that they also didn't specify what happens if the text encodes a number that doesn't fit in 4 bytes.
- dullcrisp 3y ago
  Then it’s invalid, same as if it includes unexpected characters or isn’t a multiple of five characters long.
  - gary_0 3y ago
    The "reference implementation" in C seems to just let the int wrap around. It also does an out of bounds array access if the string contains non-ASCII chars.
    - dullcrisp 3y ago
      Sounds like the reference implementation only specifies what should be done with valid values then? I’d avoid using it in production without separate validation.
- Dwedit 3y ago
  You could theoretically put 1, 2, or 3 byte sequences into the space between 4294967296 and 4437053124. That gap holds over 27 bits of information. Example: 0 to 4294967295 = 4 bytes 4294967296 to 4311744511 = 3 bytes (subtract 4294967296 first) 4311744512 to 4311810047 = 2 bytes (subtract 4311744512 first) 4311810048 to 4311810303 = 1 byte (subtract 4311810048 first) Then you're still left with 125242821 remaining numbers, over 26 bits.
  - tromp 3y ago
    That is one way to extend the encoding from 4-byte chunks to arbitrary byte lengths. Another is to encode 4k+i bytes as 5k+1+i base-85 characters, for 0<i<4. That way the encoding length immediately determines the input length. And there's again plenty of space since 85^{i+1} > 256^i for i < 4. This leaves encoding lengths of 5k+1 unused. These could be used to support arbitrary bit lengths, i.e. for encoding an input of 4(k-1) bytes + i bits, with 0<i<32. Set the final byte to i, and let the preceding 5 base-85 characters encode the i bits with 0 padding.
GoblinSlayer 3y ago
Also the interoperability rationale isn't future proof: when someone invents a variant of Z85, say, with a padding rule, then Z85 will have the same interoperability problem as base64.

drpixie 3y ago

And the author specifies Z85 only for 4 byte chunks of data - not all data comes in 32 bit chunks. If the authors want Z85 to be widely used, specifying padding would be helpful. Anyway, it all seems a bit https://xkcd.com/927/ https://xkcd.com/927/

dullcrisp 3y ago
It looks like they’re counting on most data being padded to 32 bits. The simplicity due to lack of padding rules seems to be the only real advantage this encoding has over base 64. Otherwise you’re right, offering up a new standard because there are too many competing base 64 standards makes no sense.
- nabla9 3y ago
  >The simplicity due to lack of padding rules seems to be the only real advantage Because there is no padding rule given, it's impossible to use the standard to encode most binary strings. For example, how would you encode/decode these? 0x0000000000000000 0x00000000000000 0x000000000000 0x0000000000 0x00000000 0x000000
  - fwip 3y ago
    If the situation requires decoding strings of unknown length, you simply prefix the length.
    - nabla9 3y ago
      It was rhetorical question. What you describe is not defined in the format. You can't use the format to exchange strings of unknown length. If people would start using this encoding, different users would adopt different solutions padding, length prefixes etc. and it becomes mess.
      - fwip 3y ago
        It's not defined in the format, sure. That doesn't mean the format is bad. The format also doesn't specify how to interpret the binary data, or how to encode multiple strings, or automatically checksum it for you. But you can obviously use the format to exchange strings of variable length, or encode both integers and images, send multiple strings, or add your own checksum. It's not "impossible to encode" that content just because you need to decide how to represent it. It's not "a mess" if some people use fixed-length strings and some use length-prefixed strings. It's just reality for any encoding scheme - you build layers around it, according to how you want to use it. The same way the context of your software determines whether this binary string is supposed to represent a float, a name, a hash, or a pixel-art masterpiece, it will also determine the appropriate serialization.
  - dullcrisp 3y ago
    You could only encode the ones whose length was a multiple of 8. 0x00000000 would be 00000. 0x0000000000000000 would be 00000 00000. How would you encode binary 000000 in hex?
    - yau8edq12i 3y ago
      "Hex(adecimal)" is not an encoding scheme, it's a number basis. "Binary 00000" as a number is just zero, which is written equivalently as "0", "00"... in hexadecimal. This is different from a chunk of data consisting of six bits that all happen to be zero. And indeed, you need to decide on how to convey that, and simply saying "hex" - or "binary" or "decimal" for that matter - is not enough, that is correct.
- karmakaze 3y ago
  All the issues mentioned in these comments is a good indication why this encoding isn't more widely known/used. I myself typically binary compress then base64 encode.
  - bcaxis 3y ago
    For your trouble, you save some extra bytes with 85 vs 64.
    - petre 3y ago
      With the added bonus of funny characters which are a pain to encode in JSON, XML and URLs. I guess I'll keep using using URL safe Base64, thank you.
      - bcaxis 3y ago
        Modifying the base alphabet is simple if there are a couple of problem characters. I use 85 in json just fine without escaping overhead. Never tried for URLs or XML.
  - Varriount 3y ago
    Good to know I'm not the only one who has done this! I once wrote a template generation tool that embedded its input configuration into its output files as base64-encoded gzip'd JSON. Worked quite well at allowing template regeneration.
  - jgalt212 3y ago
    and then the webserver probably recompresses again via gzip.
grodriguez100 3y ago
That xkcd comic was the first thing that came to mind as well. The article explains how Base64 is “problematic because it has more than a dozen variants”. So instead of picking one, let’s invent a new standard.
mnw21cam 3y ago
That XKCD is so old now that the alt-text: > Fortunately, the charging one has been solved now that we've all standardized on mini-USB. Or is it micro-USB? Shit. is hilariously even more true.

dullcrisp 3y ago

The mapping seems fairly clear, even if an implementation is not provided. Just write the 32-bit integer in base 85, using the provided numerals. Most significant numeral first. Since 2^32 < 85^5 you can do it in five characters, though some sequences cannot be achieved. Just like you would do with any other numeric base.

danbruc 3y ago
But where does it say that it has to be done like that? There are a lot of functions that map [0, 2³² - 1] to [0, 84]⁵ or to [0, 85⁵ - 1] which they at least imply by mentioning the base-85 representation. There is of course the canonical mapping, the identity function, but it never says that we are supposed to use it.

gary_0 3y ago

Given that 85^5 > 2^32, it's a little strange that they also didn't specify what happens if the text encodes a number that doesn't fit in 4 bytes.

dullcrisp 3y ago
Then it’s invalid, same as if it includes unexpected characters or isn’t a multiple of five characters long.
- gary_0 3y ago
  The "reference implementation" in C seems to just let the int wrap around. It also does an out of bounds array access if the string contains non-ASCII chars.
  - dullcrisp 3y ago
    Sounds like the reference implementation only specifies what should be done with valid values then? I’d avoid using it in production without separate validation.
Dwedit 3y ago
You could theoretically put 1, 2, or 3 byte sequences into the space between 4294967296 and 4437053124. That gap holds over 27 bits of information. Example: 0 to 4294967295 = 4 bytes 4294967296 to 4311744511 = 3 bytes (subtract 4294967296 first) 4311744512 to 4311810047 = 2 bytes (subtract 4311744512 first) 4311810048 to 4311810303 = 1 byte (subtract 4311810048 first) Then you're still left with 125242821 remaining numbers, over 26 bits.
- tromp 3y ago
  That is one way to extend the encoding from 4-byte chunks to arbitrary byte lengths. Another is to encode 4k+i bytes as 5k+1+i base-85 characters, for 0<i<4. That way the encoding length immediately determines the input length. And there's again plenty of space since 85^{i+1} > 256^i for i < 4. This leaves encoding lengths of 5k+1 unused. These could be used to support arbitrary bit lengths, i.e. for encoding an input of 4(k-1) bytes + i bits, with 0<i<32. Set the final byte to i, and let the preceding 5 base-85 characters encode the i bits with 0 padding.

GoblinSlayer 3y ago

Also the interoperability rationale isn't future proof: when someone invents a variant of Z85, say, with a padding rule, then Z85 will have the same interoperability problem as base64.