5 ms·

I know a guy, who was fighting hard with unicode consortium for adding two characters with meaning "begining of blob, end of blob". Imagine how simpler the codi

by stiray 7y ago

I know a guy, who was fighting hard with unicode consortium for adding two characters with meaning "begining of blob, end of blob". Imagine how simpler the coding would be without the need to escape anything. Unfortunately he didnt succeed. They were more busy adding smileys.

bonoboTP 7y ago
You'd have to escape those new symbols if they occur within the blob.
- deleted 7y ago
  [deleted]
- nixpulvis 7y ago
  Just add a `\` /s
  - joosters 7y ago
    That's too lazy, we need unicode symbols for escaped start-blob and escaped end-blob /s
    - majewsky 7y ago
      Not sure why /s. Also, you only need escaped-end-blob. start-blob can safely be used inside a blob. So it's only 3 new codepoints, which is genuinely an interesting proposal.
      - Dylan16807 7y ago
        Because that doesn't round trip. You need to be able to distinguish whether the blob originally had "end-blob" or "escaped-end-blob". So now you need another character for double-escaping, and so on, and so on. To avoid that issue you're back to either adding a backslash, or doubling up the character inside the blob... but if you're doing that you could have just used " all along. No need for new characters! If you really don't want to change the contents of the blob, and can't length-prefix, then you could also use a new UUID as your delimiter each time you embed a blob.
        stiray 7y ago
        No. It depends only on developers. If they would start embeding it into format, they would break the rules. If not, it would work.
        Dylan16807 7y ago
        If you're telling developers not to embed that character anywhere then you only need one end of blob character. So not 'no', I'm still right in saying that having two end characters is a non-solution. And you still didn't explain why the existing ASCII control codes don't solve your problem. The suitable ones are also not supposed to appear inside text.
gregmac 7y ago
It's an interesting idea, but you still wouldn't be able to encode the byte sequence for "end of blob" as part of the blob data... without escaping it.
- stiray 7y ago
  They shouldnt occur in blob as they would be designated just for this. All other characters can occur in real world, those two could get there only by mistake.
  - bonoboTP 7y ago
    What if I want to put them in a comment like this, to discuss them? Well, you might say I shouldn't do it and should just refer to their code points, but basically that's a form of escaping as well.
  - rtpg 7y ago
    but what if you need to send a word document containing an explanation about how to use the blob characters? Turtles all the way down....
  - deleted 7y ago
    [deleted]
  - viraptor 7y ago
    What about a blob containing a page which contains blobs itself?
  - wruza 7y ago
    How much blob is blob? If it is binary, then there is a chance of clash, no matter how you dare them bytes to do that. If it is always-valid utf-8, then use 11111111 and 11111110 (or any other out-of-band markers available for source encoding) to open and close your message. Not sure what’s the point for it though.
chungy 7y ago
What's wrong with U+0001 and U+0004?
- Dylan16807 7y ago
  Or two private use characters, if you want to be ASCII-clean or 8-bit-ish-clean.
- kps 7y ago
  They have different purposes. See ISO 1745 aka ECMA-16. Basically it goes: (SOH header STX body ETX)⁺ EOT
adrianmonk 7y ago
Are we talking putting a sequence of 8-bit bytes (octets) into Unicode characters by mapping all 256 possible byte values to the first 256 Unicode code points? If so, if I'm not mistaken, this is actually less efficient than just using Base64 encoding. When you put those Unicode characters into UTF-8, the first 128 code points are going to require one byte (with a leading 0 bit). The other 128 of them are going to require two bytes. So that's 50% overhead (assuming the blob's bytes are evenly distributed) because half of the values have 0% overhead and the other half have 100% overhead. Meanwhile, Base64 sticks 6 bits in each encoded character. In 4 characters, you can fit 3 bytes of your raw info. So that's only 33% overhead.
kps 7y ago
Also, it was done already. See ISO 2111 aka ECMA-24.

by stiray 7y ago

bonoboTP 7y ago

You'd have to escape those new symbols if they occur within the blob.

deleted 7y ago
[deleted]
nixpulvis 7y ago
Just add a `\` /s
- joosters 7y ago
  That's too lazy, we need unicode symbols for escaped start-blob and escaped end-blob /s
  - majewsky 7y ago
    Not sure why /s. Also, you only need escaped-end-blob. start-blob can safely be used inside a blob. So it's only 3 new codepoints, which is genuinely an interesting proposal.
    - Dylan16807 7y ago
      Because that doesn't round trip. You need to be able to distinguish whether the blob originally had "end-blob" or "escaped-end-blob". So now you need another character for double-escaping, and so on, and so on. To avoid that issue you're back to either adding a backslash, or doubling up the character inside the blob... but if you're doing that you could have just used " all along. No need for new characters! If you really don't want to change the contents of the blob, and can't length-prefix, then you could also use a new UUID as your delimiter each time you embed a blob.
      - stiray 7y ago
        No. It depends only on developers. If they would start embeding it into format, they would break the rules. If not, it would work.
        Dylan16807 7y ago
        If you're telling developers not to embed that character anywhere then you only need one end of blob character. So not 'no', I'm still right in saying that having two end characters is a non-solution. And you still didn't explain why the existing ASCII control codes don't solve your problem. The suitable ones are also not supposed to appear inside text.

gregmac 7y ago

It's an interesting idea, but you still wouldn't be able to encode the byte sequence for "end of blob" as part of the blob data... without escaping it.

stiray 7y ago
They shouldnt occur in blob as they would be designated just for this. All other characters can occur in real world, those two could get there only by mistake.
- bonoboTP 7y ago
  What if I want to put them in a comment like this, to discuss them? Well, you might say I shouldn't do it and should just refer to their code points, but basically that's a form of escaping as well.
- rtpg 7y ago
  but what if you need to send a word document containing an explanation about how to use the blob characters? Turtles all the way down....
- deleted 7y ago
  [deleted]
- viraptor 7y ago
  What about a blob containing a page which contains blobs itself?
- wruza 7y ago
  How much blob is blob? If it is binary, then there is a chance of clash, no matter how you dare them bytes to do that. If it is always-valid utf-8, then use 11111111 and 11111110 (or any other out-of-band markers available for source encoding) to open and close your message. Not sure what’s the point for it though.

chungy 7y ago

What's wrong with U+0001 and U+0004?

Dylan16807 7y ago
Or two private use characters, if you want to be ASCII-clean or 8-bit-ish-clean.
kps 7y ago
They have different purposes. See ISO 1745 aka ECMA-16. Basically it goes: (SOH header STX body ETX)⁺ EOT

adrianmonk 7y ago

Are we talking putting a sequence of 8-bit bytes (octets) into Unicode characters by mapping all 256 possible byte values to the first 256 Unicode code points? If so, if I'm not mistaken, this is actually less efficient than just using Base64 encoding. When you put those Unicode characters into UTF-8, the first 128 code points are going to require one byte (with a leading 0 bit). The other 128 of them are going to require two bytes. So that's 50% overhead (assuming the blob's bytes are evenly distributed) because half of the values have 0% overhead and the other half have 100% overhead. Meanwhile, Base64 sticks 6 bits in each encoded character. In 4 characters, you can fit 3 bytes of your raw info. So that's only 33% overhead.

kps 7y ago

Also, it was done already. See ISO 2111 aka ECMA-24.