4 ms·

Is the narrow waste concept useful? Sure everything is bytes but most things overlay structure on top of bytes, and treating those structures as bytes is probab

Is the narrow waste concept useful? Sure everything is bytes but most things overlay structure on top of bytes, and treating those structures as bytes is probably not useful. You could use “wc” to count the lines/bytes in a json file but counting bytes is likely one of the least useful/interesting things you could do with that json file.

delusional 2y ago
Unless you wanted to compress that json. If you insisted on the structure, you'd have to create a specific algorithm just for json. Because json is just bytes, you get to compress it with a generic byte compression algorithm.
- rusk 2y ago
  Is json just bytes tho? Doesn’t it allow for variable width encodings?
  - ElevenLathe 2y ago
    No matter what encoding your JSON file is, gzip will output a compressed bag of bytes that, when unzipped, will result in the same file coming out the other end. This is true of movie codecs, Word 97 files, or anything, and none of the maintainers of those formats had to be consulted about this in order to make it work. That's what is meant by "thin waist" here.
    - rusk 2y ago
      I know, but it’s not “just bytes” as per parent comment. You cannot infer the length of the content without decoding it. “By definition” it is variable width character data. I think it’s fair to be pedantic vs a fairly dramatic oversimplification.
      - ElevenLathe 2y ago
        Less specific interfaces let you do less interesting things, but are more resilient. It's an engineering tradeoff. Purpose-built interfaces that fully expose and understand domain-level semantics are great in certain circumstances, but other times you want a certain minimum abstraction (IP packets and 'bags-of-bytes' POSIX file semantics are good examples) that can be used to build better ones. If the rollout of HTTP had required that all the IP routers on the internet be updated to account for it, we likely would not have it. Likewise, if we required that all the classic Unix text utilities like wc, sort, paste, etc. did meaningful things to JSON before we could standardize JSON, adoption would likely have suffered.
        rusk 2y ago
        The basic unix tools do account for variable width though. Variable chars are baked into most OS. When you use these commands the decode is implicit.
      - samatman 2y ago
        You can transport it to an architecture of different endianness without loss of information or metadata and a transformation at destination. There are important ways in which it is, in fact, "just bytes".
        rusk 2y ago
        Endianness etc is a feature of the encoding. Most JSON implementations I’ve used require the raw bytes to first be decoded as such.
        samatman 2y ago
        No, endianness is not a feature of UTF-8 encoding. There isn't a UTF-8LE and a UTF-8BE. That's because the codeunit is bytes. Forget "decoding", you have to parse JSON. But you don't have to figure out how it's encoded first. Because it's a byte format. You already know.
  - sambazi 2y ago
    encoding decides how bytes are interpreted
    - rusk 2y ago
      Yes. The encoded content is “just bytes” - once decoded it’s logically something else (var char data structured as json) that transcends bytes at the data level.
  - jshier 2y ago
    By spec all JSON must be UTF-8. Anyone adding encodings to application/json is, at best, redundant.
    - rusk 2y ago
      UTF-8 is variable width my friend
      - chubot 2y ago
        The fact that JSON is UTF-8 doesn't contradict the fact that it's bytes! That's a feature, not a bug. i.e. "exterior designs are layered" - https://www.oilshell.org/blog/2023/06/ysh-design.html#exterior-designs-are-layered https://www.oilshell.org/blog/2023/06/ysh-design.html#exteri... This is not a trivial point -- there are plenty of specs which are sloppy about text, try to abstract over code points, and result in non-working software. A main example is the shitshow with libc and Unicode - https://thephd.dev/cuneicode-and-the-future-of-text-in-c#standard-c https://thephd.dev/cuneicode-and-the-future-of-text-in-c#sta... It suffers from what I call "interior fictions", like wchar_t.
        rusk 2y ago
        Of course it’s all bytes. It’s all bytes. That doesn’t change the fact that you need to have some awareness of encoding before those bytes are fully sensible

delusional 2y ago

Unless you wanted to compress that json. If you insisted on the structure, you'd have to create a specific algorithm just for json. Because json is just bytes, you get to compress it with a generic byte compression algorithm.

rusk 2y ago
Is json just bytes tho? Doesn’t it allow for variable width encodings?
- ElevenLathe 2y ago
  No matter what encoding your JSON file is, gzip will output a compressed bag of bytes that, when unzipped, will result in the same file coming out the other end. This is true of movie codecs, Word 97 files, or anything, and none of the maintainers of those formats had to be consulted about this in order to make it work. That's what is meant by "thin waist" here.
  - rusk 2y ago
    I know, but it’s not “just bytes” as per parent comment. You cannot infer the length of the content without decoding it. “By definition” it is variable width character data. I think it’s fair to be pedantic vs a fairly dramatic oversimplification.
    - ElevenLathe 2y ago
      Less specific interfaces let you do less interesting things, but are more resilient. It's an engineering tradeoff. Purpose-built interfaces that fully expose and understand domain-level semantics are great in certain circumstances, but other times you want a certain minimum abstraction (IP packets and 'bags-of-bytes' POSIX file semantics are good examples) that can be used to build better ones. If the rollout of HTTP had required that all the IP routers on the internet be updated to account for it, we likely would not have it. Likewise, if we required that all the classic Unix text utilities like wc, sort, paste, etc. did meaningful things to JSON before we could standardize JSON, adoption would likely have suffered.
      - rusk 2y ago
        The basic unix tools do account for variable width though. Variable chars are baked into most OS. When you use these commands the decode is implicit.
    - samatman 2y ago
      You can transport it to an architecture of different endianness without loss of information or metadata and a transformation at destination. There are important ways in which it is, in fact, "just bytes".
      - rusk 2y ago
        Endianness etc is a feature of the encoding. Most JSON implementations I’ve used require the raw bytes to first be decoded as such.
        samatman 2y ago
        No, endianness is not a feature of UTF-8 encoding. There isn't a UTF-8LE and a UTF-8BE. That's because the codeunit is bytes. Forget "decoding", you have to parse JSON. But you don't have to figure out how it's encoded first. Because it's a byte format. You already know.
- sambazi 2y ago
  encoding decides how bytes are interpreted
  - rusk 2y ago
    Yes. The encoded content is “just bytes” - once decoded it’s logically something else (var char data structured as json) that transcends bytes at the data level.
- jshier 2y ago
  By spec all JSON must be UTF-8. Anyone adding encodings to application/json is, at best, redundant.
  - rusk 2y ago
    UTF-8 is variable width my friend
    - chubot 2y ago
      The fact that JSON is UTF-8 doesn't contradict the fact that it's bytes! That's a feature, not a bug. i.e. "exterior designs are layered" - https://www.oilshell.org/blog/2023/06/ysh-design.html#exterior-designs-are-layered https://www.oilshell.org/blog/2023/06/ysh-design.html#exteri... This is not a trivial point -- there are plenty of specs which are sloppy about text, try to abstract over code points, and result in non-working software. A main example is the shitshow with libc and Unicode - https://thephd.dev/cuneicode-and-the-future-of-text-in-c#standard-c https://thephd.dev/cuneicode-and-the-future-of-text-in-c#sta... It suffers from what I call "interior fictions", like wchar_t.
      - rusk 2y ago
        Of course it’s all bytes. It’s all bytes. That doesn’t change the fact that you need to have some awareness of encoding before those bytes are fully sensible