3 ms·

Why not use a saner protocol than JSON, e.g. CBOR?

by js8 1y ago

totallymike 1y ago
Is CBOR as popularly supported as JSON? Also, to answer your question with a guess, I would suppose it’s because they wanted to use JSON and they wrote the feature.
- thechao 1y ago
  I do a lot of very low level programming with awful performance-maintenance trade-offs. Here's a great trick for a "binary" JSON: remove all of the extra whitespace, normalize your numbers, and the LZ4 the resulting string. UTF-8 is already a great wire format. I've never found a "binary JSON" that's significantly better than this; I mean you can beat it, but you need awkward encodings (prefix indices & other weird shit). You end up burning nearly-byte for any particularly clever integer encoding. Most data structures are just nested arrays of integers. If you need an integer keyed OBJECT you're SOL, but I just play fiddly games with astral plane UTF-8 characters. (Yeah yeah yeah ad hoc encodings are nasty news.) If you've got a BUTT LOAD of data just fire up a compressing SQLite DB like a normal human.
  - js8 1y ago
    If you're interested in performance, what about all the number conversion (to decimals, presumably) that is incurred with JSON?
    - thechao 1y ago
      If I'm interested in performance I'll build my data out of offset handles and lay everything down into a block and mmap() it around. That's parsing free, up to an htons() — but that's only a worst case scenario. Everything else is about not inventing something custom & being able to use easily vendored high-trust 3rd party tools. (In this case: a JSON library, LZ4, and/or SQLite.)
  - deleted 1y ago
    [deleted]
tonyarkles 1y ago
Do you have a CBOR implementation that you like? Ideally one with decent schema support? I was looking into CBOR as a replacement for Protobufs for an embedded system I work on and it's got a lot going for it but every implementation I looked at seemed to support a very different subset of the schema spec and it was brutal to try to find a pair of libraries (C for the embedded side, C++ for the host side) that could actually share a set of schema files.

Why not use a saner protocol than JSON, e.g. CBOR?

by js8 1y ago

Why not use a saner protocol than JSON, e.g. CBOR?

Is CBOR as popularly supported as JSON? Also, to answer your question with a guess, I would suppose it’s because they wanted to use JSON and they wrote the feature.

thechao 1y ago
I do a lot of very low level programming with awful performance-maintenance trade-offs. Here's a great trick for a "binary" JSON: remove all of the extra whitespace, normalize your numbers, and the LZ4 the resulting string. UTF-8 is already a great wire format. I've never found a "binary JSON" that's significantly better than this; I mean you can beat it, but you need awkward encodings (prefix indices & other weird shit). You end up burning nearly-byte for any particularly clever integer encoding. Most data structures are just nested arrays of integers. If you need an integer keyed OBJECT you're SOL, but I just play fiddly games with astral plane UTF-8 characters. (Yeah yeah yeah ad hoc encodings are nasty news.) If you've got a BUTT LOAD of data just fire up a compressing SQLite DB like a normal human.
- js8 1y ago
  If you're interested in performance, what about all the number conversion (to decimals, presumably) that is incurred with JSON?
  - thechao 1y ago
    If I'm interested in performance I'll build my data out of offset handles and lay everything down into a block and mmap() it around. That's parsing free, up to an htons() — but that's only a worst case scenario. Everything else is about not inventing something custom & being able to use easily vendored high-trust 3rd party tools. (In this case: a JSON library, LZ4, and/or SQLite.)
- deleted 1y ago
  [deleted]

tonyarkles 1y ago

Do you have a CBOR implementation that you like? Ideally one with decent schema support? I was looking into CBOR as a replacement for Protobufs for an embedded system I work on and it's got a lot going for it but every implementation I looked at seemed to support a very different subset of the schema spec and it was brutal to try to find a pair of libraries (C for the embedded side, C++ for the host side) that could actually share a set of schema files.