5 ms·

There was a Varlink talk[0] a few days ago at All Systems Go, in that talk, Lennart mentioned that JSON is unfortunate (primarily due to no 64 bit ints) but it

by deivid 2y ago

There was a Varlink talk[0] a few days ago at All Systems Go, in that talk, Lennart mentioned that JSON is unfortunate (primarily due to no 64 bit ints) but it has the surprising benefit of being able to understand the bus messages when using `strace` for debugging. [0]: https://www.youtube.com/watch?v=emaVH2oWs2Y&list=PLWYdJViL9EipIImmvuoGFAeS-lKeHH2DD&index=34 https://www.youtube.com/watch?v=emaVH2oWs2Y&list=PLWYdJViL9E...

gmokki 2y ago
I do not understand where the 64bit integers not working with JSON comes from. JSON the format had no limit on integer size. And all Java JSON libraries I know can handle arbitrary prevsion integers (BigInt) and 32/64bit int/long types when serializing and deserializing. Quick googling shows that also JavaScript has proper support for 64bit integers with their BigInt type, and it can be used to deserialize incoming data correctly with the default parser, albeit requiring a bit more work to annotate the fields. Personally I often explicitly make sure that the integers I return in trust environments as identifiers in REST APIs are by default longer than 52bits so that buggy parser libraries are caught early.
- GrayShade 2y ago
  Qt refused for almost a decade to support deserializing 64-bit integers from JSON because of compatibility concerns.
  - Spivak 2y ago
    jq couldn't handle them until relatively recently. This isn't a few bad parsers. You can't assume a json parser will handle bigints correctly and when you're angling to be low-level plumbing and work with every language and software that hasn't been recompiled in years you have to be pretty conservative in what you send.
    - capitainenemo 2y ago
      Yeah, and it isn't like jq is being incorrect in this, the JSON RFCs recommend not using values that can't be represented as a 64 bit float (pasted links to a few RFCs in another response). So if you want to represent a large number safely, better to put it in a string, where special handling can be done after the initial JSON parse without loss of information.
      - GrayShade 2y ago
        No, it doesn't. It's not a SHOULD. And why should Qt be concerned with what jq can or can't do? If you're writing and reading back from Qt, the limitations of other implementations don't apply.
- capitainenemo 2y ago
  The number type in JSON is 64 bit float, limiting integers without loss of precision to 2⁵³-1. BigInt is a new concept and not technically supported. So whether it works in your random library of choice is probably a potshoot. "Use within JSON: Using JSON.stringify() with any BigInt value will raise a TypeError, as BigInt values aren't serialized in JSON by default. " https://developer.mozilla.org/en-US/docs/Web/JavaScript/Reference/Global_Objects/BigInt#use_within_json https://developer.mozilla.org/en-US/docs/Web/JavaScript/Refe...
  - IshKebab 2y ago
    That's not true. The number type in JSON is an arbitrary sized number. You're thinking of JavaScript which is not the same thing.
    - capitainenemo 2y ago
      https://datatracker.ietf.org/doc/html/rfc7493#section-2.2 https://datatracker.ietf.org/doc/html/rfc7493#section-2.2 No. (which expands on the slightly fluffier text in https://datatracker.ietf.org/doc/html/rfc7159#section-6 https://datatracker.ietf.org/doc/html/rfc7159#section-6 which broadly says the same thing... and https://datatracker.ietf.org/doc/html/rfc8259#section-6 https://datatracker.ietf.org/doc/html/rfc8259#section-6) If you look at a parser matrix, flouting this rule is asking for trouble. Which is why the recommendations on MDN for bigint recommend an explicit parsing from a string as needed. (I'm genuinely curious why this is being downvoted. Does someone out there feel so strongly that because an arbitrary string of characters technically can be parsed as any infinitely large number, that it's wise to ignore RFCs and dozens of existing implementations when making use of JSON as a common interchange format?)
      - skissane 2y ago
        You are mixing up two different standards here, JSON and I-JSON. I-JSON is a subset of JSON with stricter requirements. JSON recommends (for interoperability) that numbers be limited to those representable as IEEE 64-bit binary floats-but does not require that. A document which ignores that recommendation is still officially valid JSON. By contrast, I-JSON, as a subset of JSON, upgrades that recommendation to a requirement, so documents which ignore it are not valid I-JSON
- theamk 2y ago
  If you fully work within a trusted environment, why bother with JSON? Use your favorite binary serialization with codegen. The whole point of JSON is almost every programming language can read and write it - and if you want this to be the case, stringify anything unusual, like large integers.
anotherhue 2y ago
With this logic (oft repeated) we should be sending TCP as JSON. { sourcePort: 443, destinationPort: 12345,...} Debugability is important but the answer is to build debugging tools, not to lobotomise the protocol for the vanishingly tiny fraction of packets that are ultimately subject to debugging.
- brookst 2y ago
  I love your TCP over JSON idea. Maybe we can implement layer 2 that way as well, with CSMA/CD: {startTime:2024-09-29T13:56:21.06551, duration:00:00:00.0024, packet: …}
  - seba_dos1 2y ago
    You forgot quotes. JSON has no date or time types :)
djbusby 2y ago
Cool, readable messages in strace but still some odd binary log format?
- criticalfault 2y ago
  He explained it here https://news.ycombinator.com/item?id=41694711 https://news.ycombinator.com/item?id=41694711
quotemstr 2y ago
Oh my God. I'm genuinely struggling to avoid using an endless stream of profanity here. If our problem is that our observability tools suck, the solution is to improve these tools, not mutilate our IPC mechanism to accommodate the limitations of these tools. Christ almighty, this is a terrible proposal.
- zbentley 2y ago
  The observability tools in question (strace) follow the UNIX tradition of displaying and processing data as text by default. I don’t think that means that they suck. I’ll go even stronger than that: IPC and RPC should prefer plaintext-representable forms by default and only abandon them for binary once the real world costs of the textually-representable protocol are found to be unacceptable and unmitigateable. The benefit of being able to use pre-existing introspection tools that were not designed with your protocol in mind—-and use those tools without extensive configuration—-is huge. I think the existence of bad textual formats (e.g. JSON) and the presence of largely-orthogonal-to-binaryness useful affordances in popular binary formats (e.g. schema validation and strong typing in Protobuf) muddies the underlying truth: textual-representability-by-default is rarely costly and often a huge boon to protocol implementors and protocol debuggers.

by deivid 2y ago

gmokki 2y ago

I do not understand where the 64bit integers not working with JSON comes from. JSON the format had no limit on integer size. And all Java JSON libraries I know can handle arbitrary prevsion integers (BigInt) and 32/64bit int/long types when serializing and deserializing. Quick googling shows that also JavaScript has proper support for 64bit integers with their BigInt type, and it can be used to deserialize incoming data correctly with the default parser, albeit requiring a bit more work to annotate the fields. Personally I often explicitly make sure that the integers I return in trust environments as identifiers in REST APIs are by default longer than 52bits so that buggy parser libraries are caught early.

GrayShade 2y ago
Qt refused for almost a decade to support deserializing 64-bit integers from JSON because of compatibility concerns.
- Spivak 2y ago
  jq couldn't handle them until relatively recently. This isn't a few bad parsers. You can't assume a json parser will handle bigints correctly and when you're angling to be low-level plumbing and work with every language and software that hasn't been recompiled in years you have to be pretty conservative in what you send.
  - capitainenemo 2y ago
    Yeah, and it isn't like jq is being incorrect in this, the JSON RFCs recommend not using values that can't be represented as a 64 bit float (pasted links to a few RFCs in another response). So if you want to represent a large number safely, better to put it in a string, where special handling can be done after the initial JSON parse without loss of information.
    - GrayShade 2y ago
      No, it doesn't. It's not a SHOULD. And why should Qt be concerned with what jq can or can't do? If you're writing and reading back from Qt, the limitations of other implementations don't apply.
capitainenemo 2y ago
The number type in JSON is 64 bit float, limiting integers without loss of precision to 2⁵³-1. BigInt is a new concept and not technically supported. So whether it works in your random library of choice is probably a potshoot. "Use within JSON: Using JSON.stringify() with any BigInt value will raise a TypeError, as BigInt values aren't serialized in JSON by default. " https://developer.mozilla.org/en-US/docs/Web/JavaScript/Reference/Global_Objects/BigInt#use_within_json https://developer.mozilla.org/en-US/docs/Web/JavaScript/Refe...
- IshKebab 2y ago
  That's not true. The number type in JSON is an arbitrary sized number. You're thinking of JavaScript which is not the same thing.
  - capitainenemo 2y ago
    https://datatracker.ietf.org/doc/html/rfc7493#section-2.2 https://datatracker.ietf.org/doc/html/rfc7493#section-2.2 No. (which expands on the slightly fluffier text in https://datatracker.ietf.org/doc/html/rfc7159#section-6 https://datatracker.ietf.org/doc/html/rfc7159#section-6 which broadly says the same thing... and https://datatracker.ietf.org/doc/html/rfc8259#section-6 https://datatracker.ietf.org/doc/html/rfc8259#section-6) If you look at a parser matrix, flouting this rule is asking for trouble. Which is why the recommendations on MDN for bigint recommend an explicit parsing from a string as needed. (I'm genuinely curious why this is being downvoted. Does someone out there feel so strongly that because an arbitrary string of characters technically can be parsed as any infinitely large number, that it's wise to ignore RFCs and dozens of existing implementations when making use of JSON as a common interchange format?)
    - skissane 2y ago
      You are mixing up two different standards here, JSON and I-JSON. I-JSON is a subset of JSON with stricter requirements. JSON recommends (for interoperability) that numbers be limited to those representable as IEEE 64-bit binary floats-but does not require that. A document which ignores that recommendation is still officially valid JSON. By contrast, I-JSON, as a subset of JSON, upgrades that recommendation to a requirement, so documents which ignore it are not valid I-JSON
theamk 2y ago
If you fully work within a trusted environment, why bother with JSON? Use your favorite binary serialization with codegen. The whole point of JSON is almost every programming language can read and write it - and if you want this to be the case, stringify anything unusual, like large integers.

anotherhue 2y ago

With this logic (oft repeated) we should be sending TCP as JSON. { sourcePort: 443, destinationPort: 12345,...} Debugability is important but the answer is to build debugging tools, not to lobotomise the protocol for the vanishingly tiny fraction of packets that are ultimately subject to debugging.

brookst 2y ago
I love your TCP over JSON idea. Maybe we can implement layer 2 that way as well, with CSMA/CD: {startTime:2024-09-29T13:56:21.06551, duration:00:00:00.0024, packet: …}
- seba_dos1 2y ago
  You forgot quotes. JSON has no date or time types :)

djbusby 2y ago

Cool, readable messages in strace but still some odd binary log format?

criticalfault 2y ago
He explained it here https://news.ycombinator.com/item?id=41694711 https://news.ycombinator.com/item?id=41694711

quotemstr 2y ago

Oh my God. I'm genuinely struggling to avoid using an endless stream of profanity here. If our problem is that our observability tools suck, the solution is to improve these tools, not mutilate our IPC mechanism to accommodate the limitations of these tools. Christ almighty, this is a terrible proposal.

zbentley 2y ago
The observability tools in question (strace) follow the UNIX tradition of displaying and processing data as text by default. I don’t think that means that they suck. I’ll go even stronger than that: IPC and RPC should prefer plaintext-representable forms by default and only abandon them for binary once the real world costs of the textually-representable protocol are found to be unacceptable and unmitigateable. The benefit of being able to use pre-existing introspection tools that were not designed with your protocol in mind—-and use those tools without extensive configuration—-is huge. I think the existence of bad textual formats (e.g. JSON) and the presence of largely-orthogonal-to-binaryness useful affordances in popular binary formats (e.g. schema validation and strong typing in Protobuf) muddies the underlying truth: textual-representability-by-default is rarely costly and often a huge boon to protocol implementors and protocol debuggers.