4 ms·

To your comment above – you can bodge around interop problems with JSON in ways that you cannot with some of these other technologies. I like to joke that I in

To your comment above – you can bodge around interop problems with JSON in ways that you cannot with some of these other technologies. I like to joke that I invented ndjson over a decade ago when I accidentally forgot to put things in an array before `json.dumps`, I just wasn't smart enough to call it a standard. But when you do end up with ndjson when you wanted an array of results, or vice versa, JSON makes it easy to munge things to where you need. Compare that to something like protobuf: it's not a self-synchronizing stream, so if you send someone multiple messages without framing them (prefix by length or delimited are popular approaches), they're going to decode a single message that doesn't make much sense on the other end. And they won't be able to fix it at all. So I guess JSON is New Jersey style design[1]. [1]: https://dreamsongs.com/RiseOfWorseIsBetter.html https://dreamsongs.com/RiseOfWorseIsBetter.html

kortex 5y ago
Well, you invented one of the best things since sliced bread! I love NDjson, being able to parse a sequence of {} objects as an array is just frankly more natural. A coworker got some absurd speedup going from some massive json array to ndjson. Honestly if json had as part of its spec line-delimited arrays, and accepting NaN, it'd be close to perfect. Oh and native ints, but that is JS's problem. Well, and a single, canonical spec. And a hard limit (however high) on nesting depth. And some other things. Ok, maybe it's far from perfect.
q3k 5y ago
> Compare that to something like protobuf: it's not a self-synchronizing stream, so if you send someone multiple messages without framing them (prefix by length or delimited are popular approaches), they're going to decode a single message that doesn't make much sense on the other end. And they won't be able to fix it at all. FWIW, this is a conscious design decision with Protobuf: it allows for easy upsert operations on serialized messages by appending another message with the updated field values. This is very useful for middleware that wants to either just add its own context to a message it doesn't even parse [1], or for middleware that might handle protobuf messages serialized with unknown fields. On the other hand, 'newline delimited protobuf' is much less useful day-to-day than ndjson, as gRPC provides message streaming, which solves the issue of wanting to stream small elements of a long response (which is the general usecase of ndjson from my experience). For on-disk storage of sequential protobufs (or any other data, really), you should be using something like riegeli [2], as it provides critical features like seek offsets, compression and corruption resiliency. [1] - eg. passing a Request message from some web server frontend, through request routers, logging, ACL and ratelimit systems up to the actual service handling the request. [2] - https://github.com/google/riegeli https://github.com/google/riegeli

kortex 5y ago

Well, you invented one of the best things since sliced bread! I love NDjson, being able to parse a sequence of {} objects as an array is just frankly more natural. A coworker got some absurd speedup going from some massive json array to ndjson. Honestly if json had as part of its spec line-delimited arrays, and accepting NaN, it'd be close to perfect. Oh and native ints, but that is JS's problem. Well, and a single, canonical spec. And a hard limit (however high) on nesting depth. And some other things. Ok, maybe it's far from perfect.

q3k 5y ago

> Compare that to something like protobuf: it's not a self-synchronizing stream, so if you send someone multiple messages without framing them (prefix by length or delimited are popular approaches), they're going to decode a single message that doesn't make much sense on the other end. And they won't be able to fix it at all. FWIW, this is a conscious design decision with Protobuf: it allows for easy upsert operations on serialized messages by appending another message with the updated field values. This is very useful for middleware that wants to either just add its own context to a message it doesn't even parse [1], or for middleware that might handle protobuf messages serialized with unknown fields. On the other hand, 'newline delimited protobuf' is much less useful day-to-day than ndjson, as gRPC provides message streaming, which solves the issue of wanting to stream small elements of a long response (which is the general usecase of ndjson from my experience). For on-disk storage of sequential protobufs (or any other data, really), you should be using something like riegeli [2], as it provides critical features like seek offsets, compression and corruption resiliency. [1] - eg. passing a Request message from some web server frontend, through request routers, logging, ACL and ratelimit systems up to the actual service handling the request. [2] - https://github.com/google/riegeli https://github.com/google/riegeli