3 ms·

Some people will curse me, but I have used JSON-over-CSV a few times, it's a good compromise if your data is indeed mostly tabular. To be clear, I mean encodin

Some people will curse me, but I have used JSON-over-CSV a few times, it's a good compromise if your data is indeed mostly tabular. To be clear, I mean encoding values as JSON, but the overall object structure in CSV, so something like: id,name,cool 1,"John Doe",false 2,"Daisy \"Obliterator of Worlds\" Wilson", true

saghm 5y ago
This is super interesting. I'd imagine there's a pretty good reason that you picked this over something like an array of arrays of values or something like { "row1": [...], "row2": [...] }, but the only reason I can think of is if there was some of external constraint that made CSV necessary. Was it that simple, or was there something else?
- kroltan 5y ago
  Embedded system with limited bandwidth, the data was mostly simple values that did not really need escaping 99% of the time (mostly no unicode or control characters in strings, but still _possible_, and we needed the streamability of array-of-structs instead of struct-of-arrays. Why not some binary format, then? Well, it's easier to debug and share the data with non-technical people (just open in excel or equivalent) without having to run a conversion step.
deleted 5y ago
[deleted]
DougBTX 5y ago
I think this would be fine, as long as the CSV layer was still parsable using the RFC 4180, then you could still use a normal CSV parser to parse the CSV layer and a normal JSON parser to parse the JSON layer. My worry with your example is that it is neither format, so it will need custom serialisation and deserialisation logic as it is essentially a bran new format. https://datatracker.ietf.org/doc/html/rfc4180 https://datatracker.ietf.org/doc/html/rfc4180 If you’re looking for line-oriented JSON, another option would be ndjson: http://ndjson.org/ http://ndjson.org/
- kroltan 5y ago
  Yeah you're right, I'm not respecting the quote-quoting rule in this example, though IIRC I did in at least one of the actual uses. ndjson has the disadvantage of repeating column names for every record, which, granted, is basically fine if you are using compression, but sometimes for whatever reason you can't.
laumars 5y ago
You’d be better off using jsonlines than creating a bespoke format that is neither CSV nor JSON. https://jsonlines.org/examples/ https://jsonlines.org/examples/
- kroltan 5y ago
  True, I guess that could be a viable alternative!

saghm 5y ago

This is super interesting. I'd imagine there's a pretty good reason that you picked this over something like an array of arrays of values or something like { "row1": [...], "row2": [...] }, but the only reason I can think of is if there was some of external constraint that made CSV necessary. Was it that simple, or was there something else?

kroltan 5y ago
Embedded system with limited bandwidth, the data was mostly simple values that did not really need escaping 99% of the time (mostly no unicode or control characters in strings, but still _possible_, and we needed the streamability of array-of-structs instead of struct-of-arrays. Why not some binary format, then? Well, it's easier to debug and share the data with non-technical people (just open in excel or equivalent) without having to run a conversion step.

deleted 5y ago

[deleted]

DougBTX 5y ago

I think this would be fine, as long as the CSV layer was still parsable using the RFC 4180, then you could still use a normal CSV parser to parse the CSV layer and a normal JSON parser to parse the JSON layer. My worry with your example is that it is neither format, so it will need custom serialisation and deserialisation logic as it is essentially a bran new format. https://datatracker.ietf.org/doc/html/rfc4180 https://datatracker.ietf.org/doc/html/rfc4180 If you’re looking for line-oriented JSON, another option would be ndjson: http://ndjson.org/ http://ndjson.org/

kroltan 5y ago
Yeah you're right, I'm not respecting the quote-quoting rule in this example, though IIRC I did in at least one of the actual uses. ndjson has the disadvantage of repeating column names for every record, which, granted, is basically fine if you are using compression, but sometimes for whatever reason you can't.

laumars 5y ago

You’d be better off using jsonlines than creating a bespoke format that is neither CSV nor JSON. https://jsonlines.org/examples/ https://jsonlines.org/examples/

kroltan 5y ago
True, I guess that could be a viable alternative!