6 ms·
Comma Separated JSON
- tlrobinson 10y agoSo literally just JSON without the "[" and "]," delimiting rows, and another "[" at the beginning and "]" at the end of the file? There are already streaming JSON parsers/serializers, why not just use (or write) one of those instead of inventing yet another file format? There's also http://ndjson.org/ http://ndjson.org/ and http://jsonlines.org/ http://jsonlines.org/
- spullara 10y agoTo add to this, APIs like Twitter's streaming API, have been sending a JSON blob per line for forever. No need to break up the line any further IMHO.
- boronine 10y agoIt's useful for situations where CSV output is a requirement. Actually, this seems like a great candidate for a CSV standard since JSON is so widely implemented.
- rspeer 10y agoBut JSON won't match the quoting rules of any existing CSV parser, so it won't actually work in a situation where CSV is a requirement.
- simlevesque 10y agoIt's not just JSON without [ and ], it's a bunch of JSONs represented as a table. The first line sets the property's key and every other line are the values. It is a representation that needs a parser to be read but that's way faster for a row based database to generate it than to output full json. Plus it's smaller than minified json.
- tlrobinson 10y agoYou can represent a table using JSON arrays, and I doubt CJS is "way faster" or smaller than gzipped JSON. There is nothing inherently preventing you from writing an almost identically performing streaming JSON parser or serializer.
- proteusguy 10y agoThis is likely content coming down from an HTTP API so likely already gzipped. Since you can gzip both CSJ and JSON I think that's not really relevant. With this proposed format you only need to parse the object definition once and you also get the ordering fixed. From then on you need only validate each field in each line as being legal and you can build your object quite fast. This format saves probably half the network bandwidth, half or more in memory requirements, and more than half the cpu complexity in parsing and getting your final objects instantiated. That's pretty good. Remember the goal isn't to replace JSON. It's to replace CSV. JSON and CSV have different goals. This is simply a JSON structure built to meet the same goals as CSV. It seems to do that quite nicely and can use existing JSON parsers internally to do it. Pretty nifty.
- dalke 10y agoThe spec doesn't specify what the first line contains. It only says "Semantically a CSJ file is an array of JSON objects which share a common set of keys." You used your knowledge of how these representations usually work to infer that the first line contains a list of keys for the remaining lines. Some obvious questions for a table spec include: 1) must those keys be strings? 2) Are duplicate keys allowed? 3) Can different rows have a different number of columns? Or must the writer generate null fields? FWIW, I've used JSON Lines for my own work. It's very easy to read a line and (in Python), json.loads(line). How should I parse this "JSON without [ and ]"? There's not a built-in function for it. Either I have to write my own JSON tokenizer, or I have to terminate the string with '[' and ']' and parse that. Both are much more complicated than using JSON Lines? You'll notice they only gave output generation timing numbers, not input parsing. If a JSON-based table format is important, why not layer those same table semantics on top of JSON Lines, rather than make a format which is harder to read?
- valarauca1 10y agoSo a sed script?
- bevacqua 10y agoOr: just use \n and also be able to read individual rows. Many programs in the wild do this.
- corysama 10y agoI really wish ASCII delimited text didn't die off. https://ronaldduncan.wordpress.com/2009/10/31/text-file-formats-ascii-delimited-text-not-csv-or-tab-delimited-text/ https://ronaldduncan.wordpress.com/2009/10/31/text-file-form... https://news.ycombinator.com/item?id=7474600 https://news.ycombinator.com/item?id=7474600 But, it suffered from not having designated keys on common keyboards, therefore it wasn't obviously necessary for plain text editors to support. Meanwhile, everyone uses plain text editors for everything --even if it means jumping through hoops full of exceptions and corner cases to pretend they can handle more than plain text.
- jameshart 10y agoNewline-delimited JSON? I've used it informally, but I've never had the nerve to suggest it as an actual documented interchange format. It makes a lot of sense for things which CSV is often used for - dumping out big tables of data into files for shipping over FTP or similar - or even for logfiles with structured data entries. The tricky thing is having to restrict the JSON on each line to not contain any newline characters. Even though all legitimate JSON can be minified into a form which has the same semantics but contains no newlines, "JSON with no newlines in it" isn't a well defined standard.
- bevacqua 10y ago"JSON with no commas in it" isn't a thing either. That's not what's being discussed here.
- mixmastamyk 10y agoHave needed something like this a few times, would be great if it was standardized.
- thewisenerd 10y agoinb4: https://xkcd.com/927/ https://xkcd.com/927/ the reason json is popular is because it's widely used and is easily parsed (without having to call any extra libraries). standardizing this would mean accepting upon a complete working alternate that'd work everywhere just akin to json (plus the added perks), which will take a lot of (dis)agreement, especially given that there are already a dozen other standards trying to replace json.
- skybrian 10y agoI think I'd rather stick to the format supported by Go and the 'jq' command, where each line is actually JSON. But if it became popular, this format seems easy to support.
- guelo 10y agoI would call this "JSON encoded CSV" since it appears to be a strict subset of CSV, not JSON.
- dalke 10y agoA cell can embed JSON objects and arrays, so it isn't a strict subset.
- dclowd9901 10y agoI appreciate the problem this is trying to solve, but it could be more easily remedied by a JSON stream processor than reinventing the wheel of data marshalling (yet again). JSON is so simplistic in structure, it's trivial to know when you've met the conditions of a completed segment of structure. There's no reason why a stream parser couldn't handle this well.
- proteusguy 10y agoHow about twice the bandwidth, twice the memory, and at least twice the CPU complexity in parson full JSON objects than the proposed CSJ format?
- mythz 10y agoI've been using my custom JSV Format for years: https://github.com/ServiceStack/ServiceStack.Text/wiki/JSV-Format https://github.com/ServiceStack/ServiceStack.Text/wiki/JSV-F... Which is just JSON with CSV-escaping which is ends up being faster, more concise and more human-friendly since quotes are only necessary when using special delimiter chars. It's just not as interoperable or ubiquitous as JSON so only an option when you control both ends.
- smegel 10y agoSo you save some space on top level keys which are stored once in the header...but you waste space by having to flatten data structures into a table to start with, with key values being repeated over lines, the more deeply nested, the worse it gets. Stick with JSON document per line.
- thewisenerd 10y agois there necessarily a need to repeat key values over lines? why not have all possible object definitions in the header and data as a single huge array?
- athenot 10y agoYou're describing the type-length-value format, used in some places in digital video or in Open Sound Control. Basically you have a TYPE (fixed width, eg. 16 bits), a LENGTH (again fixed) and a VALUE which is simply a stream of bytes that goes on for LENGTH bytes and is interpreted based on what TYPE it is. You can nest your fields and get very creative in quite a compact way. One could represent a JSON document with TLV by adding an associative map between the field names and their type numbers, then stream away. I always loved the simplicity of TLV. Though nowadays, I'm not sure this has much of an advantage over gzipped JSON.
- ubertaco 10y agoOff-topic: Howdy, Mr. Thénot! Crazy reading the comments and seeing your username here. Hope life is goin' well for ya with the HealthNinja stuff -- haven't seen you since I left Wellcentive (around the same time as mbonnette, if that gives you an idea of who I am).
- proteusguy 10y agoBecause to parse it legally you'd have to store the entire file in memory. That's not actually streaming. The CSJ format puts the object schema and order up front and just applies it to each line afterwards. Stream away...
- 10y ago
- jameshart 10y agoSo this is something that looks quite a lot like CSV, in some cases produces identical output to CSV, but which isn't compatible with CSV.
- deleted 10y ago[deleted]
- dalke 10y agoTo be fair, most CSV isn't compatible with CSV. Especially with the CSV in RFC 4180, which among other things requires a CRLF.
- gregmac 10y agoThis is actually a very elegant way of formalizing CSV. For most apps, if you were to simply output "CSV files" using this encoding, really no one consuming them would know any different -- except you could actually point them at a spec being used. I've been in arguments with customers before where they complained we were quoting strings, and CSV files shouldn't have quotes, and it would have been nice to say "We're using CSJ, please just parse that way". I think the name is perhaps misleading though, as it sounds more like it's a variant of JSON than a formalization of CSV. Perhaps "JSON flavored CSV" would be more appropriate?
- LukeShu 10y agoI mean, you could always point them to RFC 4180.
- gregmac 10y agoAh, well, look at that. Thankfully I haven't had to deal with CSV files for several years, though I wish I knew about that RFC at the time. Thanks.
- gregmac 10y agoOkay talk about ironic. Yesterday in our ops meeting, it came up that some tool another department had built was exporting csv "with extra quotes that were causing problems" (I've had no involvement in any of this). My ears perked up remembering this comment and I was able to point out this RFC, so hopefully it will save them some time in arguing about csv formatting. Thanks!
- astrodust 10y ago"Comma-Separated JSON" implies objects, "object-notation" is integral to the definition. What about "Comma-Separated JSON Values"? CSJV!
- gweinberg 10y agodoesn't look very JSONy without any curly braces.
- henvic 10y agoElasticSearch bulk API deals with streaming by using line breaks of JSON docs instead of an array. If you are interested in using this Comma Separated JSON, maybe you want to try this simpler approach before (though the comma separated version seems to be lighter, if I want really to reduce weight I'd consider protobuf, probably). Reference: https://www.elastic.co/guide/en/elasticsearch/reference/current/docs-bulk.html https://www.elastic.co/guide/en/elasticsearch/reference/curr...
- querulous 10y agono one uses csv by choice, they use it because they have to work with tools (excel) that can only produce/consume csv given you don't have that constraint there are much better options (json lines, avro, parquet, protobufs...)
- vortico 10y agoI use TSV often when I want to easily dump data from a C program to gnuplot or similar. It's more of an ad hoc file format then a serious one though.
- th0mat 10y agoJSON enhanced CSV might be a more suitable name. After all it is valid CSV and invalid JSON.
- ptman 10y agoI thought MIME types had to go through an application process or start with x-
- ctulek 10y agoIf you are using a proper CSV library you don't have any of the problems this format is trying to solve. The CSV format does not have any issues mentioned, it is either a buggy CSV library or some lazzy developer who thinks CSV is just fields separated with comma writing 2 line of code instead of using a proper CSV library.
- danso 10y agoWhich CSV library is able to infer that a value is a number instead of a string?
- ctulek 10y agoCSV for sure requires a convention to be shared between the producer and the consumer of the data. Once you have that I don't think you will need to deal with this problem. To be honest, what I didn't like is the tone of the article. It is just that this statement is wrong: "But the problem is that CSV isn't really a well defined file format with a well defined syntax." There is RFC 4180.