5 ms·
JSON serialized without extra white space with one line per record is superior to CSV. If you want CSV-ish, enforce an array of strings for each record. Or go
by koolba 2y ago
JSON serialized without extra white space with one line per record is superior to CSV.
If you want CSV-ish, enforce an array of strings for each record. Or go further with actual objects and non-string types.
You can even jump to an arbitrary point and then seek till you see an actual new line as it’s always a record boundary.
It’s not that CSV is an invalid format. It’s that libraries and tools to parse CSV tend to suck. Whereas JSON is the lingua franca of data.
- juliansimioni 2y agoWhat happens when you need to encode the newline character in your data? That makes splitting _either_ CSV or LDJSON files difficult.
- mananaysiempre 2y agoWhen you need to encode the newline character in your data, you say \n in the JSON. Unlike (the RFC dialect of) CSV, JSON has an escape sequence denoting a newline and in fact requires its use. The only reason to introduce newlines into JSON data is prettyprinting.
- koolba 2y agoThe new line character in a JSON string would always be \n. The new line in the record itself as whitespace would not be acceptable as that breaks the one line record contract. Remember that this does not allow arbitrary representation of serialized JSON data. But it allows for any and all JSON data as you can always roundtrip valid JSON to a compact one line representation without extra whitespace.
- nmz 2y agoIt's tricky, but simple enough, RFC states that " must be used, inserting a " is done with "". This makes knowing what a record is difficult, since you must keep a variable that keeps the entire string. How do you do this simply? you read each line, and if there's an uneven number of ", then you have an incomplete record and you will keep all lines until there is an odd number of ". after having the string, parsing the fields correctly is harder but you can do it in regex or PEGs or a disgusting state machine.
- afiori 2y agoActually even whitespace-separated json would be a valid format and if you forbid json documents to be a single integer or float then even just concatenating json gives a valid format as JSON is a prefix free language. That is[0] if a string s is a valid JSON then there is no substring s[0..i] for i < n that is a valid json. So you could just consume as many bytes you need to produce a json and then start a new one when that one is complete. To handle malformed data you just need to throw out the partial data on syntax error and start from the following byte (and likely throw away data a few more times if the error was in the middle of a document) That is [][]""[][]""[] is unambiguos to parse[1] [0] again assuming that we restrict ourselves to string, null, boolean, array and objects at the root [1] still this is not a good format as a single missing " can destroy the entire document.
- boogheta 2y ago« a single missing " can destroy the entire document » This is basically true for any data format, so really worse argument ever...
- afiori 2y agoIn jsonl a modified chunk will lose you at most the removed lines and the two adjacent ones (unless the noise is randomly valid json), in particular a single byte edit can destry at most 2 lines. utf-8 is also similarly self-correcting and so is html and many media formats. My point was that in my made-up concatenated json format []"""[][][][][][][][][][][]"""[] and []""[][][][][][][][][][][]""[] are both valid but have differ only for 2 bytes but have entirely different structures. Also it is a made-up format nobody uses (if somebody were to want this they would likely disallow strings at the root level).
- benwilber0 2y agoJSON is a textual encoding no different than CSV. It's just that people tend to use specialized tools for encoding and decoding it instead of like ",".join(row) and row.split(",") I have seen people try to build up JSON strings like that too, and then you have all the same problems. So there is no problem with CSV except that maybe it's too deceptively simple. We also see people trying to build things like URLs and query strings without using a proper library.
- int_19h 2y agoThe problem with CSV is that there's no clear standard, so even if you do reach for a library to parse it, that doesn't ensure compatibility.
- rcbdev 2y agoSame for JSON though. What Python considers a valid JSON might not be that if you ask a Java library.
- hajile 2y agoJSON has a clearly-defined standards: ISO/IEC 21778:2017, IETF RFC 7159, and ECMA-404. Additionally, Crockford has had a spec available on json.org since it's creation in 2001. Do you have any examples of Python, Java, or any of the other Tiobe top 40 languages breaking the JSON spec in their standard library? In contrast, for the few of those that have CSV libraries, how many of those libraries will simply fail to parse a large number of the .csv variations out there?
- whizzter 2y agoNot to mention that stuff like Excel loves to export CSV files in "locale specific ways". Sometimes commas to delimiter, sometimes semicolons, floating point values might have dots or commas to separate fraction digits. Not to mention text encodings, Ascii, western european character sets, or maybe utf-8 or whatever... It's a bloody mess.
- packetlost 2y agoEh, it really isn't. The format does not lend itself to tabular data, instead the most natural way of representing data involves duplicating the keys N times for each record.
- koolba 2y agoYou can easily represent it as an array: [“foo”,”bar”,123] That’s as tabular as CSV but you now have optional types. You can even have lists of lists. Lists of objects. Lists of lists of objects…
- packetlost 2y agoYou're missing my point: basically nothing spits out data in that format because it's not ergonomic to do so. JSON is designed to represent object hierarchies, not tabular data.
- koolba 2y agoJSON was designed to represent any data. There's plenty of systems that spit out data in exact that format because it's the natural way to represent tabular data using JSON serialization. And clearly if you're the one building the system you can choose to use it.
- hajile 2y agoCSV is lists of lists of fixed length. JSON is lists of lists of any length and groups of key/value pairs (basically lisp S-expressions with lots of unnecessary syntax). This makes it a superset of CSV's capabilities. JSON fundamentally IS made to represent tabular data, but it's made to represent key-value groups too. Why make it able to represent tabular data if that's not an intended use?
- meepmorp 2y ago> CSV is lists of lists of fixed length. I'd definitely put that in my list of falsehoods programmers believe about CSV files.
- derriz 2y ago> It’s that libraries and tools to parse CSV tend to suck. Whereas JSON is the lingua franca of data. This isn't the case. An incredible amount of effort and ingenuity has gone into CSV parsing because of its ubiquity. Despite the lack of any sort of specification, it's easily the most widely supported data format in existence in terms of tools and language support.
- hajile 2y agoCan you point me to a language with any significant number of users that does NOT have a JSON library? I went looking at some of the more niche languages like Prolog, COBOL, RPG, APL, Eiffel, Maple, MATLAB, tcl, and a few others. All of these and more had JSON libraries (most had one baked into the standard library). The exceptions I found (though I didn't look too far) were: Bash (use jq with it), J (an APL variant), Scratch (not exposed to users, but scratch code itself is encoded in JSON), and Forth (I could find implementations, but it's very hard to pin down forth dialects).
- derriz 2y agoI made no claim about JSON libraries. I contested the claim that "CSV libraries and tools suck". They do not.
- jstanley 2y agoYou claimed that CSV is "easily the most widely supported data format in existence in terms of tools and language support", which is a claim that CSV is better supported than JSON, which is a claim that JSON support is lacking.
- shawabawa3 2y agoCan you import .jsonl files into Google sheets or excel natively?
- freehorse 2y ago
- klysm 2y agoYou serialize the keys on every row which is a bit inefficient but it’s a text format anyway
- zeroimpl 2y agoSpace-wise, as long as you compress it, it's not going to make any difference. I suspect a JSON parser is a bit slower than a CSV parser, but the slight extra CPU usage is probably worth the benefits that come with JSON.
- nmz 2y agoThis is simply not true, parsing json v csv is a difference of thousands of lines.
- magicalhippo 2y ago> JSON [...] with one line per record Couple of standards that I know of that does this, primarily intended for logging: https://jsonlines.org/ https://jsonlines.org/ https://clef-json.org/ https://clef-json.org/ Really easy to work with in my experience. Sure some space is usually wasted on keys but compression takes care of that.
- kec 2y agoUntil you have a large amount of data & need either random access or to work on multiple full columns at once. Duplicated keys names mean it's very easy for data in jsonlines format to be orders of magnitude larger than the same data as CSV, which is incredibly annoying if your processing for it isn't amenable to streaming.