7 ms·
Documentation for the JSON Lines text file format
- larkinrichards 3y agoIsn't this just ndjson? https://github.com/ndjson/ndjson-spec https://github.com/ndjson/ndjson-spec
- rickette 3y agoPretty much, discussion: https://github.com/ndjson/ndjson.github.io/issues/1 https://github.com/ndjson/ndjson.github.io/issues/1 Also when you're using this with GeoJSON there's https://stevage.github.io/ndgeojson/ https://stevage.github.io/ndgeojson/ which has an actual RFC (https://datatracker.ietf.org/doc/html/rfc8142 https://datatracker.ietf.org/doc/html/rfc8142)
- zerocrates 3y agoWhich itself is based on an earlier RFC for not-specifically-Geo-JSON, RFC 7464. Both do things a little different than the others: they use the "record separator" character at the start of each line and actually split on that separator when parsing. The GeoJSON one pretty much seems to exist just to hang an "application/geo+json-seq" media type registration off of. Part of me wants to say this really should have been more of a "all json subtypes are also json-seq subtypes" situation but maybe that's not really feasible with the standards/registration processes.
- Towaway69 3y agoYes - > Two terms for equivalent formats of line-delimited JSON are: > Newline delimited (NDJSON)[4] - The old name was Line delimited JSON (LDJSON).[5] > JSON lines (JSONL)[6] https://en.wikipedia.org/wiki/JSON_streaming https://en.wikipedia.org/wiki/JSON_streaming
- o11c 3y agoIt's strictly worse, actually. ndjson specifies sane newline handling, since it works with terminators. jsonlines works with separators and thus fails to detect truncated values. As a result, it can silently produce incorrect numeric values.
- exyi 3y agoThis is only an issue with plain numbers, however. If you'd have the number in an object or array, you'd detect the truncation just as well. Since using jsonl for a plain list of numbers is... overkill, I'd say it's not an issue. On the other hand, requiring line terminators in the standard would inevitably lead to incompatibility issues. Most software would accept unterminated files, because text libraries do; and so some files will not be terminated. Some applications do not line-terminate files even on Linux (hello VSCode), and it would be even more problematic on Windows
- basil-rash 3y agoVSCode has the ‘files.insertFinalNewline‘ setting to configure this.
- wruza 3y agoIt makes it incompatible with concatenative streaming, requires O(n^2) reparsing on every new chunk instead of just scanning for \n. And if you have to parse values to detect end of record anyway, there’s no point in having jsonl standard at all, since you can just try to parse until the matching brace and repeat on success.
- Kwpolska 3y agoWhat’s the difference between terminators and separators here? The ndjson spec [0] doesn’t say anything like that, and it seems that ndjson and jsonlines are identical in what documents they accept. [0]: https://github.com/ndjson/ndjson-spec https://github.com/ndjson/ndjson-spec
- nasso_dev 3y agoim guessing: all values must always end with a terminator, but a separator doesn't need to be present after the last value i.e. a documemt without a newline at the end is valid jsonl, but invalid ndjson
- yencabulator 3y agoA separator separates two records, where as a terminator terminates a record. You can detect and error out if you see an unterminated record at end of transmission. With separators, the producer might not put a separator after the last record, because there's nothing to separate it from there. There's no justification for not using terminators, it's just a bad spec. Unsurprising story: the variant with better marketing has less technical chops.
- simonw 3y agoThis is such a convenient format to work with. It's especially great for streaming consumption without needing to load en entire document into memory at once - you can write a streaming parser just by consuming one line at a time and passing each line to a normal JSON parsing library.
- mojuba 3y agoIt's also great for e.g. feeding search results to a client (the list-as-you-type kind), however JSONL doesn't specify how to end a stream. You will have to come up with your own higher-level protocol that allows having an EOD record.
- throwaway143829 3y agoHandling "end of data" isn't really something that any serialization format needs to specify, so I don't think this needs to be called out for JSONL specifically. e.g. in JS if you're using a ReadableStream you can just check whether a read returns .done==true. This is true regardless of the data you're reading, JSONL or otherwise
- aidos 3y agoFunny enough I’ve found myself just doing it naturally in cases where I was transferring a list of records. I’d still been using the json extension though feeling a bit off about it. Recently (this year) I was flicking through a repo and saw a jsonl extension and immediately guessed what it would contain. It all just fits nicely.
- oulipo 3y agoI've seen "slight improvements" on the format which says: - ignore any line which is exactly "[" or "]" - ignore any trailing commas This lets you consume your API content both as a JSONL stream, and as a JSON array not sure if it's that useful though
- zX41ZdbW 3y agoClickHouse automatically detects variations of the JSONEachRow format: - allowing optionally enclosing the data into an array; - allowing comma as an optional separator between objects; - allowing JSON objects without a newline separator; - properly escaping Unicode newlines and / characters inside string literals for JavaScript compatibility; - allowing binary data inside string literals;
- enriquto 3y agoIn a few years we will rediscover column-based, punctuation-free text files that can be parsed with single call to scanf(3).
- saghm 3y agoMaybe I'm reading too generously, but my sense is that the format described in the article is intended to be used for encoding and transferring data to be used in some other application rather than something where you can assume constraints like "no punctuation". I could imagine using the "JSON lines" format for things like encoding a large number of records to insert for test cases or as a way of sharing a test dataset with another developer; I'm not convinced that either of those would work well without any punctuation whatsoever, and even if they could, I suspect I'd still rather pay the cost of slower importing in order to make my testing code (and any manual reading/transformation outside of my code) easy to do without having to add some additional processing step to deal with the punctuation. The best argument against this format to me is that it's possible to just use JSON itself for this by wrapping all of the values in an array. I'm not totally convinced that the quality of life improvements of not needing to add commas along with the single pair of square braces necessitate an entirely new file format, but realistically I think I'll just end up following whatever happens to be supported by the tooling I use for a given project. If "JSON lines" became widely used overnight, I wouldn't be bothered by it, but if I never end up seeing it supported by anything, I doubt I'll miss it.
- Arch485 3y agoOne particular downside to using an actual array instead of something like JSON lines is that you can't really stream the data. If you have so much data that you need to stream it, you probably shouldn't be using JSON, though...
- fermisea 3y agoShopify, for instance, uses this format if you request bulk downloads of records, for this exact reason.
- 3y ago
- fahhem 3y agoI can't tell if this was created before or after all the Examples shown? It feels like people 'invented' this, then this website wrote down the implicit standard everyone already converged on.
- deleted 3y ago[deleted]
- anshargal 3y agoThere is also a similar thing for CSV files encoded as JSON arrays - csvj.org
- deleted 3y ago[deleted]
- remus 3y agoIf the data can conveniently be split in to lines, I wonder if there is an advantage to this format over a single json list e.g. [ {"name":"Bob"}, {"name":"Alice"}, {"name":"Alison"} ] You're already working with json anyway so you could just stay in that world rather than mixing new lines and json.
- est 3y agomost JSON parser would have difficulty decoding an array with 1 million rows. For example in Python you could easily eat up all or RAM and swap and gets OOM'd
- zipping1549 3y agoIf we're having milions of line of anything, just go with an actual db, no?
- masklinn 3y agoHow does "an actual db" help you with an event stream e.g. over a socket? Are you going to stream WAL records and ask the client to merge that into their local database?
- rented_mule 3y agoNo, this is often used for logging. In my last job, we were generating on the order of a million lines like this per hour in the log I worked with most often (many millions per hour if you count all the logs in the company). Each line of that log was between a few hundred bytes and ~100 KB. It would take a lot of work to support that in a database. That log was processed by many Spark jobs (some in streaming fashion, others in batch). "Small" summaries were added to a columnar database each night. The main table in that database contained hundreds of columns and billions of rows for more interactive analysis.
- baq 3y agoWhy would I need a db for a meager million rows? Even excel handles a million rows
- hden 3y agoWhen will this get a proper IETF RFC?
- larsga 3y agoFrom the page: > any help writing the RFC would be greatly appreciated (see issue). I guess you can sign up if you want an RFC.
- ngalaiko 3y agocan't get over the fact that this page uses > Tactile theme by Jason Long and the format extension is .jsonl
- DonHopkins 3y agoDoes anyone find themselves accidentally mispronouncing Jason like "JSON"? It must be as annoying as mispronouncing Aaron like "A-A-Ron". https://www.youtube.com/watch?v=Dd7FixvoKBw https://www.youtube.com/watch?v=Dd7FixvoKBw
- throwanem 3y agoIt's always been odd to me that "Jason" and "JSON" didn't end up homophonous. But as an A-A-Ron, if anyone named Jason ever quotes the Key & Peele sketch at me, I'll no longer need to think about how to retort!
- DonHopkins 3y agoWithin my circle of friends, all Aarons get Key & Peele's "A-A-Ron" sketch quoted to them as often as all Daves get Cheech & Chong's "Dave's Not Here" sketch quoted to them.
- iainmerrick 3y agoWait, isn't that exactly how you say JSON??
- yogorenapan 3y agoJaysen- Jay-s-on
- chrismorgan 3y agoHere’s how I’d pronounce the two: JSON: /ˈdʒeɪˌsɒn/ (ɒ like the o in dog, and secondary stress on the “son” part) Jason: /ˈdʒeɪsən/ (ə, schwa, like the first a in bazaar)
- livrem 3y agoThat page could use an intro that explains what it is. Even the examples page begins with something about CSV instead of just a minimal jsonlines example. I THINK that the format is like a JSON file with an implicit top-level list and the top-level (only?) commas replaced by newlines? But I do not see anywhere that is spelled out.
- wruza 3y agoCan we PLEASE not screw this up in an actual standard and define a strict record terminator instead of separators? Editors that remove or fail to add a final newline can go implement themselves.
- yencabulator 3y agoNDJSON, which actually has a rudimentary spec, says > Each JSON text MUST conform to the [RFC8259] standard and MUST be written to the stream followed by the newline character \n (0x0A). https://github.com/ndjson/ndjson-spec/ https://github.com/ndjson/ndjson-spec/ But yeah, these things should be merged and formalized. Being web-adjacent, how likely is that? What we're seeing here is the usual patterns: - the less-useful newer thing with slightly better marketing wins eyeballs even when it's technically inferior - an "ecosystem cornerstone" is someone's pet project from a prior employer and now neglected (https://github.com/ndjson/ndjson-spec/issues/35#issuecomment-1285673417 https://github.com/ndjson/ndjson-spec/issues/35#issuecomment...) - web things don't get RFCs written because web people are disconnected from old school internet people - W3C adjacent people didn't get their RDF tuple world domination and industry standards don't matter to them - even MDN is switching to paywalls instead of being a useful resource
- larsga 3y ago> MIME type may be application/jsonl Don't do this. Don't encourage people to use a value that's not registered yet. Say "x-jsonl", and you're OK.
- throw0101b 3y ago> MIME type may be application/jsonl, but this is not yet standardized; any help writing the RFC would be greatly appreciated (see issue[0]). [0] https://github.com/wardi/jsonlines/issues/19 https://github.com/wardi/jsonlines/issues/19
- echoangle 3y agoWhat’s the worst that can happen when using unregistered MIME types? Is it just aesthetics or does it make some functional difference?
- deleted 3y ago[deleted]
- notso411 3y ago[dead]
- ramses0 3y agoRule #2 is "illegal". JSON is actually "JavaScript OBJECT Notation", and raw integers are not "Objects". For compatibilities sake, I'd strongly encourage that NLD-JSON (New-Line Delimited JSON) contains valid JSON OBJECTS. Some discussion here: https://stackoverflow.com/questions/17224536/why-is-one-number-valid-json https://stackoverflow.com/questions/17224536/why-is-one-numb... ...I am by no means the harbinger of all things JSON, but it's worth pointing out that this "spec" without addressing the issues that can arise from encouraging "bare" values (not wrapped in []/{}) is highly suspect.
- palotasb 3y agoRaw integers are also valid JSON according to the spec: https://www.json.org/json-en.html https://www.json.org/json-en.html json is defined as one element element is defines as ws (whitespace), value, ws value is defined as one of: object, array, string, number, "true", "false", "null" Wrapping in {} to make an object or in [] to make a list (which is also not an object) is not necessary to make a valid JSON document. This is also discussed in the Stack Overflow question linked.
- cprecioso 3y agoJSON has always accepted "bare" values (try JSON.parse in the browser devtools). In the space, a valid json document is any of those "bare" values, or an array or object of those. All of the answers in your linked StackOverflow also mention this. For example, the spec explainer at https://www.json.org/ https://www.json.org/ (on the righ-hand), tells you that a `json` document is an `element`, which is a `value` surrounded by any or no whitespace. This `value` can be a literal, or an object or array. For an (IMO) easier definition, you can check how JSON values are usually defined in TypeScript e.g. https://github.com/sindresorhus/type-fest/blob/main/source/basic.d.ts#L68 https://github.com/sindresorhus/type-fest/blob/main/source/b...
- lifthrasiir 3y agoThat's only true for older version of specifications (which do NOT include json.org, however). The current RFC 8259 simply defines `json-text` as any allowed value.
- slashdev 3y agoA drawback of JSONL (and JSON) for me is that you can't have trailing commas. If the format allows for that, and ignores trailing commas at the end of the line you can stream and parse line-by-line, but you can also take any arbitrary group of lines, wrap them in [], again stripping the last comma (arghh!), and parse it with any standard JSON parser. For example, Datadog doesn't ingest JSONL, so if you have JSONL log files you'll have to convert it to JSON first, you can't just use sendfile to upload your log in chunks, just wrapping it with [ and ] (which can be done with ordinary writes before and after sendfile.)
- lencastre 3y agoIsn’t this already “supported” by vscode?
- SkyPuncher 3y agoI don't understand what this solves. It's JSON format, with an alternative delimiter and lack of a wrapping array. I guess the file extension ensures that JSON is not pretty printed.
- serial_dev 3y agoThe point is that you don't need to read in the whole file before you can start working with the data. This is great because it saves you memory (don't need to store the whole thing, your can store one line at a time depending on your specific issue), latency (don't need to wait for the whole file to be read and parsed, you start immediately after the first line), etc. Huge JSON files might be also too big to load into memory. You can read one line at a time, you can stream your lines, process your input across multiple computers etc. And, as it's just json lines, you also don't need to invent a new notation, specialized tools, etc.
- emadda 3y agoAre there any neat log viewers that allow you to filter and tail logs in jsonl format? I have many systems that log in this format but are quite hard to visually scan during development. I typically create ad hoc filters using jq, rg or a js script.
- jurschreuder 3y agoWe always use: <origin_id> <timestamp> <event_type> {payload_json} \n This way you know what json parser to use for which line. origin_id and timestamp is just something we always use with our EventStream In the database it also has one iterator per origin_id, and within an origin one iterator per event_type. This way you can listen to both an origin_id and a specific event_type while making sure you can batch fetch any missing data if the iterators skip a beat.
- foofie 3y agoCan anyone explain to me what's the selling point of this format?
- jlas 3y agoIt's useful for streaming data where you want to pass small chunks of data incrementally. JSON by itself isn't streamable (you need the entire blob to decode it). But if we just pass smaller json-encoded chunks of data and separate them with a newline, we can make a streaming-compatible JSON protocol.
- foofie 3y ago> It's useful for streaming data where you want to pass small chunks of data incrementally. JSON by itself isn't streamable (you need the entire blob to decode it). What's wrong with JSON patch? https://jsonpatch.com/ https://jsonpatch.com/
- yencabulator 3y agoJSON Patch is incremental updates to a singular JSON document[1]. Lines-of-JSON is streaming delivery of multiple JSON documents. Most producers & consumers will only hold one item in memory at a time. [1]: And kinda sucks at that job. Last time I benchmarked it, a gzipped lines-of-json stream that just repeated the whole document for every edit was smaller than a JSON Patch stream, and re-parsing the document was faster than applying JSON Patches.
- rubatuga 3y agoThis is a good time to mention another text based data format based on readability: NestedText. You can hand edit and then pipe valid NestedText into Unix programs, which makes it useful for portability. It only supports the string type, but this can be desirable for some applications. Find out more at https://nestedtext.org/en/stable/ https://nestedtext.org/en/stable/
- kazinator 3y agoThis is useful because it gives a low-level data link framing by which a processor can recognize JSON objects without understanding JSON. A processor that only understands the data link format (lines) can do useful things like merge or interleave two or more such streams, delete the n-th record given n, and such. Though, I wouldn't call it a vista of limitless possibilities or anything of the sort. :) As soon as you have to crack the JSON to do anything interesting, the line orientation becomes more or less moot.