3 ms·
This is such a convenient format to work with. It's especially great for streaming consumption without needing to load en entire document into memory at once -
by simonw 3y ago
This is such a convenient format to work with. It's especially great for streaming consumption without needing to load en entire document into memory at once - you can write a streaming parser just by consuming one line at a time and passing each line to a normal JSON parsing library.
- mojuba 3y agoIt's also great for e.g. feeding search results to a client (the list-as-you-type kind), however JSONL doesn't specify how to end a stream. You will have to come up with your own higher-level protocol that allows having an EOD record.
- throwaway143829 3y agoHandling "end of data" isn't really something that any serialization format needs to specify, so I don't think this needs to be called out for JSONL specifically. e.g. in JS if you're using a ReadableStream you can just check whether a read returns .done==true. This is true regardless of the data you're reading, JSONL or otherwise
- aidos 3y agoFunny enough I’ve found myself just doing it naturally in cases where I was transferring a list of records. I’d still been using the json extension though feeling a bit off about it. Recently (this year) I was flicking through a repo and saw a jsonl extension and immediately guessed what it would contain. It all just fits nicely.
- oulipo 3y agoI've seen "slight improvements" on the format which says: - ignore any line which is exactly "[" or "]" - ignore any trailing commas This lets you consume your API content both as a JSONL stream, and as a JSON array not sure if it's that useful though
- zX41ZdbW 3y agoClickHouse automatically detects variations of the JSONEachRow format: - allowing optionally enclosing the data into an array; - allowing comma as an optional separator between objects; - allowing JSON objects without a newline separator; - properly escaping Unicode newlines and / characters inside string literals for JavaScript compatibility; - allowing binary data inside string literals;
- notso411 3y ago[dead]
- jgalt212 3y agoIt might be too early for my brain, but splitting on "\n" won't work if any of your json objects have newlines in the strings. Does this regex work? As I see it, the tricky one is matching a single string json object. /(\]|\})\n|(.")\n/ https://www.debuggex.com/r/LaqAP5wmPFuRLNBO https://www.debuggex.com/r/LaqAP5wmPFuRLNBO
- simonw 3y agoNew line characters aren't allowed in JSON strings - JSON uses \n for those.
- jgalt212 3y agoWhat's the difference between a newline character in a string and \n from a regex perspective?
- yogorenapan 3y agoIt would be \\n
- dwaite 3y agoJSON doesn't have native regex support, so you would be representing a regex as a string. /\n/ in regex would need the backslash escaped, so "/\\n/"
- yesco 3y ago"\n" is really just two arbitrary characters right? An actual "newline" is just a single invisible character, but we represent it as "\n" for convenience. So in json you are simply putting in the magic character sequence: "\n" in place of the real newline character. Not unlike how many compilers will auto translate "\n" sequences you put in strings into the actual newline character. I believe most json encoders will automatically convert newline characters into the two character sequence, so when you put it all together it goes: character sequence "\n" typed into source file -> compiler converts newline character into actual an actual newline character -> json encoder converts newline character back into "\n" sequence. With all this in mind, since regex is basically doing the same thing as a compiler, treating "\n" sequences as a search for a newline character, you would need to escape the backslash so it is interpreted literally, so "\\n". This gets even weirder depending on the tool/language you are doing this in though, like if you are writing out a regex within a string, the regex engine might not need to have any special logic for "\n" since the compiler is already handling it! In which case: "\\n" is an escape for the compiler not the regex engine. TLDR: You just need to escape it: "\\n"