4 ms·

one reason why JSON is superior to things like TOML or YAML for these use-cases...

e12e 4y ago
Not to worry, JSONL[1] fixes that ;) In all seriousness - just dropping 1gb json file with N million records in one end probably isn't great either. I suppose one could somehow marry js and subresource integrity protection[2] to get a json serialized structure with hash integrity check. It would probably be a terrible idea. [1] https://jsonlines.org/ https://jsonlines.org/ [2] https://developer.mozilla.org/en-US/docs/Web/Security/Subresource_Integrity https://developer.mozilla.org/en-US/docs/Web/Security/Subres...
- ehPReth 4y ago
  > Text editing programs call the first line of a text file "line 1". The first value in a JSON Lines file should also be called "value 1". I wonder why not zero?
- metadat 4y ago
  In my experience, this has been called "ndjson", or newline-delimited JSON. It's a remarkably effective and useful pattern. Apparently jsonlines is only a website, whereas ndjson is a website+spec. Given the conceptual simplicity, this claim seems a bit dubious. Having two of these identical things, each one clawing for mindshare is dumb and counter-productive. Why can't we all be friendly and get along for the greater good? Oh well, it's par for the course. http://ndjson.org/ http://ndjson.org/
  - e12e 4y ago
    That is probably the canonical term/format - I came across the reference to JSONL somewhere related to (de)serializing data for ai like: https://platform.openai.com/docs/guides/fine-tuning/prepare-training-data https://platform.openai.com/docs/guides/fine-tuning/prepare-... And googled/ducked from there. Is it really better than just using an array, though? Now you essentially first parse the data/newline files, splitting blindly on "\n" - hoping that a) you've got the data, and b) that each record can be parsed as json - by an entirely different parser. Meanwhile - a valid json file (Array of records) - can be quite quickly parsed by an optimized json parser, will error on wrong or truncated input, will (have to) handle reasonable whitespace?
    - metadat 4y ago
      > Is it really better than just using an array, though? In many cases, absolutely yes it's better. Otherwise you can end up being forced to cram several GB or even 10's of GB of JSON into working memory. Not ideal. Usually each ndjson file is generated by a single process, so there isn't much concern about invalid formatting making it's way in. It's one of those rare win-wins.
      - e12e 4y ago
        I could see how a language might have a streaming json parser, but where a naïve "for each line in file" might read the file as a string - allocating all that memory anyway?
        metadat 4y ago
        Used properly, "For each line in file" constructs in the major languages (Python, Go, Java, C#, Javascript, and so on) are well tested and really do read only one single line at a time. Just be sure to avoid the usual pitfalls, e.g. don't turn all the lines into a list before starting iteration. I actually recently tried out a python based steaming JSON parser I found on SO and it was actually even more inefficient (for both CPU and memory) than the built-in JSON package (probably because Python and not a C-module). It was also quite a complex state machine, tricky to catch all corner cases. The SO module had some bugs that I fixed, but I ended up abandoning the entire approach because it wasn't saving any memory. NDJSON certainly is a much simpler approach, which I like and appreciate.

e12e 4y ago

Not to worry, JSONL[1] fixes that ;) In all seriousness - just dropping 1gb json file with N million records in one end probably isn't great either. I suppose one could somehow marry js and subresource integrity protection[2] to get a json serialized structure with hash integrity check. It would probably be a terrible idea. [1] https://jsonlines.org/ https://jsonlines.org/ [2] https://developer.mozilla.org/en-US/docs/Web/Security/Subresource_Integrity https://developer.mozilla.org/en-US/docs/Web/Security/Subres...

ehPReth 4y ago
> Text editing programs call the first line of a text file "line 1". The first value in a JSON Lines file should also be called "value 1". I wonder why not zero?
metadat 4y ago
In my experience, this has been called "ndjson", or newline-delimited JSON. It's a remarkably effective and useful pattern. Apparently jsonlines is only a website, whereas ndjson is a website+spec. Given the conceptual simplicity, this claim seems a bit dubious. Having two of these identical things, each one clawing for mindshare is dumb and counter-productive. Why can't we all be friendly and get along for the greater good? Oh well, it's par for the course. http://ndjson.org/ http://ndjson.org/
- e12e 4y ago
  That is probably the canonical term/format - I came across the reference to JSONL somewhere related to (de)serializing data for ai like: https://platform.openai.com/docs/guides/fine-tuning/prepare-training-data https://platform.openai.com/docs/guides/fine-tuning/prepare-... And googled/ducked from there. Is it really better than just using an array, though? Now you essentially first parse the data/newline files, splitting blindly on "\n" - hoping that a) you've got the data, and b) that each record can be parsed as json - by an entirely different parser. Meanwhile - a valid json file (Array of records) - can be quite quickly parsed by an optimized json parser, will error on wrong or truncated input, will (have to) handle reasonable whitespace?
  - metadat 4y ago
    > Is it really better than just using an array, though? In many cases, absolutely yes it's better. Otherwise you can end up being forced to cram several GB or even 10's of GB of JSON into working memory. Not ideal. Usually each ndjson file is generated by a single process, so there isn't much concern about invalid formatting making it's way in. It's one of those rare win-wins.
    - e12e 4y ago
      I could see how a language might have a streaming json parser, but where a naïve "for each line in file" might read the file as a string - allocating all that memory anyway?
      - metadat 4y ago
        Used properly, "For each line in file" constructs in the major languages (Python, Go, Java, C#, Javascript, and so on) are well tested and really do read only one single line at a time. Just be sure to avoid the usual pitfalls, e.g. don't turn all the lines into a list before starting iteration. I actually recently tried out a python based steaming JSON parser I found on SO and it was actually even more inefficient (for both CPU and memory) than the built-in JSON package (probably because Python and not a C-module). It was also quite a complex state machine, tricky to catch all corner cases. The SO module had some bugs that I fixed, but I ended up abandoning the entire approach because it wasn't saving any memory. NDJSON certainly is a much simpler approach, which I like and appreciate.