4 ms·

One of the first things I did at Cloudflare was change the format of a file that contained vital information about the mapping between IPs and zones (no longer

One of the first things I did at Cloudflare was change the format of a file that contained vital information about the mapping between IPs and zones (no longer used these days) so that it had something like: START <count> <sha1> . . . END because it was just slurped up line by line assuming EOF was good.

sitkack 4y ago
Every mission critical file eventually gets version numbers and checksums.
- kridsdale1 4y ago
  And eventually a blockchain
  - smt88 4y ago
    Files can be immutable, cryptographically-verifiable, and distributed across servers without a blockchain. Adding a blockchain to this would slow things down, add surface area for errors, and provide absolutely no value. This is true of every usage of blockchain other than creating tokens.
cheeselip420 4y ago
one reason why JSON is superior to things like TOML or YAML for these use-cases...
- e12e 4y ago
  Not to worry, JSONL[1] fixes that ;) In all seriousness - just dropping 1gb json file with N million records in one end probably isn't great either. I suppose one could somehow marry js and subresource integrity protection[2] to get a json serialized structure with hash integrity check. It would probably be a terrible idea. [1] https://jsonlines.org/ https://jsonlines.org/ [2] https://developer.mozilla.org/en-US/docs/Web/Security/Subresource_Integrity https://developer.mozilla.org/en-US/docs/Web/Security/Subres...
  - ehPReth 4y ago
    > Text editing programs call the first line of a text file "line 1". The first value in a JSON Lines file should also be called "value 1". I wonder why not zero?
  - metadat 4y ago
    In my experience, this has been called "ndjson", or newline-delimited JSON. It's a remarkably effective and useful pattern. Apparently jsonlines is only a website, whereas ndjson is a website+spec. Given the conceptual simplicity, this claim seems a bit dubious. Having two of these identical things, each one clawing for mindshare is dumb and counter-productive. Why can't we all be friendly and get along for the greater good? Oh well, it's par for the course. http://ndjson.org/ http://ndjson.org/
    - e12e 4y ago
      That is probably the canonical term/format - I came across the reference to JSONL somewhere related to (de)serializing data for ai like: https://platform.openai.com/docs/guides/fine-tuning/prepare-training-data https://platform.openai.com/docs/guides/fine-tuning/prepare-... And googled/ducked from there. Is it really better than just using an array, though? Now you essentially first parse the data/newline files, splitting blindly on "\n" - hoping that a) you've got the data, and b) that each record can be parsed as json - by an entirely different parser. Meanwhile - a valid json file (Array of records) - can be quite quickly parsed by an optimized json parser, will error on wrong or truncated input, will (have to) handle reasonable whitespace?
      - metadat 4y ago
        > Is it really better than just using an array, though? In many cases, absolutely yes it's better. Otherwise you can end up being forced to cram several GB or even 10's of GB of JSON into working memory. Not ideal. Usually each ndjson file is generated by a single process, so there isn't much concern about invalid formatting making it's way in. It's one of those rare win-wins.

sitkack 4y ago

Every mission critical file eventually gets version numbers and checksums.

kridsdale1 4y ago
And eventually a blockchain
- smt88 4y ago
  Files can be immutable, cryptographically-verifiable, and distributed across servers without a blockchain. Adding a blockchain to this would slow things down, add surface area for errors, and provide absolutely no value. This is true of every usage of blockchain other than creating tokens.

cheeselip420 4y ago

one reason why JSON is superior to things like TOML or YAML for these use-cases...

e12e 4y ago
Not to worry, JSONL[1] fixes that ;) In all seriousness - just dropping 1gb json file with N million records in one end probably isn't great either. I suppose one could somehow marry js and subresource integrity protection[2] to get a json serialized structure with hash integrity check. It would probably be a terrible idea. [1] https://jsonlines.org/ https://jsonlines.org/ [2] https://developer.mozilla.org/en-US/docs/Web/Security/Subresource_Integrity https://developer.mozilla.org/en-US/docs/Web/Security/Subres...
- ehPReth 4y ago
  > Text editing programs call the first line of a text file "line 1". The first value in a JSON Lines file should also be called "value 1". I wonder why not zero?
- metadat 4y ago
  In my experience, this has been called "ndjson", or newline-delimited JSON. It's a remarkably effective and useful pattern. Apparently jsonlines is only a website, whereas ndjson is a website+spec. Given the conceptual simplicity, this claim seems a bit dubious. Having two of these identical things, each one clawing for mindshare is dumb and counter-productive. Why can't we all be friendly and get along for the greater good? Oh well, it's par for the course. http://ndjson.org/ http://ndjson.org/
  - e12e 4y ago
    That is probably the canonical term/format - I came across the reference to JSONL somewhere related to (de)serializing data for ai like: https://platform.openai.com/docs/guides/fine-tuning/prepare-training-data https://platform.openai.com/docs/guides/fine-tuning/prepare-... And googled/ducked from there. Is it really better than just using an array, though? Now you essentially first parse the data/newline files, splitting blindly on "\n" - hoping that a) you've got the data, and b) that each record can be parsed as json - by an entirely different parser. Meanwhile - a valid json file (Array of records) - can be quite quickly parsed by an optimized json parser, will error on wrong or truncated input, will (have to) handle reasonable whitespace?
    - metadat 4y ago
      > Is it really better than just using an array, though? In many cases, absolutely yes it's better. Otherwise you can end up being forced to cram several GB or even 10's of GB of JSON into working memory. Not ideal. Usually each ndjson file is generated by a single process, so there isn't much concern about invalid formatting making it's way in. It's one of those rare win-wins.