4 ms·

NDJSON? Sqlite?

by easrng 4y ago

NDJSON? Sqlite?

ironick09 4y ago
csv is many times over again a simpler format than sqlite and easier to understand by anyone across the world. Never heard of ndjson, can’t see one publishing this data in a format that isn’t nearly as common as something like csv (or regular json which some of the data is published in).
- easrng 4y ago
  CSV only seems simple. Lots of parsing edge cases. Sqlite isn't readable by hand but it's basically bulletproof. NDJSON is literally just newline seperated JSON, it's just easier to process as a stream without a special parser.
  - irrelative 4y ago
    Sure, could be simpler, but there's a spec and multiple implementations: https://www.rfc-editor.org/rfc/rfc4180.html https://www.rfc-editor.org/rfc/rfc4180.html And the spec is ~5 pages. Not sure why NDJSON is considered simpler, as json objects can be arbitrarily nested. Breaking into records is easier, but parsing is harder.
    - easrng 4y ago
      (ND)JSON is simpler because people actually follow the spec, unlike with CSV.
      - Dylan16807 4y ago
        People violate the JSON spec all the time. And following the CSV spec is much easier than following the JSON spec. And there's only like three edge cases.
        mschuster91 4y ago
        > And there's only like three edge cases. Unless you involve MS Excel or, worse, MS Excel on macOS. OTOH, pitfalls are: UTF-8 BOM, comma vs semicolon, single vs double quote, multiline cell content and escaping, escaping in general...
mr_toad 4y ago
Then you’d have to get your hands on a parser. Who wants to write up the business case and spend months going through change control (they’d probably just say no). JSON and NDJSON can also be much larger than CSV files if the wrong structure is used.
mattewong 4y ago
All this banter arguing over CSV, JSON, sqlite seems unnecessary when you can just push format X through a pipe and get whichever format Y you want back out: https://github.com/liquidaty/zsv/blob/main/docs/csv_json_sqlite.md https://github.com/liquidaty/zsv/blob/main/docs/csv_json_sql... (disclaimer: I'm one of the zsv authors)

NDJSON? Sqlite?

by easrng 4y ago

NDJSON? Sqlite?

csv is many times over again a simpler format than sqlite and easier to understand by anyone across the world. Never heard of ndjson, can’t see one publishing this data in a format that isn’t nearly as common as something like csv (or regular json which some of the data is published in).

easrng 4y ago
CSV only seems simple. Lots of parsing edge cases. Sqlite isn't readable by hand but it's basically bulletproof. NDJSON is literally just newline seperated JSON, it's just easier to process as a stream without a special parser.
- irrelative 4y ago
  Sure, could be simpler, but there's a spec and multiple implementations: https://www.rfc-editor.org/rfc/rfc4180.html https://www.rfc-editor.org/rfc/rfc4180.html And the spec is ~5 pages. Not sure why NDJSON is considered simpler, as json objects can be arbitrarily nested. Breaking into records is easier, but parsing is harder.
  - easrng 4y ago
    (ND)JSON is simpler because people actually follow the spec, unlike with CSV.
    - Dylan16807 4y ago
      People violate the JSON spec all the time. And following the CSV spec is much easier than following the JSON spec. And there's only like three edge cases.
      - mschuster91 4y ago
        > And there's only like three edge cases. Unless you involve MS Excel or, worse, MS Excel on macOS. OTOH, pitfalls are: UTF-8 BOM, comma vs semicolon, single vs double quote, multiline cell content and escaping, escaping in general...

mr_toad 4y ago

Then you’d have to get your hands on a parser. Who wants to write up the business case and spend months going through change control (they’d probably just say no). JSON and NDJSON can also be much larger than CSV files if the wrong structure is used.

mattewong 4y ago

All this banter arguing over CSV, JSON, sqlite seems unnecessary when you can just push format X through a pipe and get whichever format Y you want back out: https://github.com/liquidaty/zsv/blob/main/docs/csv_json_sqlite.md https://github.com/liquidaty/zsv/blob/main/docs/csv_json_sql... (disclaimer: I'm one of the zsv authors)