3 ms·

I find JSON “array of arrays” to be a better solution while also remaining human-readable. Especially when formatted as one array per line, analogous to CSV.

xtracto 5y ago
Right on. And with the inclusion of a JSON-schema header, it makes data parsing a breeze: { "$schema": "http://json-schema.org/draft-07/schema#", "type": "array", "items": {"type":"array","items": [ { "type": "number" }, { "type": "string" }, { "enum": ["Street", "Avenue", "Boulevard"] },{ "enum": ["NW", "NE", "SW", "SE"] } ]} } [[3,"some", "Street", "NE"], [4,"other", "Avenue", "SE"], [5,"some", "Boulevard", "SW"]]
- Existenceblinks 5y ago
  With draft 2020-12, you can even have module-like based on [Compound Document spec](https://json-schema.org/draft/2020-12/json-schema-core.html#rfc.section.9.3 https://json-schema.org/draft/2020-12/json-schema-core.html#...) Now you have tables and relation. Only few people understand the usefulness of Compound Document, currently.
lufte 5y ago
Parsing a CSV can be done in a line-per-line basis, while a an array of arrays in JSON is not valid until you reach the end. How would any existing JSON parser handle 75GB of data in a single array of arrays?
- WorldMaker 5y ago
  Many JSON Parsers already have an incremental forward read mode (think SAX-style parser if you are familiar with XML parser styles) where you can ask for each array inside that array of arrays one at a time as it reads them. If something unexpected happens at the end of the large outer array such as syntax error you can decide at that point if you rollback what you've already read/operated on or not. JSON.Parse() in the browser isn't that sort of parser obviously, but there options in many languages such as both major .NET JSON libraries (Newtonsoft and System.Text.Json) have ways to parse that way. Similar examples exist in other languages too.
  - lufte 5y ago
    Interesting. I don't know of any similar option in Python's json module, which is the one that I mostly use, but good to know this exists.
    - WorldMaker 5y ago
      Yes, Python's bundled json module does not support that style of parsing. I saw several suggestions in a quick search to try NAYA [1] if you need that in Python (it's a no dependency Python 3 library) or one of the C-backed wrapper libraries NAYA mentions at the bottom of its Readme if you can afford native dependencies. [1] https://github.com/danielyule/naya https://github.com/danielyule/naya

xtracto 5y ago

Right on. And with the inclusion of a JSON-schema header, it makes data parsing a breeze: { "$schema": "http://json-schema.org/draft-07/schema#", "type": "array", "items": {"type":"array","items": [ { "type": "number" }, { "type": "string" }, { "enum": ["Street", "Avenue", "Boulevard"] },{ "enum": ["NW", "NE", "SW", "SE"] } ]} } [[3,"some", "Street", "NE"], [4,"other", "Avenue", "SE"], [5,"some", "Boulevard", "SW"]]

Existenceblinks 5y ago
With draft 2020-12, you can even have module-like based on [Compound Document spec](https://json-schema.org/draft/2020-12/json-schema-core.html#rfc.section.9.3 https://json-schema.org/draft/2020-12/json-schema-core.html#...) Now you have tables and relation. Only few people understand the usefulness of Compound Document, currently.

lufte 5y ago

Parsing a CSV can be done in a line-per-line basis, while a an array of arrays in JSON is not valid until you reach the end. How would any existing JSON parser handle 75GB of data in a single array of arrays?

WorldMaker 5y ago
Many JSON Parsers already have an incremental forward read mode (think SAX-style parser if you are familiar with XML parser styles) where you can ask for each array inside that array of arrays one at a time as it reads them. If something unexpected happens at the end of the large outer array such as syntax error you can decide at that point if you rollback what you've already read/operated on or not. JSON.Parse() in the browser isn't that sort of parser obviously, but there options in many languages such as both major .NET JSON libraries (Newtonsoft and System.Text.Json) have ways to parse that way. Similar examples exist in other languages too.
- lufte 5y ago
  Interesting. I don't know of any similar option in Python's json module, which is the one that I mostly use, but good to know this exists.
  - WorldMaker 5y ago
    Yes, Python's bundled json module does not support that style of parsing. I saw several suggestions in a quick search to try NAYA [1] if you need that in Python (it's a no dependency Python 3 library) or one of the C-backed wrapper libraries NAYA mentions at the bottom of its Readme if you can afford native dependencies. [1] https://github.com/danielyule/naya https://github.com/danielyule/naya