3 ms·

JSONH couldn't replace CSV in the one place where it actually makes sense to use CSV: very large files. It's possible to process a CSV file of unbounded size w

by rcfox 12y ago

JSONH couldn't replace CSV in the one place where it actually makes sense to use CSV: very large files. It's possible to process a CSV file of unbounded size with a bounded amount of memory because you can just read it line-by-line. With JSON, you need to read the whole file into memory before you can get at the data.

burntsushi 12y ago
I'm not familiar with JSONH, but with JSON, you can cheat by encoding a single "record" via JSON on a single line. :-) That way, you can read one line at a time: {"field": "some\\ntext with new lines"} {"field": "more text"}
- deleted 12y ago
  [deleted]
leeoniya 12y ago
well, there's nothing stopping you from writing a streaming json parser. that can read <x> fields at a time. you can format your input json to have whatever whitespace you need, including line breaks. the assumption is that the json will be encoded and not hand-written anyways, so the validity of the entire file isnt important since it will barf at record-level if malformed, similar to csv. so rather than: [2,"col_1","col_2","a","b","c","d"] you can write: [2,"col_1","col_2", "a","b", "c","d" ] and it will still be valid JSON/H and line-by-line parsed. or as 'burntsushi suggested, you can wrap each record in an array to pass through a normal json parser, though this wouldn't be JSONH any longer: [ ["col_1","col_2"], ["a","b"], ["c","d"] ]
- rcfox 12y ago
  Sure, you can write a custom parser to handle this, but then you're no longer working with JSON/JSONH. It's a subset that's nearly identical, but with an extra constraint imposed on the serialization structure. (We'll call it JSON`) It sounds like I'm being pedantic, but if you're not your sole producer and consumer of the data you'll inevitably have to deal with someone giving you valid JSON that isn't valid JSON`.
  - leeoniya 12y ago
    all of the above are valid json. especially the last simple variation where you literally just have a 2-level nested array.

by rcfox 12y ago

burntsushi 12y ago

I'm not familiar with JSONH, but with JSON, you can cheat by encoding a single "record" via JSON on a single line. :-) That way, you can read one line at a time: {"field": "some\\ntext with new lines"} {"field": "more text"}

deleted 12y ago
[deleted]

leeoniya 12y ago

well, there's nothing stopping you from writing a streaming json parser. that can read <x> fields at a time. you can format your input json to have whatever whitespace you need, including line breaks. the assumption is that the json will be encoded and not hand-written anyways, so the validity of the entire file isnt important since it will barf at record-level if malformed, similar to csv. so rather than: [2,"col_1","col_2","a","b","c","d"] you can write: [2,"col_1","col_2", "a","b", "c","d" ] and it will still be valid JSON/H and line-by-line parsed. or as 'burntsushi suggested, you can wrap each record in an array to pass through a normal json parser, though this wouldn't be JSONH any longer: [ ["col_1","col_2"], ["a","b"], ["c","d"] ]

rcfox 12y ago
Sure, you can write a custom parser to handle this, but then you're no longer working with JSON/JSONH. It's a subset that's nearly identical, but with an extra constraint imposed on the serialization structure. (We'll call it JSON`) It sounds like I'm being pedantic, but if you're not your sole producer and consumer of the data you'll inevitably have to deal with someone giving you valid JSON that isn't valid JSON`.
- leeoniya 12y ago
  all of the above are valid json. especially the last simple variation where you literally just have a 2-level nested array.