3 ms·

An example I currently have, Statistics Norway recently released version 2 of their API. You can now download about every data source they have. However they pr

by olavgg 10y ago

An example I currently have, Statistics Norway recently released version 2 of their API. You can now download about every data source they have. However they prefer that developers use JSON-Stat(https://json-stat.org/ https://json-stat.org/) instead of CVS's. Luckily they do support CSV, but it is messy with inconsistent layout. I still prefer it over JSON-Stat though, as I feel JSON-Stat is way too complicated for serious use. That is also probably why there are so few JSON-Stat clients. CSV's are simple to parse! JSON, XML are more complex beasts to parse, they are especially hard to parse when they don't fit in memory. I do use a lot JSON, dont get me wrong, but it's mostly for small data sizes. Do anyone else here feel the same about JSON-Stat?

lolive 10y ago
I typed "stream json parse java" this afternoon and got a pretty good answer from Google.
alblue 10y ago
Anyone who claims that CSVs are easy to parse have obviously never had to deal with the kind of junk that you see in real CSV files. Quite apart from quotes (is a "a,b" two fields with '"a' and 'b"' or a single field with three characters?) there are also things like new lines like like,"this newline" Which are part of the field and not two records. Or having to deal with user names like Blue, Al (outlook's preferred format for representing people's names). Or you end up with O'Reilly - is the ' part of a quoted string or not? And of course a comma isn't necessarily the only separator - what about tabs? Or a document where an intermediary has saved it in an editor that has "helpfully" converted tabs to N spaces (where N is a universally disagreed upon nonzero positive integer)
- olavgg 10y ago
  You're absolutely right, but this is still a problem with JSON and XML. I have seen dates being represented in all kind of formats in the same document. Data cleansing has always been important when working with data in any format.
  - reverius42 10y ago
    The difference is that JSON and XML can be described by a grammar and parsed by any standards-conforming parser. Good luck doing that with CSV.

by olavgg 10y ago

lolive 10y ago

I typed "stream json parse java" this afternoon and got a pretty good answer from Google.

alblue 10y ago

Anyone who claims that CSVs are easy to parse have obviously never had to deal with the kind of junk that you see in real CSV files. Quite apart from quotes (is a "a,b" two fields with '"a' and 'b"' or a single field with three characters?) there are also things like new lines like like,"this newline" Which are part of the field and not two records. Or having to deal with user names like Blue, Al (outlook's preferred format for representing people's names). Or you end up with O'Reilly - is the ' part of a quoted string or not? And of course a comma isn't necessarily the only separator - what about tabs? Or a document where an intermediary has saved it in an editor that has "helpfully" converted tabs to N spaces (where N is a universally disagreed upon nonzero positive integer)

olavgg 10y ago
You're absolutely right, but this is still a problem with JSON and XML. I have seen dates being represented in all kind of formats in the same document. Data cleansing has always been important when working with data in any format.
- reverius42 10y ago
  The difference is that JSON and XML can be described by a grammar and parsed by any standards-conforming parser. Good luck doing that with CSV.