4 ms·

>Require a standard format. Pick JSON or CSV in some standard encoding. We don't need hospitals publishing XML files and Excel spreadsheets. Definitely CSV. Al

>Require a standard format. Pick JSON or CSV in some standard encoding. We don't need hospitals publishing XML files and Excel spreadsheets. Definitely CSV. Always CSV. It's easy to produce, easy to parse, and kind of human readable. And easy to do some basic analysis with tools that everyone has access to - Excel, Google Sheets etc.

sp332 4y ago
I didn't like CSV because too much actual data has commas in it, so commas aren't a good delimiter.
- slivanes 4y ago
  Quoting every column is a safe way to handle this, at the cost of slightly larger files. You can then escape a quote with another quote.
  - robocat 4y ago
    That is just one simple problem. People say CSV is hard because it is hard: https://www.delimited.dev/csv-file-quirks/ https://www.delimited.dev/csv-file-quirks/
- patwolf 4y ago
  From the perspective of reading/writing CSVs programmatically, I actually prefer that it's a common character. You'll discover very quickly whether the parser handles escapes correctly. With a more exotic delimiter it might be a long time to discover the problem, and by then it will be harder to fix. From the perspective of writing CSVs by hand in a text editor, commas suck.
- EvanAnderson 4y ago
  Tabs are a better delimiter than commas. Using the ASCII control characters meant for record and field delimiters are even better: https://en.wikipedia.org/wiki/C0_and_C1_control_codes https://en.wikipedia.org/wiki/C0_and_C1_control_codes
albertopv 4y ago
In 2007 (15 years ago) Veneto and Lombardia italian regions elder health care ptivate facilities had to provide monthly data with a specific, mandatory excel template provided by regional health authorities. The company I worked for was in charge to collect data and create those files. It was not a perfect format, it was closed source format (still .xls) but everyone knew it was that format. Standards matter.
zarzavat 4y ago
You cannot require CSV as a standard format because CSV itself has no standard. There are multiple variants of CSV and there's no way for a CSV file to declare which variant it's using in a standard way, which means you have to manually check that every single file parsed correctly, because if you accidentally parse a CSV file as the wrong variant there is no way for the parser to know and it will just silently corrupt your data. Sometimes CSV files themselves are ambiguous because they don't use any quoting so you end up with some lines that have more columns than others, and require manual repair line-by-line by a human. Don't use CSV, it's a dreadful format.
- hwillis 4y ago
  > Sometimes CSV files themselves are ambiguous because they don't use any quoting so you end up with some lines that have more columns than others, and require manual repair line-by-line by a human. It's wild that we haven't just started using unicode and line/group separator characters for tabular data instead of csv/tsv. Whoever decided to use a common semantic character as syntax is an evil, devious bastard. That said, there is one thing I like about CSV and similar formats: they are inherently tabular. You don't have to worry about someone adding sub-objects to your JSON (or replacing a primitive field with an object/array). Extra columns are brainless to add or process, but converting sub-data to a line is not.

sp332 4y ago

I didn't like CSV because too much actual data has commas in it, so commas aren't a good delimiter.

slivanes 4y ago
Quoting every column is a safe way to handle this, at the cost of slightly larger files. You can then escape a quote with another quote.
- robocat 4y ago
  That is just one simple problem. People say CSV is hard because it is hard: https://www.delimited.dev/csv-file-quirks/ https://www.delimited.dev/csv-file-quirks/
patwolf 4y ago
From the perspective of reading/writing CSVs programmatically, I actually prefer that it's a common character. You'll discover very quickly whether the parser handles escapes correctly. With a more exotic delimiter it might be a long time to discover the problem, and by then it will be harder to fix. From the perspective of writing CSVs by hand in a text editor, commas suck.
EvanAnderson 4y ago
Tabs are a better delimiter than commas. Using the ASCII control characters meant for record and field delimiters are even better: https://en.wikipedia.org/wiki/C0_and_C1_control_codes https://en.wikipedia.org/wiki/C0_and_C1_control_codes

albertopv 4y ago

In 2007 (15 years ago) Veneto and Lombardia italian regions elder health care ptivate facilities had to provide monthly data with a specific, mandatory excel template provided by regional health authorities. The company I worked for was in charge to collect data and create those files. It was not a perfect format, it was closed source format (still .xls) but everyone knew it was that format. Standards matter.

zarzavat 4y ago

You cannot require CSV as a standard format because CSV itself has no standard. There are multiple variants of CSV and there's no way for a CSV file to declare which variant it's using in a standard way, which means you have to manually check that every single file parsed correctly, because if you accidentally parse a CSV file as the wrong variant there is no way for the parser to know and it will just silently corrupt your data. Sometimes CSV files themselves are ambiguous because they don't use any quoting so you end up with some lines that have more columns than others, and require manual repair line-by-line by a human. Don't use CSV, it's a dreadful format.

hwillis 4y ago
> Sometimes CSV files themselves are ambiguous because they don't use any quoting so you end up with some lines that have more columns than others, and require manual repair line-by-line by a human. It's wild that we haven't just started using unicode and line/group separator characters for tabular data instead of csv/tsv. Whoever decided to use a common semantic character as syntax is an evil, devious bastard. That said, there is one thing I like about CSV and similar formats: they are inherently tabular. You don't have to worry about someone adding sub-objects to your JSON (or replacing a primitive field with an object/array). Extra columns are brainless to add or process, but converting sub-data to a line is not.