4 ms·

Every single use I've ever seen of CSV would be improved by the very simple change to TSV. Even Excel can handle it. It is far safer to munge data containing

Every single use I've ever seen of CSV would be improved by the very simple change to TSV. Even Excel can handle it. It is far safer to munge data containing tabs (convert to spaces, etc), than commas (remove? convert to dots? escape?). The better answer is to use ASCII separators as Lyndon Johnson intended, but that turns out to be asking a lot of data producers. Generating TSV is usually easier than generating CSV.

chirau 3y ago
you are assuming the regular end user knows the difference between 4 spaces and a tab and the nuances that come with them trying to replace one with the other or why space between two values is different at one point from the next. Commas are, by far, better delimeters than tabs in the grand scheme of things and with both expert and regular users considered.
- quesera 3y ago
  I disagree. The end user doesn't need to know the difference between 4 spaces and a tab. Tabs are just whitespace. Tabs are uncommon but convenient whitespace. Commas are extremely common content. Tabs are a vastly better delimiter. If you are wrapping source code in a CSV, a) you're doing it wrong, and b) you'll get bitten by newlines just as quickly! If you're including content that requires specific whitespace preservation, just escape the (usually rare) tabs. TSV certainly is not perfect. But it solves the major problems for 95% of CSVs, and it's just as convenient for humans. I do agree that one should not arbitrarily munge content. But note that HTML does munge whitespace, and we've never suffered meaningfully for it.
  - chirau 3y ago
    I don't think you fully understood what i was saying. If a regular user had rows like this Adam Smith 27 WA JonathanBoyd 23 NC They are likely going to have tougher time adding a new row as compared to if it was comma delimited. You underestimate the simplicity of end users and how tabs and spaces can confuse them. This is why they prefer Excel, with boxes, because they cannot keep up with formatting and such. Tabs are spaces to many people. Commas are clearer.
croes 3y ago
Don't change the data by replacing the separator. If the data contains a comma, tab etc. I want to receive a comma, tab etc. With proper escaping the separator doesn't matter. After that csv generating is pretty easy.
- quesera 3y ago
  Right, the problem with CSV is that it's often generated incorrectly. Commas do not need to be escaped inside quoted blocks. But now quoted blocks are not literal data, and you need to drop the enclosing quotes. And escape quotes inside them, or outside if the quote is literal. Also quotes must be balanced inside a delimited field. Etc. Commas are just too common of a character to be a good delimiter. Tabs are an improvement, because usually tabs are just whitespace, which is usually arbitrary, and never needs to be balanced. But tabs are not perfect either. Just a vast improvement for most data, and a functional compromise for humans.
croes 3y ago
BTW is Excel better in reading UTF8 files? Last time I checked it can't read it without errors if you just open the file with it without using the import function an explicitly select UTF8.
- Kwpolska 3y ago
  The trick is using UTF-8 with BOM.
jillesvangurp 3y ago
And as a practical matter, Google sheets can export tsv. It didn't always have csv as an export (it does now). I've used Google sheets as a business friendly way to edit data for many years. With a little bit of instruction, this can work pretty well. Usually it just involves me telling people to not get creative with the column names and leave them alone. This may shock some people but Excell and other MS Office products aren't that common any more to have around for developers. Through the nineties, MS had a defacto monopoly on desktop back in the day. But these days, a lot of developers use macs and they are also popular with business people. Google docs seems popular with startups. All companies I've been in for the last 12 years default to that. Anyway, I've done this on a few teams where it just short cuts the whole discussion about needing a bespoke UI to edit some table of stuff for managers. I've even replaced a failed project to build such a UI with simple stuff like this. All you need is a bit of validation when importing and you can keep the last know good export in git and do pull requests to update the file. There's a weird dynamic where giving this level of control to some managers actually makes them feel more involved and engaged. My preferred format to work with is actually ndjson (newline delimited json). Mainly because it's easier to deal with nested objects and lists in that. Whenever people start putting lists in table cells or start adding dots or other separators to their column names to indicate some kind of hierarcy, ndjson is the better solution. I've seen all sorts of attempts by business people to stuff non tabular data into a table.

chirau 3y ago

you are assuming the regular end user knows the difference between 4 spaces and a tab and the nuances that come with them trying to replace one with the other or why space between two values is different at one point from the next. Commas are, by far, better delimeters than tabs in the grand scheme of things and with both expert and regular users considered.

quesera 3y ago
I disagree. The end user doesn't need to know the difference between 4 spaces and a tab. Tabs are just whitespace. Tabs are uncommon but convenient whitespace. Commas are extremely common content. Tabs are a vastly better delimiter. If you are wrapping source code in a CSV, a) you're doing it wrong, and b) you'll get bitten by newlines just as quickly! If you're including content that requires specific whitespace preservation, just escape the (usually rare) tabs. TSV certainly is not perfect. But it solves the major problems for 95% of CSVs, and it's just as convenient for humans. I do agree that one should not arbitrarily munge content. But note that HTML does munge whitespace, and we've never suffered meaningfully for it.
- chirau 3y ago
  I don't think you fully understood what i was saying. If a regular user had rows like this Adam Smith 27 WA JonathanBoyd 23 NC They are likely going to have tougher time adding a new row as compared to if it was comma delimited. You underestimate the simplicity of end users and how tabs and spaces can confuse them. This is why they prefer Excel, with boxes, because they cannot keep up with formatting and such. Tabs are spaces to many people. Commas are clearer.

croes 3y ago

Don't change the data by replacing the separator. If the data contains a comma, tab etc. I want to receive a comma, tab etc. With proper escaping the separator doesn't matter. After that csv generating is pretty easy.

quesera 3y ago
Right, the problem with CSV is that it's often generated incorrectly. Commas do not need to be escaped inside quoted blocks. But now quoted blocks are not literal data, and you need to drop the enclosing quotes. And escape quotes inside them, or outside if the quote is literal. Also quotes must be balanced inside a delimited field. Etc. Commas are just too common of a character to be a good delimiter. Tabs are an improvement, because usually tabs are just whitespace, which is usually arbitrary, and never needs to be balanced. But tabs are not perfect either. Just a vast improvement for most data, and a functional compromise for humans.

croes 3y ago

BTW is Excel better in reading UTF8 files? Last time I checked it can't read it without errors if you just open the file with it without using the import function an explicitly select UTF8.

Kwpolska 3y ago
The trick is using UTF-8 with BOM.

jillesvangurp 3y ago

And as a practical matter, Google sheets can export tsv. It didn't always have csv as an export (it does now). I've used Google sheets as a business friendly way to edit data for many years. With a little bit of instruction, this can work pretty well. Usually it just involves me telling people to not get creative with the column names and leave them alone. This may shock some people but Excell and other MS Office products aren't that common any more to have around for developers. Through the nineties, MS had a defacto monopoly on desktop back in the day. But these days, a lot of developers use macs and they are also popular with business people. Google docs seems popular with startups. All companies I've been in for the last 12 years default to that. Anyway, I've done this on a few teams where it just short cuts the whole discussion about needing a bespoke UI to edit some table of stuff for managers. I've even replaced a failed project to build such a UI with simple stuff like this. All you need is a bit of validation when importing and you can keep the last know good export in git and do pull requests to update the file. There's a weird dynamic where giving this level of control to some managers actually makes them feel more involved and engaged. My preferred format to work with is actually ndjson (newline delimited json). Mainly because it's easier to deal with nested objects and lists in that. Whenever people start putting lists in table cells or start adding dots or other separators to their column names to indicate some kind of hierarcy, ndjson is the better solution. I've seen all sorts of attempts by business people to stuff non tabular data into a table.