3 ms·

If csv files bring criminal liability then I am guilty. Sidenote: this particular criminal conspiracy is open to potential new members. Please join the Commit

If csv files bring criminal liability then I am guilty. Sidenote: this particular criminal conspiracy is open to potential new members. Please join the Committee To Keep Csv Evil: https://discord.gg/uqu4BkNP5G https://discord.gg/uqu4BkNP5G Jokes aside, talking about the future is grand but the problem is that data was written in the past and we need to read it in the present. That means that you do have to detect encoding and you can't just decide that the world runs on UTF-8. Even today, mainland China does not use UTF-8 and is not, as far as I know, in the process of switching. I understand UTF-8 is mostly fine even for east asian languages though - and bytes are cheap

mikhailfranco 2y ago
I explained how old files could be read. Let me reframe it as a Schelling Point [1] - the uncoordinated coordination problem. You arrange to meet your file on a certain day in New York, no place or time were mentioned. When and where will you go? It seems impossible. But perhaps you go at noon, to the UTF-8 Building in midtown Manhattan. Are you there now? [1] https://en.wikipedia.org/wiki/Focal_point_(game_theory) https://en.wikipedia.org/wiki/Focal_point_(game_theory)
- calpaterson 2y ago
  I'm sorry, quite right - you did. Using the computer locale. The trouble is that people transmit data from one computer to another and so from one locale to another. And sadly, they do not always set the character encoding header correctly, if they even know what that is. I mean take csvbase's case. It has to accept csv files from anyone. And christ preserve me, they aren't going to label them as "Win-1252" or "UTF-16" or whatever. There is no alternative but statistical detection. And there is good evidence that this solution is fairly satisfactory, because millions of computers are using it right now! csvbase uploads run into more problems with trailing commas than with character detection getting it wrong at this point, that is your "Schelling Point" I'm afraid!
  - mikhailfranco 2y ago
    So trailing commas are another failure. That issue is mildly entangled with headers. Perhaps start making a list...
  - mikhailfranco 2y ago
    HTTP actually does quite a good job of providing headers containing MIME type and encoding. There is a little work to get the default (e.g. HTML and XML are different), and decide on the case where the XML payload encoding is different to the HTTP transport encoding (e.g. perhaps XML parsers need a way to override the embedded header). So we end up at another plausible future-directed design decision: computer-computer communication should use HTTP. I think many systems have ended up there already, perhaps prompted by the issues we have discussed. Moral: good specs attract usage; bad, incomprehensible or inconsistently implemented specs fade away.
mikhailfranco 2y ago
Old CSV files bring no stigma. There is no CSV specification. That does bring opprobrium. RFC4180 is from 2005, long after Unicode and XML, so people should have known better. The absence of a standard points to disagreement, perhaps commercial disagreement, but IETF should be independent, is it not? That failure to standardize encoding, and other issues (headers, etc.), has wasted an enormous amount of time for many creative hackers, who could have been producing value, rather than banging their head against a vague assembly of pseudo-spec-illusions. Me included, sadly.
- eduction 2y ago
  You're blaming the lack of a spec for all the wasted time but that's not the cause. The cause is that CSV is popular and is popular because it is incompletely defined. (See also: HTML, RSS.) Making a CSV spec post-hoc solves nothing, as others here have pointed out, because there is already an "installed base" so to speak. If anything it's worse because it might mislead some people into thinking they can write to the spec and handle any CSV file. The right move, if you want a nicely precise and strict spec, is to admit it's a new thing and give it a new name, maybe CSVS or something like that. But good luck making it popular - there are plenty of CSV libraries out there that handle most CSV files well enough, just as there is tons of software that handles HTML and RSS well enough (which is why I am a fan of both those formats).
  - bee_rider 2y ago
    The only argument I’ll present for giving over CSV to those who want to flail at the idea of standardizing it: the name implies too much standardization already. Why “C”SV? Most CSV processing tools accept a delineator, right? They are just whatever separated values, use semicolons or tabs if you want.
  - calpaterson 2y ago
    > The cause is that CSV is popular and is popular because it is incompletely defined. (See also: HTML, RSS.) This is a surprisingly deep thought. I'm going to need a couple of days to reflect on this...
e63f67dd-065b 2y ago
> China does not use UTF-8 and is not, as far as I know, in the process of switching. That’s … not true? Most Chinese software and websites are utf8 by default, and it’s been that way for a while. GBK and her sisters might linger around in legacy formats, but UTF-8 has certainly reached the point of mass adoption from what I can tell.

mikhailfranco 2y ago

I explained how old files could be read. Let me reframe it as a Schelling Point [1] - the uncoordinated coordination problem. You arrange to meet your file on a certain day in New York, no place or time were mentioned. When and where will you go? It seems impossible. But perhaps you go at noon, to the UTF-8 Building in midtown Manhattan. Are you there now? [1] https://en.wikipedia.org/wiki/Focal_point_(game_theory) https://en.wikipedia.org/wiki/Focal_point_(game_theory)

calpaterson 2y ago
I'm sorry, quite right - you did. Using the computer locale. The trouble is that people transmit data from one computer to another and so from one locale to another. And sadly, they do not always set the character encoding header correctly, if they even know what that is. I mean take csvbase's case. It has to accept csv files from anyone. And christ preserve me, they aren't going to label them as "Win-1252" or "UTF-16" or whatever. There is no alternative but statistical detection. And there is good evidence that this solution is fairly satisfactory, because millions of computers are using it right now! csvbase uploads run into more problems with trailing commas than with character detection getting it wrong at this point, that is your "Schelling Point" I'm afraid!
- mikhailfranco 2y ago
  So trailing commas are another failure. That issue is mildly entangled with headers. Perhaps start making a list...
- mikhailfranco 2y ago
  HTTP actually does quite a good job of providing headers containing MIME type and encoding. There is a little work to get the default (e.g. HTML and XML are different), and decide on the case where the XML payload encoding is different to the HTTP transport encoding (e.g. perhaps XML parsers need a way to override the embedded header). So we end up at another plausible future-directed design decision: computer-computer communication should use HTTP. I think many systems have ended up there already, perhaps prompted by the issues we have discussed. Moral: good specs attract usage; bad, incomprehensible or inconsistently implemented specs fade away.

mikhailfranco 2y ago

Old CSV files bring no stigma. There is no CSV specification. That does bring opprobrium. RFC4180 is from 2005, long after Unicode and XML, so people should have known better. The absence of a standard points to disagreement, perhaps commercial disagreement, but IETF should be independent, is it not? That failure to standardize encoding, and other issues (headers, etc.), has wasted an enormous amount of time for many creative hackers, who could have been producing value, rather than banging their head against a vague assembly of pseudo-spec-illusions. Me included, sadly.

eduction 2y ago
You're blaming the lack of a spec for all the wasted time but that's not the cause. The cause is that CSV is popular and is popular because it is incompletely defined. (See also: HTML, RSS.) Making a CSV spec post-hoc solves nothing, as others here have pointed out, because there is already an "installed base" so to speak. If anything it's worse because it might mislead some people into thinking they can write to the spec and handle any CSV file. The right move, if you want a nicely precise and strict spec, is to admit it's a new thing and give it a new name, maybe CSVS or something like that. But good luck making it popular - there are plenty of CSV libraries out there that handle most CSV files well enough, just as there is tons of software that handles HTML and RSS well enough (which is why I am a fan of both those formats).
- bee_rider 2y ago
  The only argument I’ll present for giving over CSV to those who want to flail at the idea of standardizing it: the name implies too much standardization already. Why “C”SV? Most CSV processing tools accept a delineator, right? They are just whatever separated values, use semicolons or tabs if you want.
- calpaterson 2y ago
  > The cause is that CSV is popular and is popular because it is incompletely defined. (See also: HTML, RSS.) This is a surprisingly deep thought. I'm going to need a couple of days to reflect on this...

e63f67dd-065b 2y ago

> China does not use UTF-8 and is not, as far as I know, in the process of switching. That’s … not true? Most Chinese software and websites are utf8 by default, and it’s been that way for a while. GBK and her sisters might linger around in legacy formats, but UTF-8 has certainly reached the point of mass adoption from what I can tell.