4 ms·

The header should be optional. That’s a schema specific detail like the number of columns. The character set is also forgivable given the age of the format; it

The header should be optional. That’s a schema specific detail like the number of columns. The character set is also forgivable given the age of the format; it comes from an era when there wasn’t an agreed single character encoding to rule them all. Sure, RFC 4180 probably should have specified UTF-8 but by that point there was already three decades of CSV usage in different encodings on mainframes which are likely still in use even now to make specifying any character encoding rather pointless. Don’t get me wrong, I’m not a massive fan of CSV either. The way quoted new lines are encoded, for example, is horrible and I’ve never liked the “” format for escaping quotation marks. But a lot of CSVs biggest problems are ironically a result of its success: as a format it’s so easy to use that a great many developers can hand crank their own parsers without looking at the spec. So I find it hard being critical about the lack of standardisation in CSV when the real problem is the number of developers who don’t follow the standardisation.

mulmen 6y ago
Even if RFC 4180 was a standard, which isn’t, and everyone in the world strictly adhered to it, it would still be a poor choice for archival purposes.
- foolmeonce 6y ago
  > it would still be a poor choice for archival purposes. It isn't being compared to all formats, it is being compared to sqlite. Some random version of one implementation written in C, stored in "fossil" may be good enough for the library of Congress, but that kind of nonsense went too far for even browser makers to take seriously. Meanwhile it is trivial to read and infer data from formats like CSV to either use directly or automate a reading process with any tools that exist now or in the future.
  - foolmeonce 6y ago
    Perhaps I should have cited the official reference which makes it clear that the viewpoint in this article was at best too optimistic: https://www.w3.org/TR/webdatabase/ https://www.w3.org/TR/webdatabase/
- laumars 6y ago
  > Even if RFC 4180 was a standard, which isn’t It is an informal. RFC just says it's not defining the Internet standard. The reason being, and as I'd already pointed out, CSV predates RFC 4180 by 3 decades so what the RFC is really setting out is to describe what those established conventions are and what the MIME type should be. That doesn't mean that what the RFC describes isn't already a convention (it's the same specifications as published by IBM, W3C and many other big hitters). > it would still be a poor choice for archival purposes. I hadn't suggested it was a good choice for archival purposes. You're building a straw man argument there. My point was that there is a conventional standard to CSV and that is very well documented. The issue with CSV is that it's so easy to write a parser that some developers do so without bothering to read any documentation on how CSV should be parsed. That's not the fault of CSV, that's the fault of lazy developers. But I do completely agree that CSV has other faults (which I had also discussed too).
geocar 6y ago
> the real problem is the number of developers who don’t follow the standardisation. RFC4180 isn't an Internet standard. Says so right at the top in the first paragraph. I think I have a few thousand files on my computer right now whose names end in ".csv", and I'll bet money not one of them agrees with RFC4180 to the letter except by accident of the data itself, and that's a Real Problem to me. I can concede that two parties could agree to interchange according to RFC4180, but as a general format I maintain that for archival and interchange purposes CSV cannot be divorced from the rather complex schema I alluded to without data loss.
- laumars 6y ago
  > RFC4180 isn't an Internet standard. Says so right at the top in the first paragraph. I didn't say it was an Internet standard. I said standardised. Ok, I'll concede it is more of an informal or de facto standard but when IBM, W3C, IETF, OKF and others all publish the same parsing rules for CSV, it's hard to agree when people make statements like "there's no standard in CSV". The problem isn't that there isn't a standard to CSV, the problem is that people often don't follow those conventions. But you have that same problem with other file formats too. > I think I have a few thousand files on my computer right now whose names end in ".csv", and I'll bet money not one of them agrees with RFC4180 to the letter except by accident of the data itself, and that's a Real Problem to me. That's just conjecture. And even if that were proven true, it's still only anecdotal. That said, I do sympathise with your point. But you could make the same argument for - JSON files that don't follow spec (support for comments, aren't UTF-8 encoded, have been manually written so don't follow the escaping rules correctly and thus only parse correctly by chance). - XML files that have been manually cranked and so don't follow schema - HTML documents that don't follow specification and thus browsers do a lot of non-specification interpretation work to render correctly The IT industry is littered with example of people not following the docs. CSV isn't unique in that regard. > I can concede that two parties could agree to interchange according to RFC4180, but as a general format I maintain that for archival and interchange purposes CSV cannot be divorced from the rather complex schema I alluded to without data loss. I wasn't commenting on whether it's a better format than another file format. I was commenting on your points about standardisation saying there are an abundance of published documents on how to read and write a standard CSV file and C-style escaping isn't part of that specification.

mulmen 6y ago

Even if RFC 4180 was a standard, which isn’t, and everyone in the world strictly adhered to it, it would still be a poor choice for archival purposes.

foolmeonce 6y ago
> it would still be a poor choice for archival purposes. It isn't being compared to all formats, it is being compared to sqlite. Some random version of one implementation written in C, stored in "fossil" may be good enough for the library of Congress, but that kind of nonsense went too far for even browser makers to take seriously. Meanwhile it is trivial to read and infer data from formats like CSV to either use directly or automate a reading process with any tools that exist now or in the future.
- foolmeonce 6y ago
  Perhaps I should have cited the official reference which makes it clear that the viewpoint in this article was at best too optimistic: https://www.w3.org/TR/webdatabase/ https://www.w3.org/TR/webdatabase/
laumars 6y ago
> Even if RFC 4180 was a standard, which isn’t It is an informal. RFC just says it's not defining the Internet standard. The reason being, and as I'd already pointed out, CSV predates RFC 4180 by 3 decades so what the RFC is really setting out is to describe what those established conventions are and what the MIME type should be. That doesn't mean that what the RFC describes isn't already a convention (it's the same specifications as published by IBM, W3C and many other big hitters). > it would still be a poor choice for archival purposes. I hadn't suggested it was a good choice for archival purposes. You're building a straw man argument there. My point was that there is a conventional standard to CSV and that is very well documented. The issue with CSV is that it's so easy to write a parser that some developers do so without bothering to read any documentation on how CSV should be parsed. That's not the fault of CSV, that's the fault of lazy developers. But I do completely agree that CSV has other faults (which I had also discussed too).

geocar 6y ago

> the real problem is the number of developers who don’t follow the standardisation. RFC4180 isn't an Internet standard. Says so right at the top in the first paragraph. I think I have a few thousand files on my computer right now whose names end in ".csv", and I'll bet money not one of them agrees with RFC4180 to the letter except by accident of the data itself, and that's a Real Problem to me. I can concede that two parties could agree to interchange according to RFC4180, but as a general format I maintain that for archival and interchange purposes CSV cannot be divorced from the rather complex schema I alluded to without data loss.

laumars 6y ago
> RFC4180 isn't an Internet standard. Says so right at the top in the first paragraph. I didn't say it was an Internet standard. I said standardised. Ok, I'll concede it is more of an informal or de facto standard but when IBM, W3C, IETF, OKF and others all publish the same parsing rules for CSV, it's hard to agree when people make statements like "there's no standard in CSV". The problem isn't that there isn't a standard to CSV, the problem is that people often don't follow those conventions. But you have that same problem with other file formats too. > I think I have a few thousand files on my computer right now whose names end in ".csv", and I'll bet money not one of them agrees with RFC4180 to the letter except by accident of the data itself, and that's a Real Problem to me. That's just conjecture. And even if that were proven true, it's still only anecdotal. That said, I do sympathise with your point. But you could make the same argument for - JSON files that don't follow spec (support for comments, aren't UTF-8 encoded, have been manually written so don't follow the escaping rules correctly and thus only parse correctly by chance). - XML files that have been manually cranked and so don't follow schema - HTML documents that don't follow specification and thus browsers do a lot of non-specification interpretation work to render correctly The IT industry is littered with example of people not following the docs. CSV isn't unique in that regard. > I can concede that two parties could agree to interchange according to RFC4180, but as a general format I maintain that for archival and interchange purposes CSV cannot be divorced from the rather complex schema I alluded to without data loss. I wasn't commenting on whether it's a better format than another file format. I was commenting on your points about standardisation saying there are an abundance of published documents on how to read and write a standard CSV file and C-style escaping isn't part of that specification.