4 ms·

I can't see this gaining any traction. There's nothing wrong with CSV as it is for basic data transfer. CSV had been unchanged now for so long, that any attempt

I can't see this gaining any traction. There's nothing wrong with CSV as it is for basic data transfer. CSV had been unchanged now for so long, that any attempt to update it will fail - there are simply too many CSV parsers already implemented. If you are worried about people in the year 3000 understanding your data, add a .txt file alongside the CSV explaining the fields.

Lazare 8y ago
> There's nothing wrong with CSV as it is for basic data transfer. I disagree. CSV is horribly underspecified and many parsers have conflicting ideas on how things should work. I've hit areas where, for example, an API was generating a CSV that another API could read, but NOT if it was opened and then re-saved by Excel first (just re-saving, not editing), because Excel was making some trivial change in the format that the consuming parser couldn't handle. Then there's issues with encoding, localisation, and escaping (see, eg, https://chriswarrick.com/blog/2017/04/07/csv-is-not-a-standard/ https://chriswarrick.com/blog/2017/04/07/csv-is-not-a-standa... although that's just scratching the surface). That being said, this proposal seems worse in every way that existing CSVs, but I don't think the issue is "there's a well specified CSV standard with tons of parsers that won't change to support this new one", it's more that there is no standard, all the parsers are incompatible, and no meaningful improvements are possible. :(
- hodgesrm 8y ago
  To your point it's hard to see how the referenced article qualifies as a standard. It does not deal with escape characters or missing values, two places where CSV implementations tend to vary in random ways.
  - Lazare 8y ago
    My comment may have been unclear, but I agree with this completely.
  - wlesieutre 8y ago
    I've recently been dealing with some CSV data exported from Excel. One of the columns has lengths measured in inches or feet. A column that says 14" exports as , '"14"""', It feels downright silly to me.
    - deathanatos 8y ago
      A cell w/ 14" would be ,"14""", And I verified that all four of Excel's CSV varieties write this, but this is also what RFC CSV writers would emit. Not saying you can't pick up oddities elsewhere, but as stated, it's not that bad. (Not that I condone use of CSV… it's an awful format.)
      - wlesieutre 8y ago
        Ah you're right, the spot I was looking at was after having read it into Python and split into a list on the commas, so the outer ' is my string container. Whoops.
- ken 8y ago
  What we need, I think, is an HTML5-like effort from the industry. That's the one format I can think of which went from "complete mess" to "parses the same everywhere". 1. Look at how everybody is extending CSV in the real world 2. Make a new format which does those things, as closely to all the existing implementations as is reasonable 3. Write a fully-specified new syntax for it, including a complete parser (state machine including error cases) 4. Make a complete set of tests, which make it very obvious when you don't pass, for shaming everyone into compliance 5. Get buy-in from a major standards organization, and all of the major corporate players I don't see any improvements/replacements succeeding if they don't hit all of these points. "CSV 1.1" here hits #1 and #2 only.
Someone1234 8y ago
I agree. The biggest problem with CSV is Excel's terrible support. And while CSV is primarily used for data transfer, Excel could hypothetically be a great debug-viewer if it didn't do brain dead stuff like unrecoverably corrupt UPCs and other long numbers by default. There's an Excel Uservoice about this very issue from 2015: https://excel.uservoice.com/forums/304921-excel-for-windows-desktop-application/suggestions/10374741-stop-excel-from-changing-large-numbers-actually?page=8&per_page=20 https://excel.uservoice.com/forums/304921-excel-for-windows-... They're "looking into it." Just as they've been looking into it for the last twenty years. Any. Day. Now.
- infogulch 8y ago
  Excel's braindead csv corruption is why I never use it for csv now. Add guids to the list of things it can corrupt. (if the first group is numeric it parses as an expression which then causes an error because the next group has hex digits, what!?) Also any long string that happens to only be digits. And two numbers separated by a "/". It's so bad I check any csv file I get if it's been opened in Excel, and if I find any evidence that it has I reject it and make whoever I got it from give me the original source, or transform it with literally any other tool.
- deleted 8y ago
  [deleted]
- PakG1 8y ago
  I was surprised by how bad Microsoft's CSV support is in Excel. So much that I use Numbers or text editors for everything CSV, never do I use Excel. Excel is still great for a basic level of intensive data analysis though, so can't get away from it.
- paulryanrogers 8y ago
  IIRC there is a hack where you put a single quote after the comma and Excel will treat it as a string. Unfortunately it doesn't help exporting to unknown systems which l likely don't do the same.
- tomc1985 8y ago
  Fix it so now its easier for customers to not use Excel? Heretic!
geraldbauer 8y ago
> If you are worried about people in the year 3000 understanding your data, add a .txt file alongside the CSV explaining the fields. Good point. That's what (tabular) data packages are for, see https://github.com/csv11/csvpack https://github.com/csv11/csvpack for some real-world examples.

Lazare 8y ago

> There's nothing wrong with CSV as it is for basic data transfer. I disagree. CSV is horribly underspecified and many parsers have conflicting ideas on how things should work. I've hit areas where, for example, an API was generating a CSV that another API could read, but NOT if it was opened and then re-saved by Excel first (just re-saving, not editing), because Excel was making some trivial change in the format that the consuming parser couldn't handle. Then there's issues with encoding, localisation, and escaping (see, eg, https://chriswarrick.com/blog/2017/04/07/csv-is-not-a-standard/ https://chriswarrick.com/blog/2017/04/07/csv-is-not-a-standa... although that's just scratching the surface). That being said, this proposal seems worse in every way that existing CSVs, but I don't think the issue is "there's a well specified CSV standard with tons of parsers that won't change to support this new one", it's more that there is no standard, all the parsers are incompatible, and no meaningful improvements are possible. :(

hodgesrm 8y ago
To your point it's hard to see how the referenced article qualifies as a standard. It does not deal with escape characters or missing values, two places where CSV implementations tend to vary in random ways.
- Lazare 8y ago
  My comment may have been unclear, but I agree with this completely.
- wlesieutre 8y ago
  I've recently been dealing with some CSV data exported from Excel. One of the columns has lengths measured in inches or feet. A column that says 14" exports as , '"14"""', It feels downright silly to me.
  - deathanatos 8y ago
    A cell w/ 14" would be ,"14""", And I verified that all four of Excel's CSV varieties write this, but this is also what RFC CSV writers would emit. Not saying you can't pick up oddities elsewhere, but as stated, it's not that bad. (Not that I condone use of CSV… it's an awful format.)
    - wlesieutre 8y ago
      Ah you're right, the spot I was looking at was after having read it into Python and split into a list on the commas, so the outer ' is my string container. Whoops.
ken 8y ago
What we need, I think, is an HTML5-like effort from the industry. That's the one format I can think of which went from "complete mess" to "parses the same everywhere". 1. Look at how everybody is extending CSV in the real world 2. Make a new format which does those things, as closely to all the existing implementations as is reasonable 3. Write a fully-specified new syntax for it, including a complete parser (state machine including error cases) 4. Make a complete set of tests, which make it very obvious when you don't pass, for shaming everyone into compliance 5. Get buy-in from a major standards organization, and all of the major corporate players I don't see any improvements/replacements succeeding if they don't hit all of these points. "CSV 1.1" here hits #1 and #2 only.

Someone1234 8y ago

I agree. The biggest problem with CSV is Excel's terrible support. And while CSV is primarily used for data transfer, Excel could hypothetically be a great debug-viewer if it didn't do brain dead stuff like unrecoverably corrupt UPCs and other long numbers by default. There's an Excel Uservoice about this very issue from 2015: https://excel.uservoice.com/forums/304921-excel-for-windows-desktop-application/suggestions/10374741-stop-excel-from-changing-large-numbers-actually?page=8&per_page=20 https://excel.uservoice.com/forums/304921-excel-for-windows-... They're "looking into it." Just as they've been looking into it for the last twenty years. Any. Day. Now.

infogulch 8y ago
Excel's braindead csv corruption is why I never use it for csv now. Add guids to the list of things it can corrupt. (if the first group is numeric it parses as an expression which then causes an error because the next group has hex digits, what!?) Also any long string that happens to only be digits. And two numbers separated by a "/". It's so bad I check any csv file I get if it's been opened in Excel, and if I find any evidence that it has I reject it and make whoever I got it from give me the original source, or transform it with literally any other tool.
deleted 8y ago
[deleted]
PakG1 8y ago
I was surprised by how bad Microsoft's CSV support is in Excel. So much that I use Numbers or text editors for everything CSV, never do I use Excel. Excel is still great for a basic level of intensive data analysis though, so can't get away from it.
paulryanrogers 8y ago
IIRC there is a hack where you put a single quote after the comma and Excel will treat it as a string. Unfortunately it doesn't help exporting to unknown systems which l likely don't do the same.
tomc1985 8y ago
Fix it so now its easier for customers to not use Excel? Heretic!

geraldbauer 8y ago

> If you are worried about people in the year 3000 understanding your data, add a .txt file alongside the CSV explaining the fields. Good point. That's what (tabular) data packages are for, see https://github.com/csv11/csvpack https://github.com/csv11/csvpack for some real-world examples.