6 ms·

OP and you gave me an idea : "The only true successor of CSV should be forward/backward compatible with any existing CSV variant" If we manage to write a spec

OP and you gave me an idea : "The only true successor of CSV should be forward/backward compatible with any existing CSV variant" If we manage to write a spec that meet this criteria we'll have a powerful standard with easy adoption.

masklinn 5y ago
Given the infinity of CSV variants existing is the very reason why CSV blows goats, creating a spec compatible with that is 1. a fool's errand, CSV "variants" are not compatible with one another and regularly contradict one another (one needs not look any further than Excel's localised CSVs) 2. resulting in getting CSV anyway, which is a lot of efforts to do nothing
ivanbakel 5y ago
That's ignoring two big points in the OP: that CSV variants are ill-defined and conflicting; and that existing CSV data is too poorly-specified. There is no reason to try to be "backwards-compatible" with existing CSV files - we don't have a single definition of correctness to use to check that the compatibility is correct. Every attempt to be parse existing data would result in unexpected results or even data loss for some CSVs in the wild, because there is no way to reconcile all the different expectations and specifications that people have for their own CSV data.
Macha 5y ago
The issue is CSV is not compatible with CSV. It's not possible to write a spec that covers all CSV files in the world. CSV means things that are mutually incompatible in the less common cases, and the only way to really parse them correctly is to know which variant generated it. But you can't even tag that variant in the file by your criteria, as existing CSV parsers won't understand it.
- Twisell 5y ago
  By this argument UTF8 can't exist. And yet here it is. PS: I never said 100% forward/backward compatible with all variant at the same time and without any noticeable artifact. I meant compatible in a non blocking way.
  - masklinn 5y ago
    What are you talking about? UTF8 is a single well-defined specification, and detecting that data is definitely not UTF8 is trivial.
    - deleted 5y ago
      [deleted]
    - Twisell 5y ago
      And yet it is forward/backward compatible with ASCII and non blocking against all it's ill defined variants.
      - Macha 5y ago
        ASCII was well defined, CSV was not. Therefore they could take the highest bit, which they could know that was unused per the ASCII spec, and use that to encode their extra UTF-8 information. Also UTF-8/ascii compatibility is unidirectional. A tool that understands ASCII is going to print nonsense when it encounters emoji or whatever in UTF-8. Even the idea that tools that only understand ASCII won't mangle UTF-8 is limited - sure dumb passthroughs are fine, but if it manipulates the text at all, then you're out of luck - what does it mean to uppercase the first byte of a flag emoji?
        tsimionescu 5y ago
        To be fair, there is basically no way to manipulate arbitrary text at all without mangling it, UTF-8-aware or not. What does it mean to take the first 7 characters of a UTF-8 string which might contain combinator characters and left-to-right special chars? What if the text uses special shaping chars, such as arranging hieroglyphs in cartouches? You basically need a text-rendering aware library to manipulate arbitrary strings.
- godshatter 5y ago
  Couldn't tools that read CSV files scan them first and see which variant best matches the file? The questions arise about which end of line character(s) are used and how double-quotes and commas are handled. There can't be that many ways to escape them, and there are three sets of end of line characters mentioned - just see which one is used (i.e. don't assume only \n if you run into \r\n or \r alone). The assumption that most software uses is that the import file will be in the same variant of the format as what that tool exports. That seems to be more of a problem than anything else.
  - masklinn 5y ago
    > Couldn't tools that read CSV files scan them first and see which variant best matches the file? Sure, and they sometimes do that if they have to ingest CSVs whose origin they don't control (although not every system implementor cares enough to do it). But that's still just a bunch of shitty faillible heuristics which would not be necessary if the format was not so horrible.
    - Macha 5y ago
      It also doesn't prevent a human or other system doing: cat input1.csv input2.csv > output.csv resulting in a single file containing multiple formats. Also, what variant is this: 1,5,Here is a string "" that does stuff,2021-1-1 What is the value of the third column? Is this a CSV file without quoting? Then it's Here is a string "" that does stuff Or is it a CSV file with double quote escaping? Then it's Here is a string " that does stuff This is fundamentally undecidable without knowledge of what the format it is. You can decide to just assume RFC compliant CSVs in the event of ambiguity, but then you absolutely will get bugs from users with non-RFC compliant CSV files.
      - godshatter 5y ago
        That's true. You could scan the file and see if there are any other types of double quote escaping happening, but if there isn't any that wouldn't help either. It's also negated by the multiple formats in the same file point. So, yeah. Can't really be done without making too many assumptions that will break later.
wernercd 5y ago
obligatory: https://xkcd.com/927/ https://xkcd.com/927/
- sicariusnoctis 5y ago
  This is true. The best way would be if everyone generated CSV files that were compatible with on one or two already existing CSV formats. (e.g. no-quote-plain-old-csv or quotes-forced-csv.) Hopefully, it would be easy to distinguish between the two in some well-defined manner. Maybe call it .csvs (CSV standardized). Or just introduce new extensions like .csvq ("CSV quoted"). Note the difference is that I am suggesting reducing the number of standards in-use by using only one already existing CSV format. :)
dragonwriter 5y ago
> If we manage to write a spec that meet this criteria we'll have a powerful standard with easy adoption. So, a binary format consisting of: (1) a text data segment (2) and end of file character (3) a second text data segment with structured metadata describing the layout of the first text data segment, which can be as simple (in terms of meaning; the structure should be more constrained for machine readability) as “It’s some kind of CSV, yo!” to a description of specific CSV variations (headers? column data types? escaping mechanisms? etc.) or even specify that the main body is JSON, YAML, XML, etc. (which would probably often be detectable by inspection, but this removes any ambiguity).
- Twisell 5y ago
  You got my vibe Almost any current CSV parser, even the bad ones, tolerate a header line. So it should be possible to define a compact and standardized syntax that is appended before the real header of the first cell (separator,encoding,decimal separator (often disregarded by most parsers but crucial outside USA),quote character,escape character,etc...). Following headers would just use special notation to inform on (data-type,length,comment). Newest parsers would use theses clues, older ones would just append some manageable junk to headers.
  - Macha 5y ago
    So someone opens this CSV in Excel and there's garbage in A1? Does this really count as compatible? You will get user bugs for this.
    - dragonwriter 5y ago
      > So someone opens this CSV in Excel and there's garbage in A1? Yeah, that's why I chose the “thing that looks like a text file—including optionally CSV—but has additional metadata after the EOF mark” approach instead of stuffing additional metadata in the CSV; there's no way to guarantee that existing implementations will safely ignore any added metadata the main CSV body. (My mechanism has some risk in that there are probably CSV readers that treat the file as a binary byte stream and use the file size rather than a text stream that ends at EOF, but I expect its far fewer than will do the wrong thing with additional metadata before the first header.

masklinn 5y ago

Given the infinity of CSV variants existing is the very reason why CSV blows goats, creating a spec compatible with that is 1. a fool's errand, CSV "variants" are not compatible with one another and regularly contradict one another (one needs not look any further than Excel's localised CSVs) 2. resulting in getting CSV anyway, which is a lot of efforts to do nothing

ivanbakel 5y ago

That's ignoring two big points in the OP: that CSV variants are ill-defined and conflicting; and that existing CSV data is too poorly-specified. There is no reason to try to be "backwards-compatible" with existing CSV files - we don't have a single definition of correctness to use to check that the compatibility is correct. Every attempt to be parse existing data would result in unexpected results or even data loss for some CSVs in the wild, because there is no way to reconcile all the different expectations and specifications that people have for their own CSV data.

Macha 5y ago

The issue is CSV is not compatible with CSV. It's not possible to write a spec that covers all CSV files in the world. CSV means things that are mutually incompatible in the less common cases, and the only way to really parse them correctly is to know which variant generated it. But you can't even tag that variant in the file by your criteria, as existing CSV parsers won't understand it.

Twisell 5y ago
By this argument UTF8 can't exist. And yet here it is. PS: I never said 100% forward/backward compatible with all variant at the same time and without any noticeable artifact. I meant compatible in a non blocking way.
- masklinn 5y ago
  What are you talking about? UTF8 is a single well-defined specification, and detecting that data is definitely not UTF8 is trivial.
  - deleted 5y ago
    [deleted]
  - Twisell 5y ago
    And yet it is forward/backward compatible with ASCII and non blocking against all it's ill defined variants.
    - Macha 5y ago
      ASCII was well defined, CSV was not. Therefore they could take the highest bit, which they could know that was unused per the ASCII spec, and use that to encode their extra UTF-8 information. Also UTF-8/ascii compatibility is unidirectional. A tool that understands ASCII is going to print nonsense when it encounters emoji or whatever in UTF-8. Even the idea that tools that only understand ASCII won't mangle UTF-8 is limited - sure dumb passthroughs are fine, but if it manipulates the text at all, then you're out of luck - what does it mean to uppercase the first byte of a flag emoji?
      - tsimionescu 5y ago
        To be fair, there is basically no way to manipulate arbitrary text at all without mangling it, UTF-8-aware or not. What does it mean to take the first 7 characters of a UTF-8 string which might contain combinator characters and left-to-right special chars? What if the text uses special shaping chars, such as arranging hieroglyphs in cartouches? You basically need a text-rendering aware library to manipulate arbitrary strings.
godshatter 5y ago
Couldn't tools that read CSV files scan them first and see which variant best matches the file? The questions arise about which end of line character(s) are used and how double-quotes and commas are handled. There can't be that many ways to escape them, and there are three sets of end of line characters mentioned - just see which one is used (i.e. don't assume only \n if you run into \r\n or \r alone). The assumption that most software uses is that the import file will be in the same variant of the format as what that tool exports. That seems to be more of a problem than anything else.
- masklinn 5y ago
  > Couldn't tools that read CSV files scan them first and see which variant best matches the file? Sure, and they sometimes do that if they have to ingest CSVs whose origin they don't control (although not every system implementor cares enough to do it). But that's still just a bunch of shitty faillible heuristics which would not be necessary if the format was not so horrible.
  - Macha 5y ago
    It also doesn't prevent a human or other system doing: cat input1.csv input2.csv > output.csv resulting in a single file containing multiple formats. Also, what variant is this: 1,5,Here is a string "" that does stuff,2021-1-1 What is the value of the third column? Is this a CSV file without quoting? Then it's Here is a string "" that does stuff Or is it a CSV file with double quote escaping? Then it's Here is a string " that does stuff This is fundamentally undecidable without knowledge of what the format it is. You can decide to just assume RFC compliant CSVs in the event of ambiguity, but then you absolutely will get bugs from users with non-RFC compliant CSV files.
    - godshatter 5y ago
      That's true. You could scan the file and see if there are any other types of double quote escaping happening, but if there isn't any that wouldn't help either. It's also negated by the multiple formats in the same file point. So, yeah. Can't really be done without making too many assumptions that will break later.

wernercd 5y ago

obligatory: https://xkcd.com/927/ https://xkcd.com/927/

sicariusnoctis 5y ago
This is true. The best way would be if everyone generated CSV files that were compatible with on one or two already existing CSV formats. (e.g. no-quote-plain-old-csv or quotes-forced-csv.) Hopefully, it would be easy to distinguish between the two in some well-defined manner. Maybe call it .csvs (CSV standardized). Or just introduce new extensions like .csvq ("CSV quoted"). Note the difference is that I am suggesting reducing the number of standards in-use by using only one already existing CSV format. :)

dragonwriter 5y ago

> If we manage to write a spec that meet this criteria we'll have a powerful standard with easy adoption. So, a binary format consisting of: (1) a text data segment (2) and end of file character (3) a second text data segment with structured metadata describing the layout of the first text data segment, which can be as simple (in terms of meaning; the structure should be more constrained for machine readability) as “It’s some kind of CSV, yo!” to a description of specific CSV variations (headers? column data types? escaping mechanisms? etc.) or even specify that the main body is JSON, YAML, XML, etc. (which would probably often be detectable by inspection, but this removes any ambiguity).

Twisell 5y ago
You got my vibe Almost any current CSV parser, even the bad ones, tolerate a header line. So it should be possible to define a compact and standardized syntax that is appended before the real header of the first cell (separator,encoding,decimal separator (often disregarded by most parsers but crucial outside USA),quote character,escape character,etc...). Following headers would just use special notation to inform on (data-type,length,comment). Newest parsers would use theses clues, older ones would just append some manageable junk to headers.
- Macha 5y ago
  So someone opens this CSV in Excel and there's garbage in A1? Does this really count as compatible? You will get user bugs for this.
  - dragonwriter 5y ago
    > So someone opens this CSV in Excel and there's garbage in A1? Yeah, that's why I chose the “thing that looks like a text file—including optionally CSV—but has additional metadata after the EOF mark” approach instead of stuffing additional metadata in the CSV; there's no way to guarantee that existing implementations will safely ignore any added metadata the main CSV body. (My mechanism has some risk in that there are probably CSV readers that treat the file as a binary byte stream and use the file size rather than a text stream that ends at EOF, but I expect its far fewer than will do the wrong thing with additional metadata before the first header.