3 ms·

>...then that mainframe outputs data following a different document format, one that doesn't comply with the language defined by the standard. Yes, and so what

by jasode 9y ago

>...then that mainframe outputs data following a different document format, one that doesn't comply with the language defined by the standard. Yes, and so what? Stating RFC4180 as a retroactive fact isn't going to make the job get done. Again, look at the date of RFC4180.[1] It is October 2005. There have been millions of csv files made for decades before that pseudo-standard even existed. (Yes, RFC4180 is not even a formal specification -- it is a pseudo-specification trying to codify the most prevalent and sane existing practices.) The programmer still has to parse all the bad pre-2005 csv files so making the parsing dependent on RFC4180 is irrelevant. Even today, the Citi bank cc transactions export to csv is _not_ RFC4180 compliant. So what? I still have to parse it. Getting pedantic about noncompliant csv files not following RFC4180 is not productive for the task at hand. The programmer trying to parse them doesn't control the other entities that produce bad csv files. >Is it that hard? The conversation is not about difficulty. It is about where in the workflow a pseudo-standard like RFC4180 is most applicable: the writing of output not parsing of input. [1] https://tools.ietf.org/html/rfc4180 https://tools.ietf.org/html/rfc4180

burntsushi 9y ago
Agreed. I think this thread can be summarized simply as: "In theory, theory and practice are the same. In practice, they're completely different." In theory, adhering to the CSV specification strictly is the Right Thing to Do. It's simple and makes it easy to confirm that the data you're parsing is correct. But in practice, that all goes out the window. There are few things worse than running your CSV data through the standard parser and discovering that a smattering of fields contain stray quotes. And what do you get for that? The entire parser falls over and you can't do anything until you go in and fix the data. Or, if we're to believe other users in this thread, until someone goes and writes a specialty parser just for this task to fix the data. In practice, CSV data is messy and therefore a strictly conforming parser tends to be incredibly annoying to use. IMHO, the authors of RFC 4180 failed to recognize this. As a result, most people just ignore it.
geezerjay 9y ago
> The conversation is not about difficulty. It is about where in the workflow a pseudo-standard like RFC4180 is most applicable: Exactly, and obviously it's in writing a parser. Outputting CSV files is essentially a call to printf in a for loop. If a string needs escaping, the string is escaped. It's as simple as it gets. It only gets complicated when you need to write a parser and handle stuff such as how to parse language violations.
- jasode 9y ago
  >Exactly, and obviously it's in writing a parser. No, you've got that wrong but I think I see the underlying issue. You believe that the suggested RFC4180 by is a formal language spec such that you have an exact BNF grammar. Therefore, parsing is a binary result: csv validity is either true or false. This is why you believe RFC4180 is a guide to parsing. But this is wrong. Unfortunately, the evolution of csv isn't like languages with official specs such as ISO C++ standard, or ECMAScript, or HTML. > and handle stuff such as how to parse language violations. ...which renders RFC4180 useless for real-world parsing. The supreme authority of usefully parsing csv is the actual csv file you have to deal with. That file supersedes anything RFC4180 says. Why? Because business people want the data in those broken and non-compliant csv files! This is the opposite situation with parsing source code like C++ and Javascript: the broken source doesn't dictate the correct parsing, the language spec does. With csv, the data itself (broken or not) is what drives the parsing. There is no "true/false compliant" parse. Instead, assume every csv row with incorrect commas and double quotes is parseable and write the parser to handle it no matter what RFC4180 says. An "error in csv" is not the same treatment as an "error in C/C++ source file". Since programmers have to overrule RFC4180 to get useful data, the spec can't be obeyed for real-world parsing. The permissive-parsers are more useful than RFC4180-parsers that don't know how to handle broken csv. This makes RFC4180 much more realistic to guide the output of csv. This is especially true if the programmer is aware of that pseudo-specification after October 2005. This is why you have RFC4180's usefulness in the workflow backwards. An RFC4180-compliant-writer is much more useful than an RFC4180-compliant-reader. There are csv files that MS Excel can successfully import that a strict RFC4180-parser cannot handle.

by jasode 9y ago

burntsushi 9y ago

Agreed. I think this thread can be summarized simply as: "In theory, theory and practice are the same. In practice, they're completely different." In theory, adhering to the CSV specification strictly is the Right Thing to Do. It's simple and makes it easy to confirm that the data you're parsing is correct. But in practice, that all goes out the window. There are few things worse than running your CSV data through the standard parser and discovering that a smattering of fields contain stray quotes. And what do you get for that? The entire parser falls over and you can't do anything until you go in and fix the data. Or, if we're to believe other users in this thread, until someone goes and writes a specialty parser just for this task to fix the data. In practice, CSV data is messy and therefore a strictly conforming parser tends to be incredibly annoying to use. IMHO, the authors of RFC 4180 failed to recognize this. As a result, most people just ignore it.

geezerjay 9y ago

> The conversation is not about difficulty. It is about where in the workflow a pseudo-standard like RFC4180 is most applicable: Exactly, and obviously it's in writing a parser. Outputting CSV files is essentially a call to printf in a for loop. If a string needs escaping, the string is escaped. It's as simple as it gets. It only gets complicated when you need to write a parser and handle stuff such as how to parse language violations.

jasode 9y ago
>Exactly, and obviously it's in writing a parser. No, you've got that wrong but I think I see the underlying issue. You believe that the suggested RFC4180 by is a formal language spec such that you have an exact BNF grammar. Therefore, parsing is a binary result: csv validity is either true or false. This is why you believe RFC4180 is a guide to parsing. But this is wrong. Unfortunately, the evolution of csv isn't like languages with official specs such as ISO C++ standard, or ECMAScript, or HTML. > and handle stuff such as how to parse language violations. ...which renders RFC4180 useless for real-world parsing. The supreme authority of usefully parsing csv is the actual csv file you have to deal with. That file supersedes anything RFC4180 says. Why? Because business people want the data in those broken and non-compliant csv files! This is the opposite situation with parsing source code like C++ and Javascript: the broken source doesn't dictate the correct parsing, the language spec does. With csv, the data itself (broken or not) is what drives the parsing. There is no "true/false compliant" parse. Instead, assume every csv row with incorrect commas and double quotes is parseable and write the parser to handle it no matter what RFC4180 says. An "error in csv" is not the same treatment as an "error in C/C++ source file". Since programmers have to overrule RFC4180 to get useful data, the spec can't be obeyed for real-world parsing. The permissive-parsers are more useful than RFC4180-parsers that don't know how to handle broken csv. This makes RFC4180 much more realistic to guide the output of csv. This is especially true if the programmer is aware of that pseudo-specification after October 2005. This is why you have RFC4180's usefulness in the workflow backwards. An RFC4180-compliant-writer is much more useful than an RFC4180-compliant-reader. There are csv files that MS Excel can successfully import that a strict RFC4180-parser cannot handle.