5 ms·

> RFC4180 is more helpful to generate correct CSV files. It's not as useful for parsing files You've got it all backwards. The people writing parsers are the

> RFC4180 is more helpful to generate correct CSV files. It's not as useful for parsing files You've got it all backwards. The people writing parsers are the ones who need a language specification to decide what to parse, what to ignore, and what to consider errors. Meanwhile, writing CSV docs is just a for loop with a printf. That's it.

jasode 9y ago
>The people writing parsers are the ones who need a language specification to decide what to parse, If an old 1970s mainframe is spitting out "broken" csv files that you need to parse, a new 2005 csv formal specification trying to impose some sanity to the landscape is irrelevant. That short RFC4180 tries to constrain what is "correct". It can't enumerate all the ways csv files are "incorrect".[1] The programmer has to look at the actual bad csv files he's trying to parse. As of today, Citi bank still exports broken csv for credit-card transactions. (Citi inserts extra CRLF in between quotes which breaks MS Excel csv import.) I don't need a formal spec telling me how it's broken. It's quite obvious by simply viewing it in Notepad. >Meanwhile, writing CSV docs is just a for loop with a printf. That's it. No. For example, if the field has a movie title with embedded quotes[2] or maybe a measurement using a one double quote for inches (e.g. [Sony 4k TV 65"])[3], using a naïve printf loop without generating extra escape characters will break many csv parsers that try to conform to RFC4180. [1] although someone did try to enumerate bad csv files: https://donatstudios.com/Falsehoods-Programmers-Believe-About-CSVs https://donatstudios.com/Falsehoods-Programmers-Believe-Abou... [2] example movie title [Precious: Based on the Novel "Push" by Sapphire] : https://en.wikipedia.org/wiki/Precious_(film) https://en.wikipedia.org/wiki/Precious_(film) [3] https://stackoverflow.com/questions/17808511/properly-escape-a-double-quote-in-csv https://stackoverflow.com/questions/17808511/properly-escape...
- jononor 9y ago
  Make the parser RFC compliant. Use a separate tool for normalizing the broken cave files to RFC compliant.
- geezerjay 9y ago
  > If an old 1970s mainframe is spitting out "broken" csv files ...then that mainframe outputs data following a different document format, one that doesn't comply with the language defined by the standard. If you need to parse a language, you target a language. You can't simply write a document in english and call it mandarin. > No. For example, if the field has a movie title with embedded quotes[2] ...then you escape the quotes, and proceed to printf each field. Is it that hard?
  - jasode 9y ago
    >...then that mainframe outputs data following a different document format, one that doesn't comply with the language defined by the standard. Yes, and so what? Stating RFC4180 as a retroactive fact isn't going to make the job get done. Again, look at the date of RFC4180.[1] It is October 2005. There have been millions of csv files made for decades before that pseudo-standard even existed. (Yes, RFC4180 is not even a formal specification -- it is a pseudo-specification trying to codify the most prevalent and sane existing practices.) The programmer still has to parse all the bad pre-2005 csv files so making the parsing dependent on RFC4180 is irrelevant. Even today, the Citi bank cc transactions export to csv is _not_ RFC4180 compliant. So what? I still have to parse it. Getting pedantic about noncompliant csv files not following RFC4180 is not productive for the task at hand. The programmer trying to parse them doesn't control the other entities that produce bad csv files. >Is it that hard? The conversation is not about difficulty. It is about where in the workflow a pseudo-standard like RFC4180 is most applicable: the writing of output not parsing of input. [1] https://tools.ietf.org/html/rfc4180 https://tools.ietf.org/html/rfc4180
    - burntsushi 9y ago
      Agreed. I think this thread can be summarized simply as: "In theory, theory and practice are the same. In practice, they're completely different." In theory, adhering to the CSV specification strictly is the Right Thing to Do. It's simple and makes it easy to confirm that the data you're parsing is correct. But in practice, that all goes out the window. There are few things worse than running your CSV data through the standard parser and discovering that a smattering of fields contain stray quotes. And what do you get for that? The entire parser falls over and you can't do anything until you go in and fix the data. Or, if we're to believe other users in this thread, until someone goes and writes a specialty parser just for this task to fix the data. In practice, CSV data is messy and therefore a strictly conforming parser tends to be incredibly annoying to use. IMHO, the authors of RFC 4180 failed to recognize this. As a result, most people just ignore it.

jasode 9y ago

>The people writing parsers are the ones who need a language specification to decide what to parse, If an old 1970s mainframe is spitting out "broken" csv files that you need to parse, a new 2005 csv formal specification trying to impose some sanity to the landscape is irrelevant. That short RFC4180 tries to constrain what is "correct". It can't enumerate all the ways csv files are "incorrect".[1] The programmer has to look at the actual bad csv files he's trying to parse. As of today, Citi bank still exports broken csv for credit-card transactions. (Citi inserts extra CRLF in between quotes which breaks MS Excel csv import.) I don't need a formal spec telling me how it's broken. It's quite obvious by simply viewing it in Notepad. >Meanwhile, writing CSV docs is just a for loop with a printf. That's it. No. For example, if the field has a movie title with embedded quotes[2] or maybe a measurement using a one double quote for inches (e.g. [Sony 4k TV 65"])[3], using a naïve printf loop without generating extra escape characters will break many csv parsers that try to conform to RFC4180. [1] although someone did try to enumerate bad csv files: https://donatstudios.com/Falsehoods-Programmers-Believe-About-CSVs https://donatstudios.com/Falsehoods-Programmers-Believe-Abou... [2] example movie title [Precious: Based on the Novel "Push" by Sapphire] : https://en.wikipedia.org/wiki/Precious_(film) https://en.wikipedia.org/wiki/Precious_(film) [3] https://stackoverflow.com/questions/17808511/properly-escape-a-double-quote-in-csv https://stackoverflow.com/questions/17808511/properly-escape...

jononor 9y ago
Make the parser RFC compliant. Use a separate tool for normalizing the broken cave files to RFC compliant.
geezerjay 9y ago
> If an old 1970s mainframe is spitting out "broken" csv files ...then that mainframe outputs data following a different document format, one that doesn't comply with the language defined by the standard. If you need to parse a language, you target a language. You can't simply write a document in english and call it mandarin. > No. For example, if the field has a movie title with embedded quotes[2] ...then you escape the quotes, and proceed to printf each field. Is it that hard?
- jasode 9y ago
  >...then that mainframe outputs data following a different document format, one that doesn't comply with the language defined by the standard. Yes, and so what? Stating RFC4180 as a retroactive fact isn't going to make the job get done. Again, look at the date of RFC4180.[1] It is October 2005. There have been millions of csv files made for decades before that pseudo-standard even existed. (Yes, RFC4180 is not even a formal specification -- it is a pseudo-specification trying to codify the most prevalent and sane existing practices.) The programmer still has to parse all the bad pre-2005 csv files so making the parsing dependent on RFC4180 is irrelevant. Even today, the Citi bank cc transactions export to csv is _not_ RFC4180 compliant. So what? I still have to parse it. Getting pedantic about noncompliant csv files not following RFC4180 is not productive for the task at hand. The programmer trying to parse them doesn't control the other entities that produce bad csv files. >Is it that hard? The conversation is not about difficulty. It is about where in the workflow a pseudo-standard like RFC4180 is most applicable: the writing of output not parsing of input. [1] https://tools.ietf.org/html/rfc4180 https://tools.ietf.org/html/rfc4180
  - burntsushi 9y ago
    Agreed. I think this thread can be summarized simply as: "In theory, theory and practice are the same. In practice, they're completely different." In theory, adhering to the CSV specification strictly is the Right Thing to Do. It's simple and makes it easy to confirm that the data you're parsing is correct. But in practice, that all goes out the window. There are few things worse than running your CSV data through the standard parser and discovering that a smattering of fields contain stray quotes. And what do you get for that? The entire parser falls over and you can't do anything until you go in and fix the data. Or, if we're to believe other users in this thread, until someone goes and writes a specialty parser just for this task to fix the data. In practice, CSV data is messy and therefore a strictly conforming parser tends to be incredibly annoying to use. IMHO, the authors of RFC 4180 failed to recognize this. As a result, most people just ignore it.