4 ms·
RFC4180 is more helpful to generate correct CSV files. It's not as useful for parsing files since many (most?) csv writers will not have followed RFC4180 in th
by jasode 9y ago
RFC4180 is more helpful to generate correct CSV files. It's not as useful for parsing files since many (most?) csv writers will not have followed RFC4180 in the first place.
Or to put it differently:
"Be conservative in what you send {use RFC4180!}, be liberal in what you accept {parse broken ambiguous csv that was ignorant of RFC4180}." -- from the https://en.wikipedia.org/wiki/Robustness_principle https://en.wikipedia.org/wiki/Robustness_principle
It also doesn't help that RFC4180 is dated 2005 which is trying to enforce a standard 20 to 30 years after csv files have been created adhoc in the wild without a formal specification.
- pdkl95 9y ago> Robustness_principle A slight variation on this is important for security. Be liberal in what you accept, but define what "liberal" means. It's important to define what counts as a valid input. If you don't have a validation/recognizer verify that the input is acceptable, you are probably creating a "weird machine"[1] waiting to be exploited in ways you didn't expect. I highly recommend Meredith and Sergey's talk[2][3] at 28c3, "The Science of Insecurity", where they explain this from a language-theoretic perspective. [1] http://www.cs.dartmouth.edu/~sergey/wm/ http://www.cs.dartmouth.edu/~sergey/wm/ [2] https://media.ccc.de/v/28c3-4763-en-the_science_of_insecurity https://media.ccc.de/v/28c3-4763-en-the_science_of_insecurit... [3] http://www.cs.dartmouth.edu/~sergey/langsec/insecurity-theory-28c3.pdf http://www.cs.dartmouth.edu/~sergey/langsec/insecurity-theor...
- MichaelGG 9y agoExcept that's not an interpretation of liberal anyone uses. That'd just be "by the spec". The robustness principle in practice has made things a nightmare. Consider SIP and HTTP. SIP explicitly encourages implementers to guess at the meaning of messages, and delights in how complicated their parsing rules are. But even without getting crazy, simple line endings end up being a security issue. Some treat LF like CRLF. Or CRCR as a single line break. So you end up with a setup where your SIP or HTTP proxy will interpret a message one way, and your server interprets it differently. This lets you exploit things by sneaking in headers that your proxy would otherwise deny.
- pdkl95 9y agoThe point is it's important to define a new spec if you are being "liberal" in what you accept. For example, while RFC 4180 defines line endings as CRLF, it might be reasonable to "be liberal" and accept files with just LF if and only if you add a rule similar to <line-ending> ::= "\r\n" | "\n" to your input grammar, and properly recognize that the input matches that grammar before using any parsed values. It's an extension ("being liberal") to the official spec, but still strictly defined in the implemented parser. > This lets you exploit things by sneaking in headers In the talk at my previous [2][3], Serge specifically mentions Travis Goodspeed's packet-in-packet[4] exploit as an example of why you have to define what "liberal" means if you want to implement Postel's principle. (You should also limit input grammar's complexity to deterministic context-free, or the recognizer will be undecidable) > in practice has made things a nightmare That's what [2][3] is trying to fix. Stop using Turing Complete input languages, explicitly define your input grammar no matter what you're choosing to parse, and formally recognize all input against that grammar. The data either matches the defined grammar or it should be failed as invalid. Any allowances for sloppy input must be baked into that process. [4] http://travisgoodspeed.blogspot.com/2011/09/remotely-exploiting-phy-layer.html http://travisgoodspeed.blogspot.com/2011/09/remotely-exploit...
- sevensor 9y agoI've saved myself a number of headaches by taking the opposite approach and being conservative in what I accept: I ignore the quoting rules. Comma or newline in the middle of your field? Malformed. Double-quote by itself in the middle of your field? Fine by me. Most of the weird stuff comes from Excel anyway, and I have a reader for Excel files that tolerates those things just fine. My users are happy with it because I've removed a source of ambiguity for them. My users are also highly technical, so I can get away with this sort of thing.
- burntsushi 9y agoThere's a certain amount of that in my CSV library as well, which is strongly influenced by Python's standard library `csv` behavior. Example: extern crate csv; fn main() { let mut rdr = csv::ReaderBuilder::new() .has_headers(false) .from_reader("abc,foo\"baz,xyz".as_bytes()); println!("{:?}", rdr.records().collect::<Result<Vec<_>, _>>().unwrap()); } Output: [StringRecord(["abc", "foo\"baz", "xyz"])] Python: >>> import csv >>> list(csv.reader(['abc,foo"baz,xyz'])) [['abc', 'foo"baz', 'xyz']] This is what's meant by "prefer a parse over returning an error." I would say that this is the liberal philosophy rather than conservative, though.
- geezerjay 9y ago> RFC4180 is more helpful to generate correct CSV files. It's not as useful for parsing files You've got it all backwards. The people writing parsers are the ones who need a language specification to decide what to parse, what to ignore, and what to consider errors. Meanwhile, writing CSV docs is just a for loop with a printf. That's it.
- jasode 9y ago>The people writing parsers are the ones who need a language specification to decide what to parse, If an old 1970s mainframe is spitting out "broken" csv files that you need to parse, a new 2005 csv formal specification trying to impose some sanity to the landscape is irrelevant. That short RFC4180 tries to constrain what is "correct". It can't enumerate all the ways csv files are "incorrect".[1] The programmer has to look at the actual bad csv files he's trying to parse. As of today, Citi bank still exports broken csv for credit-card transactions. (Citi inserts extra CRLF in between quotes which breaks MS Excel csv import.) I don't need a formal spec telling me how it's broken. It's quite obvious by simply viewing it in Notepad. >Meanwhile, writing CSV docs is just a for loop with a printf. That's it. No. For example, if the field has a movie title with embedded quotes[2] or maybe a measurement using a one double quote for inches (e.g. [Sony 4k TV 65"])[3], using a naïve printf loop without generating extra escape characters will break many csv parsers that try to conform to RFC4180. [1] although someone did try to enumerate bad csv files: https://donatstudios.com/Falsehoods-Programmers-Believe-About-CSVs https://donatstudios.com/Falsehoods-Programmers-Believe-Abou... [2] example movie title [Precious: Based on the Novel "Push" by Sapphire] : https://en.wikipedia.org/wiki/Precious_(film) https://en.wikipedia.org/wiki/Precious_(film) [3] https://stackoverflow.com/questions/17808511/properly-escape-a-double-quote-in-csv https://stackoverflow.com/questions/17808511/properly-escape...