4 ms·

> For example, "All CSVs contains a single consistent encoding". If this is not the case I'd (rightfully) reject the file as being a proper CSV file. That's li

> For example, "All CSVs contains a single consistent encoding". If this is not the case I'd (rightfully) reject the file as being a proper CSV file. That's literally the only one on there I felt that way about. All the other ones can and should be accounted for in your application if it accepts arbitrary CSVs. Encoding is a special case because it's impossible to do anything other than guess at a plaintext document's encoding. If you're getting byte code parsing errors, the code should try a few different encodings before rejecting the document. This won't help if there's a mix of encodings in the document. If this is the case, nothing can help and you will have garbage going in.

arghwhat 9y ago
The vast amount of entries are duplicates. Sorting those out, you're really not left with that much, and some of those are just invalid CSV's. Assuming I haven't miscounted: There are fifteen points effectively addressing delimiters, which all sum up to "delimiters may be arbitrary byte sequences, and they should be ignored in quoted fields or if escaped". There's nine points about encoding, three of which if violated just result in a flat out invalid file. There's four points that effectively claim that you must parse CSV files that adhere to no known formal or informal standard, which is just silly.
- vinceguidry 9y ago
  These "Falsehoods programmers believe about X" articles lists all revolve around the same thing, accepting arbitrary inputs. If you can constrain your input, then great, you don't need to read the article. Or read it anyway, just so you get a better idea for exactly how to constrain it. But if you can't, and business needs force you to allow anyone and their dog Sue to pass input into your software system, then you need to be able to handle garbage input effectively, up to and including just correcting it for them if that's possible. The biases listed are good to check your code against if that's indeed the case.
  - arghwhat 9y ago
    The idea of "accepting arbitrary input" is nonsense. Arbitrary input would imply that I can feed the CSV parser a JPEG and expect some non-error result. A CSV parser will obviously only accept CSV. That list includes a few (5?) things that might end up being common bugs in hand-rolled CSV parsers. However, most of is is just bogus. The only possible way to handle "garbage" input is sensible errors. Of course, is someone is presenting a valid CSV variant that you do not support, then it's not garbage, but just a variant that you may wish to implement support for. A parser that tries to continue on error would be absolutely outrageous for a business to implement. How do you think the finance department would feel about their accounting being rendered invalid because of a parser trying to be smart?
    - vinceguidry 9y ago
      > The idea of "accepting arbitrary input" is nonsense. Arbitrary input would imply that I can feed the CSV parser a JPEG and expect some non-error result. This is completely wrong. The idea is to sift through the input and return appropriate error codes for garbage input, while remaining flexible enough to handle malformed input that's not quite garbage. As the maxim goes, be flexible in what you accept, strict in what you hand out.
      - arghwhat 9y ago
        Any input that your program accepts that produces a sensible output is within the set of inputs for which your program is designed to be valid, which is a subset of all possible inputs. Arbitrary input means any possible input, beyond the set for which your program is valid, including those that causes your program to segfault or set your cat on fire. Expecting to handle "arbitrary input data" would mean to try to accept data outside of that set. In the case of a CSV parser, it would mean parsing /dev/urandom as a CSV and producing sensible output ("sensible" defined by the users, which in your example is the dog "Sue"). What the maxim says about being flexible in what you accept is not trying to tell you that everything should be considered valid, but instead that it doesn't hurt to allow certain rules to be broken if it cannot do any harm. This could mean supporting trailing commas in JSON arrays, which while illegal is a harmless "extension". However, it is not trying to tell you that you should accept garbage and "silently recover", trying to do something with invalid input. Doing so does more harm than good. Be flexible, not stupid.

arghwhat 9y ago

The vast amount of entries are duplicates. Sorting those out, you're really not left with that much, and some of those are just invalid CSV's. Assuming I haven't miscounted: There are fifteen points effectively addressing delimiters, which all sum up to "delimiters may be arbitrary byte sequences, and they should be ignored in quoted fields or if escaped". There's nine points about encoding, three of which if violated just result in a flat out invalid file. There's four points that effectively claim that you must parse CSV files that adhere to no known formal or informal standard, which is just silly.

vinceguidry 9y ago
These "Falsehoods programmers believe about X" articles lists all revolve around the same thing, accepting arbitrary inputs. If you can constrain your input, then great, you don't need to read the article. Or read it anyway, just so you get a better idea for exactly how to constrain it. But if you can't, and business needs force you to allow anyone and their dog Sue to pass input into your software system, then you need to be able to handle garbage input effectively, up to and including just correcting it for them if that's possible. The biases listed are good to check your code against if that's indeed the case.
- arghwhat 9y ago
  The idea of "accepting arbitrary input" is nonsense. Arbitrary input would imply that I can feed the CSV parser a JPEG and expect some non-error result. A CSV parser will obviously only accept CSV. That list includes a few (5?) things that might end up being common bugs in hand-rolled CSV parsers. However, most of is is just bogus. The only possible way to handle "garbage" input is sensible errors. Of course, is someone is presenting a valid CSV variant that you do not support, then it's not garbage, but just a variant that you may wish to implement support for. A parser that tries to continue on error would be absolutely outrageous for a business to implement. How do you think the finance department would feel about their accounting being rendered invalid because of a parser trying to be smart?
  - vinceguidry 9y ago
    > The idea of "accepting arbitrary input" is nonsense. Arbitrary input would imply that I can feed the CSV parser a JPEG and expect some non-error result. This is completely wrong. The idea is to sift through the input and return appropriate error codes for garbage input, while remaining flexible enough to handle malformed input that's not quite garbage. As the maxim goes, be flexible in what you accept, strict in what you hand out.
    - arghwhat 9y ago
      Any input that your program accepts that produces a sensible output is within the set of inputs for which your program is designed to be valid, which is a subset of all possible inputs. Arbitrary input means any possible input, beyond the set for which your program is valid, including those that causes your program to segfault or set your cat on fire. Expecting to handle "arbitrary input data" would mean to try to accept data outside of that set. In the case of a CSV parser, it would mean parsing /dev/urandom as a CSV and producing sensible output ("sensible" defined by the users, which in your example is the dog "Sue"). What the maxim says about being flexible in what you accept is not trying to tell you that everything should be considered valid, but instead that it doesn't hurt to allow certain rules to be broken if it cannot do any harm. This could mean supporting trailing commas in JSON arrays, which while illegal is a harmless "extension". However, it is not trying to tell you that you should accept garbage and "silently recover", trying to do something with invalid input. Doing so does more harm than good. Be flexible, not stupid.