12 ms·

I disagree with this list of "falsehoods programmers believe". Most of those lists describes objectively wrong assumptions programmers have about some real-wor

I disagree with this list of "falsehoods programmers believe". Most of those lists describes objectively wrong assumptions programmers have about some real-world phenomenon. But in this case I'd argue that a lot of these points are simply not correctly formatted CSV files. Just because someone handed you a file with the extension .csv does not mean it's a CSV file proper, and it certainly does not mean that you have to guess at what it intends to encode without assuming any of the things on the list. For example, "All CSVs contains a single consistent encoding". If this is not the case I'd (rightfully) reject the file as being a proper CSV file.

k__ 9y ago
This. I mean if an API sends invalid JSON you can't parse it...
- masklinn 9y ago
  JSON is pretty extensively specified. Saying that CSV is not is generous.
  - vageli 9y ago
    Not a standard _per se_ [0] but goes to show that some level of effort has been applied to CSV standardization. [0]: https://tools.ietf.org/html/rfc4180 https://tools.ietf.org/html/rfc4180
    - justinclift 9y ago
      Yeah, that's the most commonly referenced standard. Approaching it from a database perspective (eg sqlitebrowser.org), where we need to both import and export CSV's... it can be tricky. For example, if an imported CSV file has two commas ",," that generally indicates an empty field. But, does that mean an empty string? Or does it mean we should use the default value for the field (as per database spec)? That specific case can be handled by a user preference on the import/export dialog ("What to do with empty fields?"). Other's less so. For example, database fields can contain data types (eg images, binary blobs) that have no representation defined in the RFC. Literally any decision about how to encode them is going to be non-standard. Ugh. There really needs to be a follow up RFC that specifies useful, practical things to tighten up the gaps.
  - dmitriid 9y ago
    Why is parsing JSON a minefield then? ;) http://seriot.ch/parsing_json.php http://seriot.ch/parsing_json.php
    - masklinn 9y ago
      Because "pretty extensively specified" does not mean "fully specified and perfectly implemented". The equivalent page for CSV would not have an end.
    - arghwhat 9y ago
      Where is the minefield? In accepting things that are not JSON, such as trailing commas and comments? That's silly. Everything blue and yellow in that matrix is entirely OK and should be green ("undefined" means "anything goes", and the spec specifically allows accepting more than JSON, so "should have failed" is bogus). That leaves a few parser bugs in form of orange and red, but they're corner cases (like surrogate pairs or 10000 nested arrays) and fixable.
  - paulddraper 9y ago
    But.... * Do you accept trailing commas? * Do you accept multiline comments? * Do you accept single line comments? * Do you accept literal newlines in strings? * Do you accept single quotes in place of double quotes? * Do you accept integers over 8 bytes? * Do you autodetect Unicode encoding? * Do you accept numbers with leading zeros? * Do you accept numbers with a leading plus sign? * Do you accept NaN? * Do you accept non-quoted object keys? * Do you accept duplicate object keys? In other words, > A JSON parser MAY accept non-JSON forms or extensions. https://tools.ietf.org/html/rfc7159 https://tools.ietf.org/html/rfc7159
    - arghwhat 9y ago
      As the quote states, none of that would be JSON. Thus, if you have a document that contains that, it is no longer JSON. A parser may accept things that are not JSON, but that's rather irrelevant. JSON does not have comments, support trailing comments, single quotes, etc. etc. etc.
      - paulddraper 9y ago
        Certainly. And some programs may accept things are not CSVs.
        arghwhat 9y ago
        The vast majority of programs support things that are not CSVs. :) (Couldn't help it.)
codingdave 9y ago
True, but if your job involves data transformations or migrations, then bad files, dirty data and such things are just part of the job. You deal with them. Rejecting incoming data because it isn't perfect is a path to bad customer service, and a poor reputation.
- sevensor 9y ago
  It depends on the context. If you're pulling in data from a source whose format the user can't control, then you have to deal with the broken CSV emitter. But if you're pulling in data from Excel, you're better off telling the user to save as xlsx because Excel makes such a hash of CSVs. In Python, I've had way fewer problems using xlrd and openpyxl than I have with any of the ways you can read CSVs.
- sametmax 9y ago
  Yeah but this has nothing to do with CSV. It's just about handling user input in general. try standard parsing, if fail try exotic features, if fails try so common hacks around common mistake, if fails just reject it as invalid data. We all do that all the time for everything. Form input, video upload, user click in a MMO, etc.
  - ryanbrunner 9y ago
    One challenge is that rejecting, or even detecting that something is wrong is often not a simple decision to make. Parsing a CSV in an invalid encoding will usually produce "valid" but nonsensical output. Determining that that has happened when your destination schema is somewhat freeform can be a challenge.
masklinn 9y ago
> For example, "All CSVs contains a single consistent encoding". If this is not the case I'd (rightfully) reject the file as being a proper CSV file. "CSV" only specify the metacharacters (at best), unless it's explicitly stated that the CSV is a single consistent encoding it's no more improper than inconsistently encoded Linux paths. It's stupid and a pain in the ass to be sure, but it's not improper. And when you don't control the entire pipeline, you will eventually hit this issue. In fact the first falsehood programmers believe about CSVs is that it's a good serialisation, interchange (between separate systems) or human-edition format. While it's routinely used for all of those it's also absolute garbage at all of them.
- xienze 9y ago
  > In fact the first falsehood programmers believe about CSVs is that it's a good serialisation, interchange (between separate systems) or human-edition format. I would argue that most programmers realize CSV sucks and would rather use something structured and sane. In my experience CSV support almost always comes about because someone from another business unit (e.g., marketing) needs self-service import facilities. Those folks live in Excel and CSV becomes the de facto import format.
  - masklinn 9y ago
    > Those folks live in Excel and CSV becomes the de facto import format. Which is an error, and you'd save yourself much pain if you just worked directly with excel files. Most languages have libraries allowing that.
    - paulryanrogers 9y ago
      IME none works exactly like Excel and therefore will inevitably fail as users push more and more of it's edges and bugs on the system
      - tinus_hn 9y ago
        Csv files don’t work like excel either. Typically all you need is a table with headings and then text and number fields.
DeusExMachina 9y ago
You can certainly reject the file, but that's not going to make your user happy. The (sad) reality is that you have often to deal with these edge cases in software. The user does not care that the CSV is not proper. He just wants to open it, and if it doesn't work, they will blame your program, not the source of the CSV. And you only control the former. I have had to deal with a CSV export from a web service that was "not proper". I notified their support, but that problem was never fixed. Luckily, the user in that case was just me. But if I was making software for someone else, blaming the right source of the problem would have not taken me anywhere.
- sametmax 9y ago
  If you put a phone number form and the user put letters inside, you will just say "invalid number". You won't try to parse it. Same for invalid CSV. There is nothing wrong with this. You can't accept all the garbage from the user.
  - epicide 9y ago
    I think his point is that it's not garbage from the user but garbage that the user needs to use. It's not as simple as invalid input or a bad copy/paste. The user likely cannot correct the CSV.
    - jalk 9y ago
      But every piece of software that accepts csv files cannot be expected to include an editor/data Wrangler mode that will allow the user to resolve encoding errors, ambiguities and inconsistencies, nor can it be expected to resolve those problems automagically
      - masklinn 9y ago
        Not only can they be, they are. Or they stop accepting CSV.
  - pilsetnieks 9y ago
kennydude 9y ago
Sometimes you have to, or customers will hate you for it
purple-again 9y ago
Lol "reject" the file. The author in this story could be me and the reason we both have so much experience working with that long list of fucked up CSV files is because we don't have the power to 'reject' anything. Our client gives us the data and tells us to fuck off until we solve their problem. Professional services != consumer services in servicer/servicee power disparity.
- macintux 9y ago
  Yeah, almost by definition if you’re dealing with crappy CSV files it’s because you’re part of a pipeline crossing organization boundaries with manual processes and you rarely have the authority or means to reject input.
vinceguidry 9y ago
> For example, "All CSVs contains a single consistent encoding". If this is not the case I'd (rightfully) reject the file as being a proper CSV file. That's literally the only one on there I felt that way about. All the other ones can and should be accounted for in your application if it accepts arbitrary CSVs. Encoding is a special case because it's impossible to do anything other than guess at a plaintext document's encoding. If you're getting byte code parsing errors, the code should try a few different encodings before rejecting the document. This won't help if there's a mix of encodings in the document. If this is the case, nothing can help and you will have garbage going in.
- arghwhat 9y ago
  The vast amount of entries are duplicates. Sorting those out, you're really not left with that much, and some of those are just invalid CSV's. Assuming I haven't miscounted: There are fifteen points effectively addressing delimiters, which all sum up to "delimiters may be arbitrary byte sequences, and they should be ignored in quoted fields or if escaped". There's nine points about encoding, three of which if violated just result in a flat out invalid file. There's four points that effectively claim that you must parse CSV files that adhere to no known formal or informal standard, which is just silly.
  - vinceguidry 9y ago
    These "Falsehoods programmers believe about X" articles lists all revolve around the same thing, accepting arbitrary inputs. If you can constrain your input, then great, you don't need to read the article. Or read it anyway, just so you get a better idea for exactly how to constrain it. But if you can't, and business needs force you to allow anyone and their dog Sue to pass input into your software system, then you need to be able to handle garbage input effectively, up to and including just correcting it for them if that's possible. The biases listed are good to check your code against if that's indeed the case.
    - arghwhat 9y ago
      The idea of "accepting arbitrary input" is nonsense. Arbitrary input would imply that I can feed the CSV parser a JPEG and expect some non-error result. A CSV parser will obviously only accept CSV. That list includes a few (5?) things that might end up being common bugs in hand-rolled CSV parsers. However, most of is is just bogus. The only possible way to handle "garbage" input is sensible errors. Of course, is someone is presenting a valid CSV variant that you do not support, then it's not garbage, but just a variant that you may wish to implement support for. A parser that tries to continue on error would be absolutely outrageous for a business to implement. How do you think the finance department would feel about their accounting being rendered invalid because of a parser trying to be smart?
nerdponx 9y ago
Fine, then call it "falsehoods programmers believe about people who try to create CSV files".
- sametmax 9y ago
  Well, you can create a zip full of nulls, that doesn't make it a zip. You can't deal with all the configuration of all the atoms in the universe, you need to draw a line. If something just can't be reasonably parsed, it's fair to reject it as invalid CSV.
  - majewsky 9y ago
    The problem is that your manager draws the line, not you.
- tjwds 9y ago
  Something that's being lost here is that this article is in the same vein as the seminal "Falsehoods programmers believe about time" [0] which contains quite a few statements which describe user error that developers still have to deal with (e.x. "The system clock will always be set to the correct local time.") 0. http://infiniteundo.com/post/25326999628/falsehoods-programmers-believe-about-time http://infiniteundo.com/post/25326999628/falsehoods-programm...
philjr 9y ago
Dealing with real world data and applications means being liberal in what you accept. If you're producing CSV files, then you can be more strict. https://en.wikipedia.org/wiki/Robustness_principle https://en.wikipedia.org/wiki/Robustness_principle
- masklinn 9y ago
  The original statement was about coverage of the breadth of the specification: if you're consuming data you need to accept everything the specification allow, but if you're generating data you should be producing as little of the specification's productions as you can. This does not work when the format is mostly unspecified and the vast majority of what you'll find in the wild is a tire fire.
lifthrasiir 9y ago
Not surprisingly, most lists of "falsehoods" are like that. I believe it is naturally impossible or intractable to consider every item of those lists. Rather, they are convenient checklists for consideration when you are delving into the realm of those complicated subjects. Do you reject, say, 30% of those items? That's fine. Just make sure that you are explicitly making tradeoffs and will consult the lists again when your initial tradeoffs wear off.
lou1306 9y ago
A colleague got stuck trying to parse a 1-million-lines CSV for a while, then he found out that the last line was a PHPMyAdmin timeout error. Maybe they should add "All CSV contain no error messages" to the list :)

k__ 9y ago

This. I mean if an API sends invalid JSON you can't parse it...

masklinn 9y ago
JSON is pretty extensively specified. Saying that CSV is not is generous.
- vageli 9y ago
  Not a standard _per se_ [0] but goes to show that some level of effort has been applied to CSV standardization. [0]: https://tools.ietf.org/html/rfc4180 https://tools.ietf.org/html/rfc4180
  - justinclift 9y ago
    Yeah, that's the most commonly referenced standard. Approaching it from a database perspective (eg sqlitebrowser.org), where we need to both import and export CSV's... it can be tricky. For example, if an imported CSV file has two commas ",," that generally indicates an empty field. But, does that mean an empty string? Or does it mean we should use the default value for the field (as per database spec)? That specific case can be handled by a user preference on the import/export dialog ("What to do with empty fields?"). Other's less so. For example, database fields can contain data types (eg images, binary blobs) that have no representation defined in the RFC. Literally any decision about how to encode them is going to be non-standard. Ugh. There really needs to be a follow up RFC that specifies useful, practical things to tighten up the gaps.
- dmitriid 9y ago
  Why is parsing JSON a minefield then? ;) http://seriot.ch/parsing_json.php http://seriot.ch/parsing_json.php
  - masklinn 9y ago
    Because "pretty extensively specified" does not mean "fully specified and perfectly implemented". The equivalent page for CSV would not have an end.
  - arghwhat 9y ago
    Where is the minefield? In accepting things that are not JSON, such as trailing commas and comments? That's silly. Everything blue and yellow in that matrix is entirely OK and should be green ("undefined" means "anything goes", and the spec specifically allows accepting more than JSON, so "should have failed" is bogus). That leaves a few parser bugs in form of orange and red, but they're corner cases (like surrogate pairs or 10000 nested arrays) and fixable.
- paulddraper 9y ago
  But.... * Do you accept trailing commas? * Do you accept multiline comments? * Do you accept single line comments? * Do you accept literal newlines in strings? * Do you accept single quotes in place of double quotes? * Do you accept integers over 8 bytes? * Do you autodetect Unicode encoding? * Do you accept numbers with leading zeros? * Do you accept numbers with a leading plus sign? * Do you accept NaN? * Do you accept non-quoted object keys? * Do you accept duplicate object keys? In other words, > A JSON parser MAY accept non-JSON forms or extensions. https://tools.ietf.org/html/rfc7159 https://tools.ietf.org/html/rfc7159
  - arghwhat 9y ago
    As the quote states, none of that would be JSON. Thus, if you have a document that contains that, it is no longer JSON. A parser may accept things that are not JSON, but that's rather irrelevant. JSON does not have comments, support trailing comments, single quotes, etc. etc. etc.
    - paulddraper 9y ago
      Certainly. And some programs may accept things are not CSVs.
      - arghwhat 9y ago
        The vast majority of programs support things that are not CSVs. :) (Couldn't help it.)

codingdave 9y ago

True, but if your job involves data transformations or migrations, then bad files, dirty data and such things are just part of the job. You deal with them. Rejecting incoming data because it isn't perfect is a path to bad customer service, and a poor reputation.

sevensor 9y ago
It depends on the context. If you're pulling in data from a source whose format the user can't control, then you have to deal with the broken CSV emitter. But if you're pulling in data from Excel, you're better off telling the user to save as xlsx because Excel makes such a hash of CSVs. In Python, I've had way fewer problems using xlrd and openpyxl than I have with any of the ways you can read CSVs.
sametmax 9y ago
Yeah but this has nothing to do with CSV. It's just about handling user input in general. try standard parsing, if fail try exotic features, if fails try so common hacks around common mistake, if fails just reject it as invalid data. We all do that all the time for everything. Form input, video upload, user click in a MMO, etc.
- ryanbrunner 9y ago
  One challenge is that rejecting, or even detecting that something is wrong is often not a simple decision to make. Parsing a CSV in an invalid encoding will usually produce "valid" but nonsensical output. Determining that that has happened when your destination schema is somewhat freeform can be a challenge.

masklinn 9y ago

> For example, "All CSVs contains a single consistent encoding". If this is not the case I'd (rightfully) reject the file as being a proper CSV file. "CSV" only specify the metacharacters (at best), unless it's explicitly stated that the CSV is a single consistent encoding it's no more improper than inconsistently encoded Linux paths. It's stupid and a pain in the ass to be sure, but it's not improper. And when you don't control the entire pipeline, you will eventually hit this issue. In fact the first falsehood programmers believe about CSVs is that it's a good serialisation, interchange (between separate systems) or human-edition format. While it's routinely used for all of those it's also absolute garbage at all of them.

xienze 9y ago
> In fact the first falsehood programmers believe about CSVs is that it's a good serialisation, interchange (between separate systems) or human-edition format. I would argue that most programmers realize CSV sucks and would rather use something structured and sane. In my experience CSV support almost always comes about because someone from another business unit (e.g., marketing) needs self-service import facilities. Those folks live in Excel and CSV becomes the de facto import format.
- masklinn 9y ago
  > Those folks live in Excel and CSV becomes the de facto import format. Which is an error, and you'd save yourself much pain if you just worked directly with excel files. Most languages have libraries allowing that.
  - paulryanrogers 9y ago
    IME none works exactly like Excel and therefore will inevitably fail as users push more and more of it's edges and bugs on the system
    - tinus_hn 9y ago
      Csv files don’t work like excel either. Typically all you need is a table with headings and then text and number fields.

DeusExMachina 9y ago

You can certainly reject the file, but that's not going to make your user happy. The (sad) reality is that you have often to deal with these edge cases in software. The user does not care that the CSV is not proper. He just wants to open it, and if it doesn't work, they will blame your program, not the source of the CSV. And you only control the former. I have had to deal with a CSV export from a web service that was "not proper". I notified their support, but that problem was never fixed. Luckily, the user in that case was just me. But if I was making software for someone else, blaming the right source of the problem would have not taken me anywhere.

sametmax 9y ago
If you put a phone number form and the user put letters inside, you will just say "invalid number". You won't try to parse it. Same for invalid CSV. There is nothing wrong with this. You can't accept all the garbage from the user.
- epicide 9y ago
  I think his point is that it's not garbage from the user but garbage that the user needs to use. It's not as simple as invalid input or a bad copy/paste. The user likely cannot correct the CSV.
  - jalk 9y ago
    But every piece of software that accepts csv files cannot be expected to include an editor/data Wrangler mode that will allow the user to resolve encoding errors, ambiguities and inconsistencies, nor can it be expected to resolve those problems automagically
    - masklinn 9y ago
      Not only can they be, they are. Or they stop accepting CSV.
- pilsetnieks 9y ago

kennydude 9y ago

Sometimes you have to, or customers will hate you for it

purple-again 9y ago

Lol "reject" the file. The author in this story could be me and the reason we both have so much experience working with that long list of fucked up CSV files is because we don't have the power to 'reject' anything. Our client gives us the data and tells us to fuck off until we solve their problem. Professional services != consumer services in servicer/servicee power disparity.

macintux 9y ago
Yeah, almost by definition if you’re dealing with crappy CSV files it’s because you’re part of a pipeline crossing organization boundaries with manual processes and you rarely have the authority or means to reject input.

vinceguidry 9y ago

> For example, "All CSVs contains a single consistent encoding". If this is not the case I'd (rightfully) reject the file as being a proper CSV file. That's literally the only one on there I felt that way about. All the other ones can and should be accounted for in your application if it accepts arbitrary CSVs. Encoding is a special case because it's impossible to do anything other than guess at a plaintext document's encoding. If you're getting byte code parsing errors, the code should try a few different encodings before rejecting the document. This won't help if there's a mix of encodings in the document. If this is the case, nothing can help and you will have garbage going in.

arghwhat 9y ago
The vast amount of entries are duplicates. Sorting those out, you're really not left with that much, and some of those are just invalid CSV's. Assuming I haven't miscounted: There are fifteen points effectively addressing delimiters, which all sum up to "delimiters may be arbitrary byte sequences, and they should be ignored in quoted fields or if escaped". There's nine points about encoding, three of which if violated just result in a flat out invalid file. There's four points that effectively claim that you must parse CSV files that adhere to no known formal or informal standard, which is just silly.
- vinceguidry 9y ago
  These "Falsehoods programmers believe about X" articles lists all revolve around the same thing, accepting arbitrary inputs. If you can constrain your input, then great, you don't need to read the article. Or read it anyway, just so you get a better idea for exactly how to constrain it. But if you can't, and business needs force you to allow anyone and their dog Sue to pass input into your software system, then you need to be able to handle garbage input effectively, up to and including just correcting it for them if that's possible. The biases listed are good to check your code against if that's indeed the case.
  - arghwhat 9y ago
    The idea of "accepting arbitrary input" is nonsense. Arbitrary input would imply that I can feed the CSV parser a JPEG and expect some non-error result. A CSV parser will obviously only accept CSV. That list includes a few (5?) things that might end up being common bugs in hand-rolled CSV parsers. However, most of is is just bogus. The only possible way to handle "garbage" input is sensible errors. Of course, is someone is presenting a valid CSV variant that you do not support, then it's not garbage, but just a variant that you may wish to implement support for. A parser that tries to continue on error would be absolutely outrageous for a business to implement. How do you think the finance department would feel about their accounting being rendered invalid because of a parser trying to be smart?

nerdponx 9y ago

Fine, then call it "falsehoods programmers believe about people who try to create CSV files".

sametmax 9y ago
Well, you can create a zip full of nulls, that doesn't make it a zip. You can't deal with all the configuration of all the atoms in the universe, you need to draw a line. If something just can't be reasonably parsed, it's fair to reject it as invalid CSV.
- majewsky 9y ago
  The problem is that your manager draws the line, not you.
tjwds 9y ago
Something that's being lost here is that this article is in the same vein as the seminal "Falsehoods programmers believe about time" [0] which contains quite a few statements which describe user error that developers still have to deal with (e.x. "The system clock will always be set to the correct local time.") 0. http://infiniteundo.com/post/25326999628/falsehoods-programmers-believe-about-time http://infiniteundo.com/post/25326999628/falsehoods-programm...

philjr 9y ago

Dealing with real world data and applications means being liberal in what you accept. If you're producing CSV files, then you can be more strict. https://en.wikipedia.org/wiki/Robustness_principle https://en.wikipedia.org/wiki/Robustness_principle

masklinn 9y ago
The original statement was about coverage of the breadth of the specification: if you're consuming data you need to accept everything the specification allow, but if you're generating data you should be producing as little of the specification's productions as you can. This does not work when the format is mostly unspecified and the vast majority of what you'll find in the wild is a tire fire.

lifthrasiir 9y ago

Not surprisingly, most lists of "falsehoods" are like that. I believe it is naturally impossible or intractable to consider every item of those lists. Rather, they are convenient checklists for consideration when you are delving into the realm of those complicated subjects. Do you reject, say, 30% of those items? That's fine. Just make sure that you are explicitly making tradeoffs and will consult the lists again when your initial tradeoffs wear off.

lou1306 9y ago

A colleague got stuck trying to parse a 1-million-lines CSV for a while, then he found out that the last line was a PHPMyAdmin timeout error. Maybe they should add "All CSV contain no error messages" to the list :)