14 ms·

You can certainly reject the file, but that's not going to make your user happy. The (sad) reality is that you have often to deal with these edge cases in softw

• sametmax 9y ago
  If you put a phone number form and the user put letters inside, you will just say "invalid number". You won't try to parse it. Same for invalid CSV. There is nothing wrong with this. You can't accept all the garbage from the user.
  • epicide 9y ago
    I think his point is that it's not garbage from the user but garbage that the user needs to use. It's not as simple as invalid input or a bad copy/paste. The user likely cannot correct the CSV.
    • jalk 9y ago
      But every piece of software that accepts csv files cannot be expected to include an editor/data Wrangler mode that will allow the user to resolve encoding errors, ambiguities and inconsistencies, nor can it be expected to resolve those problems automagically
      • masklinn 9y ago
        Not only can they be, they are. Or they stop accepting CSV.
  • pilsetnieks 9y ago
    The real problem is when the user complains "but it works fine in Excel." If it does work elsewhere then in the user's eyes it's your app's fault that it cannot read a working file. Yes, it does work in Excel but it's because of 40 years of various edge case fixes that you just cannot replicate in their entirety in your app.
    • ryanbrunner 9y ago
      Really what this comes down to is that the CSV "standard" is in actuality an undocumented standard with one reference implementation - Excel. If it works in Excel, it's valid CSV, if it doesn't, then it's not. It's a sad state of affairs, and probably one reason to avoid working with CSV unless absolutely necessary, but there's no getting around that fact.
      • emodendroket 9y ago
        The main reason you'd choose CSV is precisely that you can open it in Excel.
        • rjbwork 9y ago
          I use CSV (really tsv) because big data tools can open them extremely easily, because they're human readable, and because they're easy for my software stack to parse and use. I do most of my data processing for local consumption on a repl-like tool. We have a defined configuration for TSV we use internally that is lossless in terms of storing and extracting a number of complex data types. Embedded JSON with embedded separators in that JSON? No problem.
        • tinus_hn 9y ago
          Originally the xls format was the only open format in the office family and it was actually much easier to create xls files than to deal with the stupidity of csv
          • emodendroket 9y ago
  • Legogris 9y ago
    The thing is, a lot (all?) of these assumptions are not part of the CSV standard. Things you view as "garbage" might still be technically valid CSV.
  • noxToken 9y ago
    You're right, but non-technical clients don't care. It's a little different to them. With a ZIP code in the US, it consists of 5 digits that form a unique number for a geographical area with 4 more digits that represents a sub-area within the 5 digit ZIP. But there are no instances where there can be a letter. That's a hard fact. If you put 4308A-65XK as the ZIP, the post office or delivery service will not know how to deliver the mail without trying to cross reference address with possible valid ZIP codes. Invalid CSV is murky, because CSV is a serialization format. That is, invalid CSV may not play well with a program that accepts well-formed CSV, but a human may still be able to parse it with no problem. Returning "invalid ZIP Code" makes sense. The client would understand that letters do not belong in a ZIP Code. If you've never had to deal with a nightmare client of that sort, hats off to you.
    • EGreg 9y ago
      Perhaps the problem is with the client relationship. Why not explain to the client that you will need to put in manual or custom work to clean up the CSV or batch of CSVs and charge for the extra time writing a script or cleanup? Literally the only time this may not apply is if you are writing a general purpose program.
    • sametmax 9y ago
      You can also have clients that don't have a working OS. But you can't fix that. You need to choose a limit after which you say "no, I won't do that, get you s* together" Well, it all depends on the money, doesn't it?
  • asfdsfggtfd 9y ago
    Out of interest what do you regard as a valid phone number?
    • walshemj 9y ago
      as defined in E.164 (https://en.wikipedia.org/wiki/E.164 https://en.wikipedia.org/wiki/E.164)
  • doodpants 9y ago
    > If you put a phone number form and the user put letters inside, you will just say "invalid number". You won't try to parse it. Wait, what? Phone numbers could always include letters. Many businesses get phone numbers that include or consist of words relevant to their business, to make them memorable. And I've known people whose personal number happens to contain a word or words, so they give the letter version out because it's easy to remember. If a phone number field doesn't include simple logic to map letters into the proper numbers, that's just lazy programming.
    • dronescanfly 9y ago
      Wait what? There are countries that put letters into phone NUMBERS?
      • emodendroket 9y ago
        The numbers all have three or four letters above them and people (mostly businesses, really) will sometimes get a number that spells out a memorable mnemonic. That's all he's talking about.
      • lmkg 9y ago
        This is what the average phone bad looks like in the United States: http://www.pachd.com/free-images/technology-images/phone-dial-pad-01.jpg http://www.pachd.com/free-images/technology-images/phone-dia... There is a standard, universal mapping of letters onto numbers. This allows companies to advertise their phone number as something memorable like 1-800-CASH-4-AU and everyone knows how to dial it. People don't even need to remember the mapping of letters, because they're printed on every keypad.
      • sevensor 9y ago
        Used to be common in the US. You'd have two letters for the exchange, then five digits for the telephone number.
      • walshemj 9y ago
        Well there are the A B C and D DTMF tones not sure if those technically count as numbers and from memory they are without the ITU spec for a phone numbers.
• ptero 9y ago
  There are different use cases. What you say is not really CSV specific and true about most data sets: you receive data sets that are not compliant to whatever spec you expect. Different projects must handle it in different ways, either making some efforts to correct errors or not. In some cases data users wield enormous power over data submitters (e.g., many for-record safety analyses or real-time systems). If any part of the data collection process is suspect the sample is likely to get tossed. And for spec compliance, if a data set does not pass perfectly through a validator the sender will likely get some sort of a "shame on you" nastygram.
  • deleted 9y ago
    [deleted]
• lhnz 9y ago
  Why not create a transformation function which uses some heuristics to take a non-compliant sort-of-CSV and turn it into a CSV. Doing so doesn't mean that it is CSV, it just means you can parse it.
  • jacobush 9y ago
    Why would you do that?
    • lhnz 9y ago
      I guess I would do it, because I could then be explicit about the differences between the two, and explicit about how I would decode something if it was indeterminate.
  • andreareina 9y ago
    That non-compliant sort-of-CSV can be cleanly (i.e. without errors) decoded as several "dialects", producing multiple different results, of which only one (or none) will be correct.
• emodendroket 9y ago
  How could you possibly handle a single file that mixes and matches encodings? I'd think opening that in any software is going to result in some garbled characters.