5 ms·

If you put a phone number form and the user put letters inside, you will just say "invalid number". You won't try to parse it. Same for invalid CSV. There is

If you put a phone number form and the user put letters inside, you will just say "invalid number". You won't try to parse it. Same for invalid CSV. There is nothing wrong with this. You can't accept all the garbage from the user.

epicide 9y ago
I think his point is that it's not garbage from the user but garbage that the user needs to use. It's not as simple as invalid input or a bad copy/paste. The user likely cannot correct the CSV.
- jalk 9y ago
  But every piece of software that accepts csv files cannot be expected to include an editor/data Wrangler mode that will allow the user to resolve encoding errors, ambiguities and inconsistencies, nor can it be expected to resolve those problems automagically
  - masklinn 9y ago
    Not only can they be, they are. Or they stop accepting CSV.
pilsetnieks 9y ago
The real problem is when the user complains "but it works fine in Excel." If it does work elsewhere then in the user's eyes it's your app's fault that it cannot read a working file. Yes, it does work in Excel but it's because of 40 years of various edge case fixes that you just cannot replicate in their entirety in your app.
- ryanbrunner 9y ago
  Really what this comes down to is that the CSV "standard" is in actuality an undocumented standard with one reference implementation - Excel. If it works in Excel, it's valid CSV, if it doesn't, then it's not. It's a sad state of affairs, and probably one reason to avoid working with CSV unless absolutely necessary, but there's no getting around that fact.
  - emodendroket 9y ago
    The main reason you'd choose CSV is precisely that you can open it in Excel.
    - rjbwork 9y ago
      I use CSV (really tsv) because big data tools can open them extremely easily, because they're human readable, and because they're easy for my software stack to parse and use. I do most of my data processing for local consumption on a repl-like tool. We have a defined configuration for TSV we use internally that is lossless in terms of storing and extracting a number of complex data types. Embedded JSON with embedded separators in that JSON? No problem.
    - tinus_hn 9y ago
      Originally the xls format was the only open format in the office family and it was actually much easier to create xls files than to deal with the stupidity of csv
      - emodendroket 9y ago
        They're all open now, but the spec is monstrously large and difficult to work with. Definitely not easier to do than CSV.
        tinus_hn 9y ago
        .xlsx is not .xls
        emodendroket 9y ago
        Isn't xls a binary format? I can't imagine that's more pleasant to work in.
        tinus_hn 9y ago
        I take a defined format like .xls over an unspecified de facto ‘standard’ like .csv anytime. In many locales .csv files don’t even contain comma separated values because the comma is the decimal symbol already. It’s a mess, half of the time Microsoft can’t make it work either even when you only use their software.
- shawnz 9y ago
  > Yes, it does work in Excel but it's because of 40 years of various edge case fixes that you just cannot replicate in their entirety in your app. So given its impossible to compete totally with Excel, how do we decide which of these edge cases are worth our time to implement?
  - TheCoelacanth 9y ago
    The problem is that for many products, if it can't integrate with an Excel workflow, it is dead-on-arrival.
Legogris 9y ago
The thing is, a lot (all?) of these assumptions are not part of the CSV standard. Things you view as "garbage" might still be technically valid CSV.
noxToken 9y ago
You're right, but non-technical clients don't care. It's a little different to them. With a ZIP code in the US, it consists of 5 digits that form a unique number for a geographical area with 4 more digits that represents a sub-area within the 5 digit ZIP. But there are no instances where there can be a letter. That's a hard fact. If you put 4308A-65XK as the ZIP, the post office or delivery service will not know how to deliver the mail without trying to cross reference address with possible valid ZIP codes. Invalid CSV is murky, because CSV is a serialization format. That is, invalid CSV may not play well with a program that accepts well-formed CSV, but a human may still be able to parse it with no problem. Returning "invalid ZIP Code" makes sense. The client would understand that letters do not belong in a ZIP Code. If you've never had to deal with a nightmare client of that sort, hats off to you.
- EGreg 9y ago
  Perhaps the problem is with the client relationship. Why not explain to the client that you will need to put in manual or custom work to clean up the CSV or batch of CSVs and charge for the extra time writing a script or cleanup? Literally the only time this may not apply is if you are writing a general purpose program.
- sametmax 9y ago
  You can also have clients that don't have a working OS. But you can't fix that. You need to choose a limit after which you say "no, I won't do that, get you s* together" Well, it all depends on the money, doesn't it?
asfdsfggtfd 9y ago
Out of interest what do you regard as a valid phone number?
- walshemj 9y ago
  as defined in E.164 (https://en.wikipedia.org/wiki/E.164 https://en.wikipedia.org/wiki/E.164)
doodpants 9y ago
> If you put a phone number form and the user put letters inside, you will just say "invalid number". You won't try to parse it. Wait, what? Phone numbers could always include letters. Many businesses get phone numbers that include or consist of words relevant to their business, to make them memorable. And I've known people whose personal number happens to contain a word or words, so they give the letter version out because it's easy to remember. If a phone number field doesn't include simple logic to map letters into the proper numbers, that's just lazy programming.
- dronescanfly 9y ago
  Wait what? There are countries that put letters into phone NUMBERS?
  - emodendroket 9y ago
    The numbers all have three or four letters above them and people (mostly businesses, really) will sometimes get a number that spells out a memorable mnemonic. That's all he's talking about.
  - lmkg 9y ago
    This is what the average phone bad looks like in the United States: http://www.pachd.com/free-images/technology-images/phone-dial-pad-01.jpg http://www.pachd.com/free-images/technology-images/phone-dia... There is a standard, universal mapping of letters onto numbers. This allows companies to advertise their phone number as something memorable like 1-800-CASH-4-AU and everyone knows how to dial it. People don't even need to remember the mapping of letters, because they're printed on every keypad.
  - sevensor 9y ago
    Used to be common in the US. You'd have two letters for the exchange, then five digits for the telephone number.
  - walshemj 9y ago
    Well there are the A B C and D DTMF tones not sure if those technically count as numbers and from memory they are without the ITU spec for a phone numbers.

epicide 9y ago

I think his point is that it's not garbage from the user but garbage that the user needs to use. It's not as simple as invalid input or a bad copy/paste. The user likely cannot correct the CSV.

jalk 9y ago
But every piece of software that accepts csv files cannot be expected to include an editor/data Wrangler mode that will allow the user to resolve encoding errors, ambiguities and inconsistencies, nor can it be expected to resolve those problems automagically
- masklinn 9y ago
  Not only can they be, they are. Or they stop accepting CSV.

pilsetnieks 9y ago

The real problem is when the user complains "but it works fine in Excel." If it does work elsewhere then in the user's eyes it's your app's fault that it cannot read a working file. Yes, it does work in Excel but it's because of 40 years of various edge case fixes that you just cannot replicate in their entirety in your app.

ryanbrunner 9y ago
Really what this comes down to is that the CSV "standard" is in actuality an undocumented standard with one reference implementation - Excel. If it works in Excel, it's valid CSV, if it doesn't, then it's not. It's a sad state of affairs, and probably one reason to avoid working with CSV unless absolutely necessary, but there's no getting around that fact.
- emodendroket 9y ago
  The main reason you'd choose CSV is precisely that you can open it in Excel.
  - rjbwork 9y ago
    I use CSV (really tsv) because big data tools can open them extremely easily, because they're human readable, and because they're easy for my software stack to parse and use. I do most of my data processing for local consumption on a repl-like tool. We have a defined configuration for TSV we use internally that is lossless in terms of storing and extracting a number of complex data types. Embedded JSON with embedded separators in that JSON? No problem.
  - tinus_hn 9y ago
    Originally the xls format was the only open format in the office family and it was actually much easier to create xls files than to deal with the stupidity of csv
    - emodendroket 9y ago
      They're all open now, but the spec is monstrously large and difficult to work with. Definitely not easier to do than CSV.
      - tinus_hn 9y ago
        .xlsx is not .xls
        emodendroket 9y ago
        Isn't xls a binary format? I can't imagine that's more pleasant to work in.
        tinus_hn 9y ago
        I take a defined format like .xls over an unspecified de facto ‘standard’ like .csv anytime. In many locales .csv files don’t even contain comma separated values because the comma is the decimal symbol already. It’s a mess, half of the time Microsoft can’t make it work either even when you only use their software.
shawnz 9y ago
> Yes, it does work in Excel but it's because of 40 years of various edge case fixes that you just cannot replicate in their entirety in your app. So given its impossible to compete totally with Excel, how do we decide which of these edge cases are worth our time to implement?
- TheCoelacanth 9y ago
  The problem is that for many products, if it can't integrate with an Excel workflow, it is dead-on-arrival.

Legogris 9y ago

The thing is, a lot (all?) of these assumptions are not part of the CSV standard. Things you view as "garbage" might still be technically valid CSV.

noxToken 9y ago

You're right, but non-technical clients don't care. It's a little different to them. With a ZIP code in the US, it consists of 5 digits that form a unique number for a geographical area with 4 more digits that represents a sub-area within the 5 digit ZIP. But there are no instances where there can be a letter. That's a hard fact. If you put 4308A-65XK as the ZIP, the post office or delivery service will not know how to deliver the mail without trying to cross reference address with possible valid ZIP codes. Invalid CSV is murky, because CSV is a serialization format. That is, invalid CSV may not play well with a program that accepts well-formed CSV, but a human may still be able to parse it with no problem. Returning "invalid ZIP Code" makes sense. The client would understand that letters do not belong in a ZIP Code. If you've never had to deal with a nightmare client of that sort, hats off to you.

EGreg 9y ago
Perhaps the problem is with the client relationship. Why not explain to the client that you will need to put in manual or custom work to clean up the CSV or batch of CSVs and charge for the extra time writing a script or cleanup? Literally the only time this may not apply is if you are writing a general purpose program.
sametmax 9y ago
You can also have clients that don't have a working OS. But you can't fix that. You need to choose a limit after which you say "no, I won't do that, get you s* together" Well, it all depends on the money, doesn't it?

asfdsfggtfd 9y ago

Out of interest what do you regard as a valid phone number?

walshemj 9y ago
as defined in E.164 (https://en.wikipedia.org/wiki/E.164 https://en.wikipedia.org/wiki/E.164)

doodpants 9y ago

> If you put a phone number form and the user put letters inside, you will just say "invalid number". You won't try to parse it. Wait, what? Phone numbers could always include letters. Many businesses get phone numbers that include or consist of words relevant to their business, to make them memorable. And I've known people whose personal number happens to contain a word or words, so they give the letter version out because it's easy to remember. If a phone number field doesn't include simple logic to map letters into the proper numbers, that's just lazy programming.

dronescanfly 9y ago
Wait what? There are countries that put letters into phone NUMBERS?
- emodendroket 9y ago
  The numbers all have three or four letters above them and people (mostly businesses, really) will sometimes get a number that spells out a memorable mnemonic. That's all he's talking about.
- lmkg 9y ago
  This is what the average phone bad looks like in the United States: http://www.pachd.com/free-images/technology-images/phone-dial-pad-01.jpg http://www.pachd.com/free-images/technology-images/phone-dia... There is a standard, universal mapping of letters onto numbers. This allows companies to advertise their phone number as something memorable like 1-800-CASH-4-AU and everyone knows how to dial it. People don't even need to remember the mapping of letters, because they're printed on every keypad.
- sevensor 9y ago
  Used to be common in the US. You'd have two letters for the exchange, then five digits for the telephone number.
- walshemj 9y ago
  Well there are the A B C and D DTMF tones not sure if those technically count as numbers and from memory they are without the ITU spec for a phone numbers.