5 ms·

We recently received a CSV from a bank where they didn't escape properly, so one of the entries was like this: "","C/O COMPANYNAME"SUBHEADLINE"","",(...)

by kawsper 10y ago

We recently received a CSV from a bank where they didn't escape properly, so one of the entries was like this: "","C/O COMPANYNAME"SUBHEADLINE"","",(...) So yeah, they probably wrote this themselves, and didn't expect a valid company name to contain these characters. We ended up writing a custom parser around their horrible format, because their technical support and technical lead person we could reach couldn't understand our issues. :(

pdimitar 10y ago
What you describe is exactly the problem with CSV: many less tech-savvy people decide to use it to provide some sort of very minimal API (even if it's periodical manual exports sent by mail; some people consider that automation!), don't account for the possible traps (like double quotes in the value of any field), then call it a day -- more like a lifetime. Then there's inertia, there's a boss saying "we absolutely must work with these people", and... the rest is history for many of us. NOTE: I've been writing custom CSV parsers at least 10 times during my career. That makes roughly 2 such parsers every 3 years. And that comes from a guy that's not very active in his career.
- astrobe_ 10y ago
  Not just with CSV. I remember a similar comment about broken XML. It actually doesn't matter how compliant you are to the standard. Standards are certainly nice when both parties are willing to agree on things. But when one party is in position of strength and wants to minimize its costs or efforts, compliance to a standard won't help you.
  - pdimitar 10y ago
    That's the sad truth. Sometimes the thought of this makes me wanna go full Hitler on the internet -- if you can't expose a standards-compliant API, your service will be blocked by central tech authorities. We can dream, right? Still though, all giants eventually fall when they become phlegmatic. Sadly it's taking way too long.
jbax 10y ago
univocity-parsers can handle this sort of problem without a hitch: https://github.com/uniVocity/univocity-parsers https://github.com/uniVocity/univocity-parsers You should be able to process any sort of record with unescaped quotes.
lightbendover 10y ago
Writing a custom parser for "data files similar-to but different-from CSV" is day-to-day life in B2B.

by kawsper 10y ago

pdimitar 10y ago

What you describe is exactly the problem with CSV: many less tech-savvy people decide to use it to provide some sort of very minimal API (even if it's periodical manual exports sent by mail; some people consider that automation!), don't account for the possible traps (like double quotes in the value of any field), then call it a day -- more like a lifetime. Then there's inertia, there's a boss saying "we absolutely must work with these people", and... the rest is history for many of us. NOTE: I've been writing custom CSV parsers at least 10 times during my career. That makes roughly 2 such parsers every 3 years. And that comes from a guy that's not very active in his career.

astrobe_ 10y ago
Not just with CSV. I remember a similar comment about broken XML. It actually doesn't matter how compliant you are to the standard. Standards are certainly nice when both parties are willing to agree on things. But when one party is in position of strength and wants to minimize its costs or efforts, compliance to a standard won't help you.
- pdimitar 10y ago
  That's the sad truth. Sometimes the thought of this makes me wanna go full Hitler on the internet -- if you can't expose a standards-compliant API, your service will be blocked by central tech authorities. We can dream, right? Still though, all giants eventually fall when they become phlegmatic. Sadly it's taking way too long.

jbax 10y ago

univocity-parsers can handle this sort of problem without a hitch: https://github.com/uniVocity/univocity-parsers https://github.com/uniVocity/univocity-parsers You should be able to process any sort of record with unescaped quotes.

lightbendover 10y ago

Writing a custom parser for "data files similar-to but different-from CSV" is day-to-day life in B2B.