4 ms·

CSV isn‘t that complicated if done right. 1. if a value included the line separator, row separator or text qualifier surround the value with the text qualifier

by croes 2y ago

CSV isn‘t that complicated if done right. 1. if a value included the line separator, row separator or text qualifier surround the value with the text qualifier. 2. if the value contains the text qualifier double it in the value.

Macha 2y ago
That assumes you control both the producer and consumer. But if you're doing CSV, it's likely because you're looking to integrate with someone else's system. So you have to deal with whatever they're doing that they call CSV. And if "they" are "all your customers", you're going to encounter every weird quirk of different system's CSV parsing from that guy who just used String.split(",").map(it.replace("\"\"", "\"")) to Spark insisting that backslash escapes exist in CSV.
- croes 2y ago
  That’s the real problem. That those system claim the read or write CSV but in reality it’s a bastard form of CSV. Never had a problem when both sides know the rules. But I doubt those other system even support ASCII separated value files.
  - mannschott 2y ago
    It’s all bastard CSV out here in the real world. For extra fun consider the German-Speaking word where CSV files are actually Semicolon-Separated but everyone still calls them CSV and looks at you like you drooled on yourself when you point out that “;” is not a “,”. This appears to be because we use “,” as decimal separator and were too dense to learn how to use " properly in CSV. Bastard CSVs as far as the eye can see.
- nly 2y ago
  It's easier in most cases to just use a parser flexible enough for you to specify whatever variant the producer actually emitted.
  - Macha 2y ago
    How do you know the variant? If the user is just using an "import csv" on the web form, how do you know? You can't even ask the user because it's not even clear they'll know, it's just the CSV they got from Jira, their other vendor, whatever. And then the other direction is even worse, when you emit a RFC compliant CSV and the other party complains their batch job chokes on "" escaped quotes, etc., so you end up holding a mapping of clients -> "CSV" formats
ekianjo 2y ago
you don't always control how csv files are made. Most of the time you are just given them, and this is when you start pulling your hair. CSV is a terrible, terrible format, because it fails in too many use cases.
- croes 2y ago
  CSV doesn‘t fail, they fail to handle CSV. And I doubt those who create wrong CSVs can even handle ASVs. CSV can be read and edited with any text editor.
mrweasel 2y ago
But it's so rarely done right. I remember Klarna using ", " as their separator. Not ",". There had to be a space as well, which most CSV parsers can not do. So when giving us a CSV file, with currency, and Swedish kronor used "," as the decimal separator you'd get some fun result. Pretty much every CSV parser we tried would assume that kronor and øre was two separate fields.
- croes 2y ago
  I doubt ASV would be done right more often

by croes 2y ago

Macha 2y ago

That assumes you control both the producer and consumer. But if you're doing CSV, it's likely because you're looking to integrate with someone else's system. So you have to deal with whatever they're doing that they call CSV. And if "they" are "all your customers", you're going to encounter every weird quirk of different system's CSV parsing from that guy who just used String.split(",").map(it.replace("\"\"", "\"")) to Spark insisting that backslash escapes exist in CSV.

croes 2y ago
That’s the real problem. That those system claim the read or write CSV but in reality it’s a bastard form of CSV. Never had a problem when both sides know the rules. But I doubt those other system even support ASCII separated value files.
- mannschott 2y ago
  It’s all bastard CSV out here in the real world. For extra fun consider the German-Speaking word where CSV files are actually Semicolon-Separated but everyone still calls them CSV and looks at you like you drooled on yourself when you point out that “;” is not a “,”. This appears to be because we use “,” as decimal separator and were too dense to learn how to use " properly in CSV. Bastard CSVs as far as the eye can see.
nly 2y ago
It's easier in most cases to just use a parser flexible enough for you to specify whatever variant the producer actually emitted.
- Macha 2y ago
  How do you know the variant? If the user is just using an "import csv" on the web form, how do you know? You can't even ask the user because it's not even clear they'll know, it's just the CSV they got from Jira, their other vendor, whatever. And then the other direction is even worse, when you emit a RFC compliant CSV and the other party complains their batch job chokes on "" escaped quotes, etc., so you end up holding a mapping of clients -> "CSV" formats

ekianjo 2y ago

you don't always control how csv files are made. Most of the time you are just given them, and this is when you start pulling your hair. CSV is a terrible, terrible format, because it fails in too many use cases.

croes 2y ago
CSV doesn‘t fail, they fail to handle CSV. And I doubt those who create wrong CSVs can even handle ASVs. CSV can be read and edited with any text editor.

mrweasel 2y ago

But it's so rarely done right. I remember Klarna using ", " as their separator. Not ",". There had to be a space as well, which most CSV parsers can not do. So when giving us a CSV file, with currency, and Swedish kronor used "," as the decimal separator you'd get some fun result. Pretty much every CSV parser we tried would assume that kronor and øre was two separate fields.

croes 2y ago
I doubt ASV would be done right more often