4 ms·
> For example, "All CSVs contains a single consistent encoding". If this is not the case I'd (rightfully) reject the file as being a proper CSV file. "CSV" onl
by masklinn 9y ago
> For example, "All CSVs contains a single consistent encoding". If this is not the case I'd (rightfully) reject the file as being a proper CSV file.
"CSV" only specify the metacharacters (at best), unless it's explicitly stated that the CSV is a single consistent encoding it's no more improper than inconsistently encoded Linux paths. It's stupid and a pain in the ass to be sure, but it's not improper.
And when you don't control the entire pipeline, you will eventually hit this issue.
In fact the first falsehood programmers believe about CSVs is that it's a good serialisation, interchange (between separate systems) or human-edition format. While it's routinely used for all of those it's also absolute garbage at all of them.
- xienze 9y ago> In fact the first falsehood programmers believe about CSVs is that it's a good serialisation, interchange (between separate systems) or human-edition format. I would argue that most programmers realize CSV sucks and would rather use something structured and sane. In my experience CSV support almost always comes about because someone from another business unit (e.g., marketing) needs self-service import facilities. Those folks live in Excel and CSV becomes the de facto import format.
- masklinn 9y ago> Those folks live in Excel and CSV becomes the de facto import format. Which is an error, and you'd save yourself much pain if you just worked directly with excel files. Most languages have libraries allowing that.
- paulryanrogers 9y agoIME none works exactly like Excel and therefore will inevitably fail as users push more and more of it's edges and bugs on the system
- tinus_hn 9y agoCsv files don’t work like excel either. Typically all you need is a table with headings and then text and number fields.
- paulryanrogers 9y agoCSV is simpler and therefore harder for Excel to screw up. Not for lack of trying: exponential values often truncated, Windows 1252 encoding, BOM, etc.
- emodendroket 9y agoHeaven help you if whatever you're working in doesn't, though.
- majewsky 9y agoThen you make a wrapper program in a language that has an Excel library which e.g. takes your own data in some structured format (JSON, YAML, or whatever) and emits Excel.
- rhapsodic 9y ago>"CSV" only specify the metacharacters (at best), unless it's explicitly stated that the CSV is a single consistent encoding it's no more improper than inconsistently encoded Linux paths. It's stupid and a pain in the ass to be sure, but it's not improper. I think it's generally agreed that a CSV file is a plain text file. And I'm not aware of any school of thought that says that valid text files may have multiple encodings. IMO, it's ridiculous to expect any software to detect and accommodate that.
- lmkg 9y agoI don't expect a CSV to be a plain text file, unless all the data it's encoding is plain text. CSV is a data interchange format, not text. If I open a CSV in a plain text editor and get junk out of it, I would be mildly surprised but I wouldn't immediately conclude the file is invalid or corrupt, I would at least entertain the notion that the data is correct but indisplayable. Mixed encodings would make sense if the columns in a single CSV are coming from multiple data sources. Any data processing software that expects to work in the real world is expected to be able to handle inputs in different formats. Depending on the relationship between the parts of the processing pipeline, a trying to reconcile formatting issues before/while creating the CSV may be an objectively incorrect design decision.
- rhapsodic 9y ago>I don't expect a CSV to be a plain text file, unless all the data it's encoding is plain text. Really? So you might write a floating point value to a "CSV" file by writing the IEEE 754 bit pattern for that value? And you would call that a valid CSV file, and would expect generic CSV parsers to properly decode that value? That's a new one on me. I have always been under the impression that CSV files are plain text. Perhaps you should edit the Wikipedia page so it doesn't continue to give unsuspecting people the wrong information: In computing, a comma-separated values (CSV) file is a delimited text file that uses a comma to separate values (many implementations of CSV import/export tools allow other separators to be used). It stores tabular data (numbers and text) in plain text. [1] [1] https://en.wikipedia.org/wiki/Comma-separated_values https://en.wikipedia.org/wiki/Comma-separated_values