3 ms·

>but in practice I see very few problems arising with using CSV That is not my experience at all. I've been processing CSV files from financial institutions fo

>but in practice I see very few problems arising with using CSV That is not my experience at all. I've been processing CSV files from financial institutions for many years. The likelihood of brokenness must be around 40%. It's unbelievable. The main reason for this is not necessarily the CSV format as such. I believe the reason is that it is often the least experienced developers who are tasked with writing export code. And many inexperienced developers seem to think that they can generate CSV without using a library because the format is supposedly so simple. JSON is better but it doesn't help with things like getting dates right. XML can help with that but it has complexities that people get wrong all the time (such as entities), so I think JSON is the best compromise.

kragen 2y ago
> And many inexperienced developers seem to think that they can generate CSV without using a library because the format is supposedly so simple. Can't they? def excel_csv_of(rows): for row in rows: for i, field in enumerate(row): if i: yield ',' yield '"' for c in field: yield '""' if c == '"' else c yield '"' yield '\n' I haven't tested this, even to see if the code parses. What did I screw up?
- gthompson512 2y ago
  This forces each field to be quoted, and it assumes that each row has the same fields in the same order. A library can handle the quoting issues and fields more reliably. Not sure why you went with a generator for this either. Most people expect something like `12,,213,3` instead of `"12","213","3"` which yours might give. https://en.wikipedia.org/wiki/Comma-separated_values#Basic_rules https://en.wikipedia.org/wiki/Comma-separated_values#Basic_r...
  - kragen 2y ago
    Forcing each field to be quoted is always correct, isn't it? How could something be "more reliable" than something that is always correct? With respect to "the same fields in the same order", no, although you may or may not feed the CSV to an application that has such an expectation. But if you apply it to data like [("Points",),(),("x","y"),("3","4"),("6","8","10")] it will successfully preserve that wonky structure in a file Excel can ingest reliably. (As reliably as Excel can ingest anything, anyway, since Excel has its own Norway problem.) It's true that it's possible to produce more optimized output, but I didn't claim that the output was optimal, just correct. Using generators is necessary to be able to correctly output individual fields that are many times larger than physical memory.
- fauigerzigerk 2y ago
  >Can't they? If my experience reflects a relevant sample then the answer is that most can but a very significant minority fails at the job (under the given working conditions). Whether or not _you_ can is a separate question. I don't see anything wrong with your code. It does of course assume that whatever is contained in rows is correct. It also assumes that the result is correctly written to a file without making any encoding mistakes or forgetting to flush the stream. Not using name value pairs makes CSV more prone to mistakes such as incorrect ordering or number of values in some rows, a header row that doesn't correspond with the data rows, etc. Some export files are merged from multiple sources or go through many iterations over many years, which makes such mistakes far more likely. I have also seen files that end abruptly somewhere in the middle. This isn't specific to CSV but it is specific to not using libraries and not using libraries appears to be more prevalent when people generate CSV. You'd be surprised how many CSV files are out there where the developer tried to guess incorrectly whether or not a column would ever have to be escaped. Maybe they were right initially and it didn't have to be escaped but then years later something causes a change in number formats (internationalisation) and bang, silent data corruption. Prioritising correctness and robustness over efficiency as you have done is the best choice in most situations. Using a well tested library is another option to get the same result.

kragen 2y ago

> And many inexperienced developers seem to think that they can generate CSV without using a library because the format is supposedly so simple. Can't they? def excel_csv_of(rows): for row in rows: for i, field in enumerate(row): if i: yield ',' yield '"' for c in field: yield '""' if c == '"' else c yield '"' yield '\n' I haven't tested this, even to see if the code parses. What did I screw up?

gthompson512 2y ago
This forces each field to be quoted, and it assumes that each row has the same fields in the same order. A library can handle the quoting issues and fields more reliably. Not sure why you went with a generator for this either. Most people expect something like `12,,213,3` instead of `"12","213","3"` which yours might give. https://en.wikipedia.org/wiki/Comma-separated_values#Basic_rules https://en.wikipedia.org/wiki/Comma-separated_values#Basic_r...
- kragen 2y ago
  Forcing each field to be quoted is always correct, isn't it? How could something be "more reliable" than something that is always correct? With respect to "the same fields in the same order", no, although you may or may not feed the CSV to an application that has such an expectation. But if you apply it to data like [("Points",),(),("x","y"),("3","4"),("6","8","10")] it will successfully preserve that wonky structure in a file Excel can ingest reliably. (As reliably as Excel can ingest anything, anyway, since Excel has its own Norway problem.) It's true that it's possible to produce more optimized output, but I didn't claim that the output was optimal, just correct. Using generators is necessary to be able to correctly output individual fields that are many times larger than physical memory.
fauigerzigerk 2y ago
>Can't they? If my experience reflects a relevant sample then the answer is that most can but a very significant minority fails at the job (under the given working conditions). Whether or not _you_ can is a separate question. I don't see anything wrong with your code. It does of course assume that whatever is contained in rows is correct. It also assumes that the result is correctly written to a file without making any encoding mistakes or forgetting to flush the stream. Not using name value pairs makes CSV more prone to mistakes such as incorrect ordering or number of values in some rows, a header row that doesn't correspond with the data rows, etc. Some export files are merged from multiple sources or go through many iterations over many years, which makes such mistakes far more likely. I have also seen files that end abruptly somewhere in the middle. This isn't specific to CSV but it is specific to not using libraries and not using libraries appears to be more prevalent when people generate CSV. You'd be surprised how many CSV files are out there where the developer tried to guess incorrectly whether or not a column would ever have to be escaped. Maybe they were right initially and it didn't have to be escaped but then years later something causes a change in number formats (internationalisation) and bang, silent data corruption. Prioritising correctness and robustness over efficiency as you have done is the best choice in most situations. Using a well tested library is another option to get the same result.