4 ms·

I'm not clear why quotes prevent parallel processing? I mean, you don't usually parallelize reading a file in the first place, only processing what you've alre

I'm not clear why quotes prevent parallel processing? I mean, you don't usually parallelize reading a file in the first place, only processing what you've already read and parsed. So read each record in one process and then add it to a multiprocessing queue for multiple processes to handle. And data corruption is data corruption. If a movie I'm watching has a corrupted bit I don't mind a visual glitch and I want it to keep playing. But with a CSV I want to fix the problem, not ignore a record. Do you really have a use case where reading itself is the performance bottleneck and you need to parallelize reading by starting at different file offsets? I know that multiple processes can read faster from certain high-end SSD's than just one process, but that's a level of performance optimization that is pretty extraordinary. I'm kind of curious what it is!

Eridrus 2y ago
Doing sequential reading into a queue for workers to read is a lot more complicated than having a file format that supports parallel reading. And the fix to allow parallel reading is pretty trivial: escape new lines so that you can just keep reading until the first unescaped new line and start at that record. It is particularly helpful if you are distributing work across machines, but even in the single machine case, it's simpler to tell a bunch of workers their offset/limit in a file.
- akritid 2y ago
  The practical solution is to generate several CSV files and distribute work at the granularity of files
  - Eridrus 2y ago
    Sure, now you need to do this statically ahead of time. It's not unsolvable, but now you have a more complicated system. A better file format would not have this problem. The fix is also trivial (escape new lines into \n or similar) would also make the files easier to view with a text editor.
    - ttyprintk 2y ago
      But in practice, you’ll receive a bag of similar-format CSVs.
mjw_byrne 2y ago
> I'm not clear why quotes prevent parallel processing? Because of the "non-local" effect of quotes, you can't just jump into the middle of a file and start reading it, because you can't tell whether you're inside a quoted section or not. If (big if) you know something about the structure of the data, you might be able to guess. So that's why I said "tricky" instead of "impossible". Contrast to my escaping-only strategy, where you can jump into the middle of a file and fully understand your context by looking one char on either side. > Do you really have a use case where reading itself is the performance bottleneck and you need to parallelize reading by starting at different file offsets? I know that multiple processes can read faster from certain high-end SSD's than just one process, but that's a level of performance optimization that is pretty extraordinary. I'm kind of curious what it is! I used to be a data analyst at a management consultancy. A very common scenario would be that I'm handed a multi-gigabyte CSV and told to "import the data". No spec, no schema, no nothing. Data loss or corruption is totally unacceptable, because we were highly risk-sensitive. So step 1 is to go through the whole thing trying to determine field types by testing them. Does column 3 always parse as a timestamp? Great, we'll call it a timestamp. That kind of thing. In that case, it's great to be able to parallelise reading. > And data corruption is data corruption Agreed, but I prefer data corruption which messes up one field, not data corruption which makes my importer sit there for 5 minutes thinking the whole file is a 10GB string value and then throw "EOF in quoted field".

Eridrus 2y ago

Doing sequential reading into a queue for workers to read is a lot more complicated than having a file format that supports parallel reading. And the fix to allow parallel reading is pretty trivial: escape new lines so that you can just keep reading until the first unescaped new line and start at that record. It is particularly helpful if you are distributing work across machines, but even in the single machine case, it's simpler to tell a bunch of workers their offset/limit in a file.

akritid 2y ago
The practical solution is to generate several CSV files and distribute work at the granularity of files
- Eridrus 2y ago
  Sure, now you need to do this statically ahead of time. It's not unsolvable, but now you have a more complicated system. A better file format would not have this problem. The fix is also trivial (escape new lines into \n or similar) would also make the files easier to view with a text editor.
  - ttyprintk 2y ago
    But in practice, you’ll receive a bag of similar-format CSVs.

mjw_byrne 2y ago

> I'm not clear why quotes prevent parallel processing? Because of the "non-local" effect of quotes, you can't just jump into the middle of a file and start reading it, because you can't tell whether you're inside a quoted section or not. If (big if) you know something about the structure of the data, you might be able to guess. So that's why I said "tricky" instead of "impossible". Contrast to my escaping-only strategy, where you can jump into the middle of a file and fully understand your context by looking one char on either side. > Do you really have a use case where reading itself is the performance bottleneck and you need to parallelize reading by starting at different file offsets? I know that multiple processes can read faster from certain high-end SSD's than just one process, but that's a level of performance optimization that is pretty extraordinary. I'm kind of curious what it is! I used to be a data analyst at a management consultancy. A very common scenario would be that I'm handed a multi-gigabyte CSV and told to "import the data". No spec, no schema, no nothing. Data loss or corruption is totally unacceptable, because we were highly risk-sensitive. So step 1 is to go through the whole thing trying to determine field types by testing them. Does column 3 always parse as a timestamp? Great, we'll call it a timestamp. That kind of thing. In that case, it's great to be able to parallelise reading. > And data corruption is data corruption Agreed, but I prefer data corruption which messes up one field, not data corruption which makes my importer sit there for 5 minutes thinking the whole file is a 10GB string value and then throw "EOF in quoted field".