4 ms·

I posted this comment on lobste.rs when this came up:[1] It kind of looks like Julia’s CSV parser is cheating: https://github.com/JuliaData/CSV.jl/blob/9f6ef10

I posted this comment on lobste.rs when this came up:[1] It kind of looks like Julia’s CSV parser is cheating: https://github.com/JuliaData/CSV.jl/blob/9f6ef108d195f85daa535d23d398253a7ca52e20/src/detection.jl#L304-L309 https://github.com/JuliaData/CSV.jl/blob/9f6ef108d195f85daa5... It’s doing parallel parsing, but I’m pretty sure their technique won’t work for all inputs. Namely, they try to hop around the CSV data and chunk it up, and then parse each chunk in a separate thread AIUI. But you can’t do this in general because of quoting. If you read the code around where I linked, you can see they try to be a bit speculative and avoid common failures (“now we read the next 5 rows and see if we get the right # of columns”), but that isn’t going to be universally correct. It might be a fair trade off to make, since CSV data that fails there is probably quite rare. But either I’m misunderstanding their optimization or they aren’t being transparent about it. I don’t see this downside anywhere in the README or the benchmark article. [1] - https://lobste.rs/s/zksa0f/loading_csv_file_at_speed_limit_nvme#c_y2fkbj https://lobste.rs/s/zksa0f/loading_csv_file_at_speed_limit_n...

Sukera 6y ago
I wouldn't call that cheating, how else would you multithread this? You basically have to chunk it up and try to find a boundary and from what I can tell, they try to find out if they're in a quoted block. If they are, they find the end of the quoted block (or EOF, which shouldn't happen => broken file) or ask the user to report the bug. This is more like failsaving, not cheating.
- lifthrasiir 6y ago
  > [...] how else would you multithread this? Use a fast "frontier" thread to determine quotes and split works at the safe boundary. I'm personally okay with the speculative execution as long as its result is correct and I wouldn't consider it cheating. But it is concerning that the benchmark doesn't list all the caveats (as all proper benchmarks should do, like [1], which is incidentally written by who you are replying to). I can't even determine whether the machine is hitting I/O bound or not from the OP! [1] https://blog.burntsushi.net/ripgrep/ https://blog.burntsushi.net/ripgrep/
- liuliu 6y ago
  > how else would you multithread this? Doing two pass parsing: https://liuliu.me/eyes/loading-csv-file-at-the-speed-limit-of-the-nvme-storage/ https://liuliu.me/eyes/loading-csv-file-at-the-speed-limit-o...
- burntsushi 6y ago
  Multi-threading is a means, not an end. It's cheating if the optimization sacrifices correctness, especially in a meaningful way that is not documented clearly. You cannot parallelize csv parsing by chunking like this. The format prohibits it, full stop. This is why xsv comes with an `xsv index` command for example. Like I said, either I'm misunderstanding the optimization they are doing or they aren't being transparent. From a glance at the code, it looks like the latter to me. (As others have mentioned, you can parallelize csv parsing, but it requires multiple passes. Which is basically what `xsv index` is. It lets you choose when to do the first pass, and then benefit in performance in all subsequent runs.)

Sukera 6y ago

I wouldn't call that cheating, how else would you multithread this? You basically have to chunk it up and try to find a boundary and from what I can tell, they try to find out if they're in a quoted block. If they are, they find the end of the quoted block (or EOF, which shouldn't happen => broken file) or ask the user to report the bug. This is more like failsaving, not cheating.

lifthrasiir 6y ago
> [...] how else would you multithread this? Use a fast "frontier" thread to determine quotes and split works at the safe boundary. I'm personally okay with the speculative execution as long as its result is correct and I wouldn't consider it cheating. But it is concerning that the benchmark doesn't list all the caveats (as all proper benchmarks should do, like [1], which is incidentally written by who you are replying to). I can't even determine whether the machine is hitting I/O bound or not from the OP! [1] https://blog.burntsushi.net/ripgrep/ https://blog.burntsushi.net/ripgrep/
liuliu 6y ago
> how else would you multithread this? Doing two pass parsing: https://liuliu.me/eyes/loading-csv-file-at-the-speed-limit-of-the-nvme-storage/ https://liuliu.me/eyes/loading-csv-file-at-the-speed-limit-o...
burntsushi 6y ago
Multi-threading is a means, not an end. It's cheating if the optimization sacrifices correctness, especially in a meaningful way that is not documented clearly. You cannot parallelize csv parsing by chunking like this. The format prohibits it, full stop. This is why xsv comes with an `xsv index` command for example. Like I said, either I'm misunderstanding the optimization they are doing or they aren't being transparent. From a glance at the code, it looks like the latter to me. (As others have mentioned, you can parallelize csv parsing, but it requires multiple passes. Which is basically what `xsv index` is. It lets you choose when to do the first pass, and then benefit in performance in all subsequent runs.)