3 ms·

I wouldn't call that cheating, how else would you multithread this? You basically have to chunk it up and try to find a boundary and from what I can tell, they

by Sukera 6y ago

I wouldn't call that cheating, how else would you multithread this? You basically have to chunk it up and try to find a boundary and from what I can tell, they try to find out if they're in a quoted block. If they are, they find the end of the quoted block (or EOF, which shouldn't happen => broken file) or ask the user to report the bug. This is more like failsaving, not cheating.

lifthrasiir 6y ago
> [...] how else would you multithread this? Use a fast "frontier" thread to determine quotes and split works at the safe boundary. I'm personally okay with the speculative execution as long as its result is correct and I wouldn't consider it cheating. But it is concerning that the benchmark doesn't list all the caveats (as all proper benchmarks should do, like [1], which is incidentally written by who you are replying to). I can't even determine whether the machine is hitting I/O bound or not from the OP! [1] https://blog.burntsushi.net/ripgrep/ https://blog.burntsushi.net/ripgrep/
liuliu 6y ago
> how else would you multithread this? Doing two pass parsing: https://liuliu.me/eyes/loading-csv-file-at-the-speed-limit-of-the-nvme-storage/ https://liuliu.me/eyes/loading-csv-file-at-the-speed-limit-o...
burntsushi 6y ago
Multi-threading is a means, not an end. It's cheating if the optimization sacrifices correctness, especially in a meaningful way that is not documented clearly. You cannot parallelize csv parsing by chunking like this. The format prohibits it, full stop. This is why xsv comes with an `xsv index` command for example. Like I said, either I'm misunderstanding the optimization they are doing or they aren't being transparent. From a glance at the code, it looks like the latter to me. (As others have mentioned, you can parallelize csv parsing, but it requires multiple passes. Which is basically what `xsv index` is. It lets you choose when to do the first pass, and then benefit in performance in all subsequent runs.)

by Sukera 6y ago

lifthrasiir 6y ago

> [...] how else would you multithread this? Use a fast "frontier" thread to determine quotes and split works at the safe boundary. I'm personally okay with the speculative execution as long as its result is correct and I wouldn't consider it cheating. But it is concerning that the benchmark doesn't list all the caveats (as all proper benchmarks should do, like [1], which is incidentally written by who you are replying to). I can't even determine whether the machine is hitting I/O bound or not from the OP! [1] https://blog.burntsushi.net/ripgrep/ https://blog.burntsushi.net/ripgrep/

liuliu 6y ago

> how else would you multithread this? Doing two pass parsing: https://liuliu.me/eyes/loading-csv-file-at-the-speed-limit-of-the-nvme-storage/ https://liuliu.me/eyes/loading-csv-file-at-the-speed-limit-o...

burntsushi 6y ago

Multi-threading is a means, not an end. It's cheating if the optimization sacrifices correctness, especially in a meaningful way that is not documented clearly. You cannot parallelize csv parsing by chunking like this. The format prohibits it, full stop. This is why xsv comes with an `xsv index` command for example. Like I said, either I'm misunderstanding the optimization they are doing or they aren't being transparent. From a glance at the code, it looks like the latter to me. (As others have mentioned, you can parallelize csv parsing, but it requires multiple passes. Which is basically what `xsv index` is. It lets you choose when to do the first pass, and then benefit in performance in all subsequent runs.)