4 ms·
It would be great if a comparison to the rust csv library or xsv [1] could be added - it's single threaded, but it has a zero copy option as far as I know. I've
by phiresky 6y ago
It would be great if a comparison to the rust csv library or xsv [1] could be added - it's single threaded, but it has a zero copy option as far as I know. I've used it to parse / filter large PostgreSQL log files (20GB+) with great success.
[1] https://github.com/BurntSushi/xsv https://github.com/BurntSushi/xsv
- burntsushi 6y agoAs the author of xsv, I was surprised to see this comparison missing too. The other bit to mention is `xsv index`. If you can afford to make one single threaded pass over the CSV data, then future operations truly do become embarrassingly parallel. It's a simple low-tech solution, although the UX is worse because you need to take a manual step. As for zero copy, could you say where you saw that? I don't think it makes sense to describe xsv that way. The csv parser does have a lower level zero allocation API, but there is no zero copy API.
- phiresky 6y agoYeah, I meant both the ByteRecord/read_byte_record API, as well as using Serde with a struct containing &str or &[u8] references.
- burntsushi 6y agoAh yeah, those are zero allocation. A copy is always made to account for quoting/escaping. You can implement a zero copy csv parser, but at some point you have to deal with quoting. The csv crate chooses to eagerly deal with it, always.