3 ms·
shameless insertion: "raw csv processing at ~20GB/s" has been demonstrated in one my project as the tooling byproduct[1] based on Rayon(great Rust data paralle
by jinmingjian 6y ago
shameless insertion:
"raw csv processing at ~20GB/s" has been demonstrated in one my project as the tooling byproduct[1] based on Rayon(great Rust data parallelism library)[2] and wrapping of modified simdcsv[3] into one simple .rs.
just a little more:
1. simdcsv has severe bugs, so do not use it beyond demo.
2. the processing model of simdcsv still has rooms to good improvements(estimated 2x more, a.k.a. ~40GB+/s in memory in single modern socket should be achievable in some scenarios(no heavy string to complex language object conversions)).
[1] https://tensorbase.io/2020/08/04/hello-base.html#benchmark https://tensorbase.io/2020/08/04/hello-base.html#benchmark
[2] https://github.com/rayon-rs https://github.com/rayon-rs
[3] https://github.com/geofflangdale/simdcsv https://github.com/geofflangdale/simdcsv
- burntsushi 6y ago> simdcsv has severe bugs, so do not use it beyond demo Are these bugs documented anywhere? A glance at the issue tracker doesn't seem to reflect this claim.