3 ms·
> I've had to deal with CSV datasets of over a gigabyte and all libraries were kind of terrible at it Have you tried this one? https://lib.rs/crates/csvroll ht
by nextaccountic 4y ago
> I've had to deal with CSV datasets of over a gigabyte and all libraries were kind of terrible at it
Have you tried this one? https://lib.rs/crates/csvroll https://lib.rs/crates/csvroll
It calls avx2 intrinsics directly for simd (unfortunately - it's an old library in Rust time, Rust nowadays has great support for portable simd)
- anonymoushn 4y agoPortable SIMD is not that useful for string parsing as far as I know - tbl is different from pshufb in ways that usually make tbl cost an additional operation for text parsing, moving things across lanes in AVX2 or AVX-512 can be fiddly (unless you just want vcompress or a masked store, then AVX-512 has you covered) Edit: After looking over the docs, it's not clear that Rust's portable SIMD exposes tbl or pshufb *at all*, which would basically prevent it from being used for string parsing.