4 ms·

I'd be curious how this compares with the encoding_rs crate (https://github.com/hsivonen/encoding_rs https://github.com/hsivonen/encoding_rs), which also offers

by jfk13 6y ago

I'd be curious how this compares with the encoding_rs crate (https://github.com/hsivonen/encoding_rs https://github.com/hsivonen/encoding_rs), which also offers UTF-8 validation (among many other things).

cryptonector 6y ago
This is SIMD based. If the encoding_rs crate isn't SIMD based, then this thing will beat its pants off.
- burntsushi 6y ago
  It does use SIMD (when enabled). But also, the encoding_rs crate is doing more. It isn't just checking for validity. It's also doing transcoding and error detection.
  - kryps 6y ago
    As far as I can see it does not use SIMD for UTF8 validation, only likely/unlikely instrinsics, see https://github.com/hsivonen/encoding_rs/blob/e98a2096ab09c9262b2928fa22f7fcec9c7a2a00/src/utf_8.rs https://github.com/hsivonen/encoding_rs/blob/e98a2096ab09c92...
    - jfk13 6y ago
      That's not the only use of SIMD in the crate (e.g. see https://github.com/hsivonen/encoding_rs/blob/e98a2096ab09c9262b2928fa22f7fcec9c7a2a00/src/simd_funcs.rs https://github.com/hsivonen/encoding_rs/blob/e98a2096ab09c92...), but I haven't looked into exactly where/how it's used further.