4 ms·
I think that would be an interesting API design to explore, absolutely. I think you'll have a lot of issues making it fast though. There is a fair bit of SIMD g
by burntsushi 4y ago
I think that would be an interesting API design to explore, absolutely. I think you'll have a lot of issues making it fast though. There is a fair bit of SIMD going on under the hoods in both the substring routines and the UTF-8 validation routines, for example. Building APIs based around iterator adapters that munch one byte at a time are difficult to square with SIMD optimizations that want to operate on a whole bunch of bytes at a time.
Consider, for example, how you might use a routine like memchr[1] if all of your public APIs are generic iterator adapters.
And then once you get into things like regex engines, modifying them to work on Iterator<Item=u8> is a highly non-trivial affair. It is of course possible to write a regex engine that works on such things, but it's going to be limited in performance or capabilities. The way to make regexes and streaming work together is probably something more like Iterator<Item=&[u8]> (which is perhaps roughly analogous to what Hyperscan does). You really want blocks of bytes, not one-byte-at-a-time.
[1]: https://github.com/BurntSushi/memchr/blob/8037d11b4357b0f07be2bb66dc2659d9cf28ad32/src/memchr/x86/sse2.rs#L16 https://github.com/BurntSushi/memchr/blob/8037d11b4357b0f07b...