3 ms·
[ full disclosure: Intel employee, working in software that does exactly this sort of thing ] I would suggest that the best way to generally hunt down a charac
by glangdale 9y ago
[ full disclosure: Intel employee, working in software that does exactly this sort of thing ]
I would suggest that the best way to generally hunt down a character class - i.e. white space is itself an 'algorithmic' problem and a mildly interesting one. If you have a fast permute handy this can be done on both x86 and ARM.
We have some code here that does this, but it's not the easiest to understand (this is the run-time only):
https://github.com/01org/hyperscan/blob/master/src/nfa/shufti.c https://github.com/01org/hyperscan/blob/master/src/nfa/shuft...
The idea is that you represent a character class by intersecting two PSHUFB (or whatever floats your boat) results. So if you want to do a real regex-style \s, you will probably have to use one "bucket" to represent the low-order stuff (like \r and \n) and one for \x20. You then use 1 PSHUFB to look up your bucket for the high 4 bits (aka nibble) and the other PSHUFB to look up the low 4 bits.
This can't represent all character classes - it's possible to run out of buckets. There's an obvious way of doing all character classes with 3 shuffles (see truffle.c in the same directory).
As another commenter says, it's a lot harder to do this for UTF-8 as you are looking for bigger data items, variable-sized data items, and potentially a more generous definition of whitespace.
- glangdale 9y agoTo expand on the UTF-8 point - assuming you are using "UCP" type definitions of whitespace where you start looking for "Mathematical Space" (codepoint 0x205f) and "Ideographic Space" (codepoint 0x3000) and a good couple dozen other friendly spaces.... you can do a trick similar to what I describe if you really think that SIMD is going to pay off here (hey, maybe you're doing a lot of "6 EM Space" processing). It's still potentially doable in SIMD and the comparison is, well, what? One-char-at-a-time? You will need more shuffles and/or more buckets.