3 ms·

Sometimes you can do this in two instructions per 16 / 32 / 64 chars (sse / avx / avx-512 resp): shuffle & compare. The gist of it is lookup[hash(char)] =

Sometimes you can do this in two instructions per 16 / 32 / 64 chars (sse / avx / avx-512 resp): shuffle & compare. The gist of it is lookup[hash(char)] == char where 'lookup[hash(char)]' corresponds to _mm_shuffle_epi8 with a carefully crafted constant and input data, and '==' corresponds to _mm_cmpeq_epi8. The intrinsic c = _mm_shuffle_epi8(a, b) pretty much computes c[i] = b[a[i] & 0x0F], so if there are no collisions in a[i] & 0x0F, you can construct a special b so that b[hash('@')] = '@' b[hash('/')] = '/' b[hash('?')] = '?' b[hash('\\')] = '\\' b[x] = 0x00 # otherwise Unfortunately in this case, it doesn't work, since `/` and `?` have a collision, they both map to `0x0F`. If you would just search for say @, / and \, it would be possible, and shuffle + cmp is sufficient.

Sesse__ 4y ago
Shift them to the right one bit :-)
- anonymoushn 4y ago
  Unfortunately there's no 8-bit srli, and if you use the 16-bit srli, you could end up shifting a 1 into the high bit of the low 8 bits of your 16-bit word. pshufb treats a 1 in the high bit as a signal to map the input to 0 instead of using the lower 4 bits as a lookup table. So the full thing ends up being (for AVX-2): _mm256_srli_epi16 // by 1 _mm256_and_si256 // to mask off the high bit _mm256_shuffle_epi8 // b[a[i] & 0x0F] _mm256_cmpeq_epi8 // a[i] == This is still pretty good! If you end up needing more instructions to get map your chosen bytes to 16 unique low nibbles though, some other technique may pull ahead. See here for some techniques: http://0x80.pl/articles/simd-byte-lookup.html http://0x80.pl/articles/simd-byte-lookup.html Note that the above article is a bit too conservative about the applicability of some of these techniques and does not end up telling you to do cmpeq(input, pshufb(constant, input)) even though this is what you want if all the bytes of interest are <128 (i.e. they are ascii characters) which is often true.

Sesse__ 4y ago

Shift them to the right one bit :-)

anonymoushn 4y ago
Unfortunately there's no 8-bit srli, and if you use the 16-bit srli, you could end up shifting a 1 into the high bit of the low 8 bits of your 16-bit word. pshufb treats a 1 in the high bit as a signal to map the input to 0 instead of using the lower 4 bits as a lookup table. So the full thing ends up being (for AVX-2): _mm256_srli_epi16 // by 1 _mm256_and_si256 // to mask off the high bit _mm256_shuffle_epi8 // b[a[i] & 0x0F] _mm256_cmpeq_epi8 // a[i] == This is still pretty good! If you end up needing more instructions to get map your chosen bytes to 16 unique low nibbles though, some other technique may pull ahead. See here for some techniques: http://0x80.pl/articles/simd-byte-lookup.html http://0x80.pl/articles/simd-byte-lookup.html Note that the above article is a bit too conservative about the applicability of some of these techniques and does not end up telling you to do cmpeq(input, pshufb(constant, input)) even though this is what you want if all the bytes of interest are <128 (i.e. they are ascii characters) which is often true.