3 ms·

I wrote a Julia package to do this some time ago (https://github.com/jakobnissen/ScanByte.jl https://github.com/jakobnissen/ScanByte.jl). It selects the right a

I wrote a Julia package to do this some time ago (https://github.com/jakobnissen/ScanByte.jl https://github.com/jakobnissen/ScanByte.jl). It selects the right algorithm at compile time depdending on the chosen byte set and the available instructions. It turns out that for specific sets, you can significantly better than the generic algoritm. For example, to find the location of any byte in `abc`, this is the inner loop generated by Julia, checking 32 bytes per iteration: L48: vmovdqu 8(%rsi,%rcx), %ymm1 vpshufb %ymm1, %ymm0, %ymm2 vpcmpeqb %ymm2, %ymm1, %ymm1 vpmovmskb %ymm1, %edx testl %edx, %edx jne L83 addq $32, %rcx cmpq %rcx, %rax jae L48

anonymoushn 4y ago
This is super cool! As a user of software that is slow, I'd really like these techniques to be used more broadly, and while I have plans to tell people about them I don't have much hope for "telling people about techniques they can use in low-level languages" translating into fast parsers in software I spend my time waiting for anytime soon. Consider e.g. that simdjson has not been wrapped and highly adopted in many high-level languages. Your thing just generates the correct instructions without the user having to be a knower or a user of a library implementing a specific parser in terms of these techniques. Great!