3 ms·

I'm confused how the ASCII check works. It's checking if any byte is greater than zero. Wouldn't you want to check if the 8th bit in any byte is set?

• aeruder 8y ago
  The trick here is that each byte is treated as a signed 8-bit number. When the top bit is set, the number is negative.
  • zbjornson 8y ago
    Oh duh, thanks. (Checking less than zero, read it wrong.) I think it would be faster to OR the entire string with itself, then finally check the 8th bit though. On Skylake that would cut it to 0.33 cycles per 16 bytes (HSW 1 per 16). https://github.com/lemire/fastvalidate-utf-8/pull/2 https://github.com/lemire/fastvalidate-utf-8/pull/2
    • vardump 8y ago
      > I think it would be faster to OR the entire string with itself, then finally check the 8th bit though. The string could have NUL (zero) bytes in between.
      • zbjornson 8y ago
        You're right that it changes the behavior vs. what the current implementation is, but 0x0 is a valid ASCII character.
        • vardump 8y ago
          While you're technically right NUL is a part of ASCII set in practise it's rarely wanted in the data.
    • deleted 8y ago
      [deleted]
    • Sharlin 8y ago
      Depends on your input. If non-ASCII strings are frequent and likely to contain a non-ASCII character fairly close to the start of the string, then it makes sense to short circuit.
      • loeg 8y ago
        The previous >0 algorithm didn't short-circuit. There is no change to short-circuit behavior here.
        • Sharlin 8y ago
          Ah, I was thinking of the naive implementation in the previous post [1]. [1] https://lemire.me/blog/2018/05/09/how-quickly-can-you-check-that-a-string-is-valid-unicode-utf-8/ https://lemire.me/blog/2018/05/09/how-quickly-can-you-check-...