It's a bit weird that he doesn't provide his solution. I would be surprised if this isn't faster: pub fn is_token1(c: u8) -> bool { let mask =

It's a bit weird that he doesn't provide his solution. I would be surprised if this isn't faster: pub fn is_token1(c: u8) -> bool { let mask = if c < 32 { !0 } // c != 32 && c != 34 && c != 40 && c != 41 && c != 44 && c != 47 && !(c > 57 && c < 64) else if c < 64 { 1 + 4 + 256 + 512 + 4096 + 32768 - (1 << 25) } // c != 64 && !(c > 90 && c < 94) else if c < 96 { 1 + (1 << 30) - (1 << 26) } // c != 123 && c != 125 else if c < 128 { (1 << 27) + (1 << 29) } else { !0 }; ((mask >> (c & 31)) & 1) == 0 } or this for 64-bit: pub fn is_token2(c: u8) -> bool { // roughly follows the order of ascii chars: "\"(),/:;<=>?@[\\]{} \t" let mask = if c < 64 { (1u64 << 33) - 1 + ((4u64 + 256u64 + 512u64 + 4096u64 + 32768u64) << 32) + !((1u64 << 57) - 1) } else if c < 128 { 1 + (1 << 30) - (1 << 26) + (1u64 << 59) + (1u64 << 61) } else { !0u64 }; ((mask >> (c & 63)) & 1) == 0 } (Forgive me, it's the first time ever I write Rust...) and if it is slower, it could also be because LLVM generates a slow bt instruction.