4 ms·

> The hash method considers the token’s first and second characters and the length. Quite a kludgey optimization for the token hash

• tptacek 5y ago
  The language isn't extensible, so these are changes that happen very rarely; the compiler is simply optimized for the actual task it has. What would be weird would be expending any real effort --- or, worse, runtime cost --- for an engineering case we know is never going to happen.
  • benhoyt 5y ago
    Yes, and specifically in this case no keywords have been added to the Go language at all since its 1.0 release (I just checked the 1.0.1 spec and there are still 25 keywords). Even the addition of generics coming soon in 1.18 will add no new keywords (though it will add a new token, "~").
• Leszek 5y ago
  V8 developer here (who happened to also implement perfect hashing in V8's tokenizer) - perfect hashing is a very common compiler optimisation, and as the sibling comment says, is worth it for the runtime scanner speed improvement. If you do end up adding new keywords (which is ~never for anything that wants to preserve backwards compat), then you just recalculate your perfect hash with gperf or by hand or however.