The lexer reads the longest tokens it can (the "maximum munch" rule mentioned in another answer). Once it has read the 0, it has decided to read a number. It wi

The lexer reads the longest tokens it can (the "maximum munch" rule mentioned in another answer). Once it has read the 0, it has decided to read a number. It will then proceed to read the longest possible number at this position, which is 0xf. This is simpler to specify and implement than the alternative that would require looking ahead to see where to split.