3 ms·
It's a lot simpler than that. You can see in the tokenizer that the boundary for words includes the preceding space. So, since the first word doesn't have a pre
by hn_throwaway_99 4y ago
It's a lot simpler than that. You can see in the tokenizer that the boundary for words includes the preceding space. So, since the first word doesn't have a preceding space, it has a different token.