3 ms·
> So the logical word -> token(s, p) -> id(s) function definitely has 1 position parameter as well. You're missing that it groups in spaces. The position isn't
by fenomas 4y ago
> So the logical word -> token(s, p) -> id(s) function definitely has 1 position parameter as well.
You're missing that it groups in spaces. The position isn't relevant, but "day" is a different token than " day".
- eternalban 4y agoAh, you're right. [try "datedate". interesting how it partitions that as 'dated' + 'ate'. Compare with "matemate" -> 'mat', 'emate'.] p.s. "ifthiswasagermanword if this was a german word." It's not even spaces. That second sequence ' german' is chopped up as 'ag' 'erman'.