4 ms·
You could, but you wouldn't when those keywords can all change in equivalent contexts.
by muyuu 9mo ago
You could, but you wouldn't when those keywords can all change in equivalent contexts.
- eru 9mo agoWhat do you mean? `public` might have a token by itself, even though you can have `pub` occurring in other contexts, too.
- muyuu 9mo agoI meant that it wouldn't be efficient to agglomerate tokens in that way and that's why the system won't do it
- janalsncm 9mo agoThe BPE or wordpiece tokenization algorithm will greedily take the longest valid token prefix. So if your text starts with “public static void main” it will try to find the longest token which matches that prefix. Even if “public” is a token, it will prefer to tokenize “public static” together.
- muyuu 9mo agoyes, but then you have both alternatives as tokens, which nullifies GP's argument