3 ms·
My intuition says that tokenization is a factor especially if it splits up individual move descriptions differently from other LLM's If you think about how our
by pmarreck 2y ago
My intuition says that tokenization is a factor especially if it splits up individual move descriptions differently from other LLM's
If you think about how our brains handle this data input, it absolutely does not split them up between the letter and the number, although the presence of both the letter and number together would trigger the same 2 tokens I would think