4 ms·
Nice catch, seeing as this is implemented by pretty much every LLM out there, it’s probably not such a big deal?
by jerpint 3y ago
Nice catch, seeing as this is implemented by pretty much every LLM out there, it’s probably not such a big deal?
- thesz 3y agoMore or less so. The resulting vocabulary will be structurally same even if you correct for that, only indices will differ slightly. When I spotted this in my code, I did an investigation whether this affects my encoding. Turned out, it does not.