3 ms·
This is a failure of their encoder. It should encode that as five separate tokens rather than the special endoftext token.
by sillysaurusx 4y ago
This is a failure of their encoder. It should encode that as five separate tokens rather than the special endoftext token.
- nextaccountic 4y agoI thought they introduced ChatML exactly to avoid this kind of 'injection' (as in 'sql injection'). ChatML can encode <|things|> out-of-band, outside of the regular text flow https://github.com/openai/openai-python/blob/main/chatml.md https://github.com/openai/openai-python/blob/main/chatml.md
- quickthrower2 4y agoFeels like CS 101 data structures kind of stuff.