3 ms·
Yes, but it would hurt its contextual understanding and effectively reduce the context window several times.
by curioussquirrel 1y ago
Yes, but it would hurt its contextual understanding and effectively reduce the context window several times.
- viraptor 1y agoOnly in the current most popular architectures. Mamba and RWKV style LLMs may suffer a bit but don't get a reduced context in the same sense.
- curioussquirrel 1y agoYou're right. There was also an experiment in Meta which tokenized bytes directly and it didn't hurt performance much in very small models.