3 ms·
Why can't the tokenization be implicit, so we only feed bytes (or characters) to the model?
by amelius 2y ago
Why can't the tokenization be implicit, so we only feed bytes (or characters) to the model?
- killerstorm 2y agoIt can work, but you have more tokens / weaker performance. People tested it and it was worse.
- entilzha 2y ago(Author Here) Not sure what you mean by implicit? If you mean just treat bytes as tokens, one issue you run into is your sequence lengths get quite long, so compared to a regular token LLM, you can’t pack as many bytes in a batch, which means you’re pretty FLOP inefficient so scale worse. You could make the model smaller to compensate, but then the model isn’t as good.