3 ms·
I think you would get slightly better generations from the model if the input data were tokenized and token IDs were sampled instead.
by ronsor 3y ago
I think you would get slightly better generations from the model if the input data were tokenized and token IDs were sampled instead.
- canjobear 3y agoPossibly. It would definitely make it even slower though. To sample from the model, if you have a prompt/training data c, you have to compute the compressed length of all strings cx for tokens x in your vocabulary. If you have a long prompt c (like 1 million lines of wikipedia) and a big vocabulary of tokens (like 100,000 different words), you're going to have a bad time.
- vintermann 3y agoIt should be possible to save and reuse the state of the compressor right before the final byte.