2 ms·
Sounds like a good avenue to research, but you probably want to generate more than 2 tokens ahead. Try 20 tokens, but I suppose you don't want N^20 executions o
by immibis 3y ago
Sounds like a good avenue to research, but you probably want to generate more than 2 tokens ahead. Try 20 tokens, but I suppose you don't want N^20 executions of the LLM, but more like a representative sampling of say 200 combinations of the next 20 tokens. I don't know how you'd do that.