3 ms·
Random thought: right now an LLM returns a probabilities distribution, an RNG sampler picks one and apoends it to the output, then the sequence repeats; but can
by akomtu 3y ago
Random thought: right now an LLM returns a probabilities distribution, an RNG sampler picks one and apoends it to the output, then the sequence repeats; but can the RNG instead pick N tokens that approximate the distribution, ask LLM to generate N new distributions, combine them somehow, then pick another set of N tokens from the combined dustribution?
- quickthrower2 3y agoNovice here. I like the sound of that! I don't know the answer but I might experiment with it. Probably a researcher has tried it. You would need N times the compute per token generate of course. You could either pick to N, or sample N (with temperature adjustment to logits if needed)
- immibis 3y agoSounds like a good avenue to research, but you probably want to generate more than 2 tokens ahead. Try 20 tokens, but I suppose you don't want N^20 executions of the LLM, but more like a representative sampling of say 200 combinations of the next 20 tokens. I don't know how you'd do that.
- jauntbox 3y agoThis sounds pretty much like beam search (https://en.wikipedia.org/wiki/Beam_search https://en.wikipedia.org/wiki/Beam_search), which is in fact a common generation technique! See eg. https://huggingface.co/docs/transformers/internal/generation_utils https://huggingface.co/docs/transformers/internal/generation...