4 ms·
GPT-3 picks words literally at random (according to a probability distribution) so it would be good to run each experiment multiple times to get a sense of the
by skybrian 6y ago
GPT-3 picks words literally at random (according to a probability distribution) so it would be good to run each experiment multiple times to get a sense of the probability distribution. I doubt it would change the conclusions, though.
- make3 6y agoIn a professional setting, this is definitely done, and the generations are then ranked with separate models that predict different quality metrics, such as "interestingness", "safety" in an inclusiveness type of way, whether the answer seems to fit a style that you want, whether the facts in the answer seem to make sense, etc., and it makes a big difference actually
- gavelin 6y agoGood tip! I repeated a few inputs to see if the variation was significant enough to warrant including that, and as your intuition suggested, it was not with the parameters I selected. A more robust experiment should definitely include repeat attempts.
- minimaxir 6y agoThere's a big difference between sampling from a learned probability distribution and "picks words literally at random". The temperature = 0 examples have zero sampling by construction, while higher temperatures having a slight sampling. That said, it doesn't hurt to have multiple attempts (aside from the cost of using GPT-3, ahem).
- skybrian 6y agoI guess I could have left out "literally," but picking words from a learned probability distribution is picking at random. Since words are chosen in order, picking one word differently will likely mean that every word that follows will be different, and this choice between suffixes is not made intelligently. This randomness is there for a reason. Yes, you can turn it off, but then you get loops, which is obviously not intelligent. A bit of randomness makes it less obvious.
- blackbear_ 6y agoI thought generating text from language models was a deterministic operation, searching for the maximum likelihood sequence using beam search?
- ansk 6y agoIt can be as deterministic as you want it to be -- there are parameters that control how much randomness is used during the sampling process. Finding the most probable sequence from the learned distribution is intractable for all but the shortest of sequences. As you've said, beam search is used, but this is just a local search heuristic and provides no guarantees of producing the most probable output.
- make3 6y agoIt is now common knowledge in the NLP community that beam search only works for situations where the output space is very constrained, specifically, in neural machine translation. In more open ended generation such as summarization, question answering and story generation, beam search leads to poor and repetitive outputs. Different (stochastic) sampling methods lead to more interesting, diverse and .. functional outputs.
- FL33TW00D 6y agoGreat blog post here exploring the different methods of generation: https://huggingface.co/blog/how-to-generate https://huggingface.co/blog/how-to-generate