3 ms·
> Another important finding: Terry is by far the most popular name for a tortoise, followed by Turbo and Toby. Harry is a favorite for hares. All LLMs are lovin
by gwern 2y ago
> Another important finding: Terry is by far the most popular name for a tortoise, followed by Turbo and Toby. Harry is a favorite for hares. All LLMs are loving alliteration.
Mode-collapse. One reason that the tuned (or tuning-contaminated models) are bad for creative writing: every protagonist and place seems to be named the same thing.
- diggan 2y agoCouldn't you just up the temperature/change some other parameter to get it to be more random/"creative"? It wouldn't be active/intentional randomness/novelty like what a human would do, but at least it shouldn't generate exactly the same naming.
- gwern 2y agoNo. The collapse is not as simple as simply shifting down most of the logits, so ramping up the temperature does little until outputs start degenerating.