I’ve also been doing a lot of experimentation with gpt3 and gptj story generation recently. The problem is that although they always start off with something in

I’ve also been doing a lot of experimentation with gpt3 and gptj story generation recently. The problem is that although they always start off with something interesting, they will then begin to ramble, and finally go into an odd repetitive feedback loop. You can break this to some degree by injecting new, human generated content, into the prompt (which is what AI Dungeon does) but two problems always remain: the model has very little memory of context or previous events, and can’t plan into the future. So that while it will produce really interesting events that move the plot forward, it can only do so in response to an event that immediately precedes it. The “story” never finishes and there’s no real plot, it just meanders forward from event to event without development, long term consequences, goals or conclusion. This is because, as mentioned in the article, these models are always backwards looking and they don’t seem to build any kind internal model of the world they’re describing. And, the illusion of creativity is based on carefully constrained randomness, which, with enough brute force and luck, can produce interesting stuff, but can just as easily go off the rails. There may be some way to carefully design prompts so that models learn how to both summarize all past events, which could then used as context for subsequent events. And also, to train them how produce coherent story summaries and then use those as a nucleus to develop properly structured stories - but we’ll need to investigate :)