> I've tried to play dungeons and dragons with Chatai. It doesn't like to use any mechanical rules and keeps trying to wrap everything up like its got a train

> I've tried to play dungeons and dragons with Chatai. It doesn't like to use any mechanical rules and keeps trying to wrap everything up like its got a train to catch. Yes, that's the RLHF and other mechanisms. Raters presumably reward completions which are, well, complete and can be quickly judged as a whole, no matter what the intrinsic quality might be or how valid it would be to end with a 'Tune in next week for chapter 2'. What OP is describing is the emergent behavior of the base model, which acts very differently from the RLHF/instruction-tuned/who-knows-what-else ChatGPT web interface. (This is one reason I tend to avoid that in favor of the Playground and direct model access: there's much less moving machinery behind the scenes.) What the additional stuff does is quite hard to understand because the original prediction objective has been replaced by a bunch of mashed-together objectives combined with feedback loops, leading to some bizarre behavior like being unable to reliably write a nonrhyming poem. (Give that a try in ChatGPT: "write a nonrhyming poem". It's been slooowly getting better at it, perhaps because I and other people keep submitting examples of it failing to do so - but it will still usually fail! and when it does seem to be succeeding finally, if you let it keep writing lines, it will generally gradually revert back to rhyming.) If you go back to the original davinci-001, you'll find that it acts strikingly different than your description of contemporary ChatGPT. OP, incidentally, has some discussion of what it is like to interact with the real GPT-4, which is not like ChatGPT-4: https://www.lesswrong.com/posts/tbJdxJMAiehewGpq2/impressions-from-base-gpt-4 https://www.lesswrong.com/posts/tbJdxJMAiehewGpq2/impression... AFAIK, this is one of the only discussions online of what GPT-4-base is like qualitatively. (Note that it sounds a lot like Sydney - if you were around for that, the original Bing Sydney turned out to be a GPT-4 snapshot from partway through training which hadn't been RLHFed but given only some extremely inadequate custom Bing training.)