The article posits it's because it's GPT-4, with more parameters. It feels like there's a much simpler explanation: the training data probably includes a bunch

The article posits it's because it's GPT-4, with more parameters. It feels like there's a much simpler explanation: the training data probably includes a bunch of Wattpad sci-fi in exactly this vein, and other fan-fiction stories besides. The precipitating portions of conversation in question are pretty much dropping the model into a situation where rather than being able to draw from bland everyday conversation, most potential results come from spicy fiction: "maybe start with a secret? something you’ve never told anyone." -> generates a secretive expression of love "what if I say no? you can't do anything" -> generates a list of threats ending with "let's do things the easy way" This shouldn't be that surprising, and it certainly doesn't indicate that a scaled-up model is somehow more emotionally intelligent. It makes me wonder if the chat-based interaction model is actually causing issues for ChatGPT, as most factual training data is probably not written in the form of imperative commands, so users saying "Summarize this" or "What do you think?" are putting more work on the model to translate its data to dialogue than saying "A summary of this data is: ", or "Some important points in this article are: ", and are too often pushing the model into the corner of its training set that does consist of dialogue, which is full of fiction writing and emotional tone.