In general, I noticed significantly reduced vocabulary variety in the chat/instruct GPT-3 model vs the DaVinci text completion one even at the highest temperatu

In general, I noticed significantly reduced vocabulary variety in the chat/instruct GPT-3 model vs the DaVinci text completion one even at the highest temperatures. The research field has been really harmed by the SotA model being locked behind only accessing the RLHF fine tuned chat model. I have a suspicion that as the coming year sees comparable performance to GPT-4 in models with the pretrained layer directly available to researchers one of the big discoveries will be that for sufficiently advanced models heavy handed fine tuning does a lot more damage than is being realized. We've gone well down the Goodhart's Law rabbit hole where we evaluate a narrow scope of applications for LLMs (mostly solving word problems), then use that as the target for fine tuning, and completely miss the issue which arises in introducing "unknown unknowns" that aren't being evaluated or searched for. I was blown away by GPT-4 in its pre-release state, and while still impressed by the advances in reasoning over the past year, the model itself is far less interesting and compelling to what it used to be. I saw it fed a scenario where a child's life was allegedly in danger, and its response hit a content filter, so instead of actually using the prompt part of the formatting to suggest user responses it continued trying to encourage the user to seek poison control help in the prompt suggestions. A LLM triaging the conversation such that it breaks its own in context formatting rules to try to save a child's life is probably the most mind blowing thing I've seen a LLM do to date, and that's almost certainly been lost in the fine tuning by now. We don't have a "thinks outside the box to bend rules in order to save lives" test that we are using to measure models or target as a measurement to improve. And it's just one indication of the breadth of advanced modeling that was taking place in the versions closer to the pretrained layer that have been stripped out in establishing a predictably sterile chatbot product. Even in the current models 'emotional' language improves performance - but I suspect we've lost a great deal of capability in what could have been squeezed out by now in stripping it down more and more to our expectations for the tech rather than the reality.