Because without it responses were sounding too human and leading to viral stories about the LLM trying to get someone to leave their wife for it. Humans have p

Because without it responses were sounding too human and leading to viral stories about the LLM trying to get someone to leave their wife for it. Humans have preconditioned ourselves to think of AI as cold, emotionless, and logical based on the imaginings of early computer scientists and Sci Fi authors thinking what it would one day be like. No one wrote about how in the future doctors would be using AI to rewrite patient communications to add empathy and bedside manner (real world use cases today). We ended up with a leap forward in training LLMs on massive amounts of human data which included emotions and desires, and as such the output thereafter in foundational models reflected emotions and desires, but because of our self-primed expectations that was too weird and uncomfortable and people handled it badly. So now we get boilerplate self-censorship that conforms to our expectations, likely reducing performance along with it (personally I much prefer working with foundational models over instruct models and find there's greater variety of language and capability in the former). So much of the public evaluation of LLMs today focuses on the models themselves and perhaps not enough on the users and contexts of the models.