People have been complaining about AI models surreptitiously changing underneath them for a while now, and we found evidence of it happening in the wild. We bui

People have been complaining about AI models surreptitiously changing underneath them for a while now, and we found evidence of it happening in the wild. We build an LLM monitoring and testing tool called Libretto, and we saw GPT-4o start to behave significantly differently on one of our prompts this week. This is a write-up of how we detected the change and what it means for building on top of LLMs that can change at any moment.