In this case the problem is caused by citing sources - Google finds a list of links using their existing search tech and then produces a response using RAG base

In this case the problem is caused by citing sources - Google finds a list of links using their existing search tech and then produces a response using RAG based on the search results. That’s where the problem is - it’s originally citing a Reddit post where someone recommends it as a joke, then Business Insider through citogenisis from the original story. Pure LLMs (no RAG) don’t make this mistake - Claude will tell you it’s a bad idea and will taste bad.