3 ms·

Google saying “multiple websites have the fact so Gemini ran with it” is extraordinarily rich. If ever there was a company which should know that “multiple web

Google saying “multiple websites have the fact so Gemini ran with it” is extraordinarily rich. If ever there was a company which should know that “multiple websites” is not a good benchmark for accuracy, it’s Google. This feels like a good parable for Google’s search these days. I’ve seen more wrong information from them than anyone lately. I wonder if they can course correct before it’s too late.

miohtama 2y ago
Does this matter? If a human would research the matter they would arrive to the same conclusion by visiting those websites. Essentially, an average person would have made the same mistake when researching unless you have access to specific dairy industry data. We might want to hold LLMs for higher standards than human, but AFAIK every LLM comed with a disclaimer "fact check yourself." This regard Grok is the best as it gives you the source list for cross referencing yourself.
- WrongAssumption 2y ago
  The why did a human blogger find the error so quickly? Why didn't his fact check determine that the claim was true? Does Gouda being responsible for 50-60% of all global cheese consumption even sound remotely believable?
- andreimackenzie 2y ago
  I think it does because a researcher can pick up context about the quality of their sources through the course of web research. The BigCo chatbot AIs are marketed to represent that BigCo, and people generally trust Google, in this case. It's good when they cite sources, but a major point of the chatbot is to abstract that legwork for most people.
- rozab 2y ago
  Actually I would have argued that any human being with an ounce of common sense would find it obvious that gouda does not account for 50% to 60% of global cheese consumption. Like, you don't need an industry report to know for sure that is patently untrue. But apparently this did not smell off to any of the many, many people who worked on the ad. That is the most baffling part to me. Are these people so bought into the hype of the product they're promoting that they just switch their brains off?

miohtama 2y ago

Does this matter? If a human would research the matter they would arrive to the same conclusion by visiting those websites. Essentially, an average person would have made the same mistake when researching unless you have access to specific dairy industry data. We might want to hold LLMs for higher standards than human, but AFAIK every LLM comed with a disclaimer "fact check yourself." This regard Grok is the best as it gives you the source list for cross referencing yourself.

WrongAssumption 2y ago
The why did a human blogger find the error so quickly? Why didn't his fact check determine that the claim was true? Does Gouda being responsible for 50-60% of all global cheese consumption even sound remotely believable?
andreimackenzie 2y ago
I think it does because a researcher can pick up context about the quality of their sources through the course of web research. The BigCo chatbot AIs are marketed to represent that BigCo, and people generally trust Google, in this case. It's good when they cite sources, but a major point of the chatbot is to abstract that legwork for most people.
rozab 2y ago
Actually I would have argued that any human being with an ounce of common sense would find it obvious that gouda does not account for 50% to 60% of global cheese consumption. Like, you don't need an industry report to know for sure that is patently untrue. But apparently this did not smell off to any of the many, many people who worked on the ad. That is the most baffling part to me. Are these people so bought into the hype of the product they're promoting that they just switch their brains off?