4 ms·

> In conclusion, I’m pretty negative about AI in general, but this experience made me cautiously optimistic about the role the technology could play in protecti

> In conclusion, I’m pretty negative about AI in general, but this experience made me cautiously optimistic about the role the technology could play in protecting consumers from counterfeit products. Whaaaattt? How could the author go through such detail and discover that Gemini completely failed in the assigned task (calling all the products fakes when some were and some weren't means that it provided zero correct assistance in this discrimination task) and then conclude "AI could play a helpful role"??? Does the author also conclude that stopped clocks could play a helpful role telling time because the are correct twice a day? Apparently AI isn't the only thing that hallucinates.

Terr_ 2mo ago
Perhaps a softer critique would be that the author seems to assume that (A) feature-detection will map to (B) accurate and useful conclusions, and that isn't necessarily true.
muppetman 2mo ago
That was my thought too. It failed amazingly at the task yet they were still thinking it was helpful. I had to go back and reread bits because I thought I’d missed a bit where Gemini hadn’t cocked up. Very odd conclusion to an otherwise interesting experiment.
wgrover 2mo ago
I think my "cautious optimism" mostly came from the fact that Gemini found subtle problems with the counterfeit packages that I (someone who looks at fakes as part of my job) didn't notice - like mismatched info between the tube and box, and a malformed Irish postal code. If I didn't notice those, then I seriously doubt that a consumer glancing at a package would notice them. In that case, Gemini's insights would be valuable info for a consumer. Gemini wrongly called an authentic product a fake, but that was mainly because there really were typographical errors in the authentic product's ingredient list. That's more of an indictment of the manufacturer than it is Gemini. Finally, a lot of the things that Gemini got wrong seemed to me like they could reasonably be attributed to things like optical artifacts in the photos (glare, shadows, stuff like that). Better/more photos might improve that. All that being said, this is obviously a tiny study of a single AI tool with a single brand of cosmetic product, so it's probably premature for me to optimistic, even cautiously. I've edited the last section of the writeup accordingly.
- hn_throwaway_99 2mo ago
  Thanks for the response, and I was overly harsh, I apologize. I did appreciate the detail and thoroughness you went to explain your experiences. Yet I still think your study was a microcosm of the greatest dangers I think about AI. That is, it was astoundingly good at doing small scale feature detection, but astoundingly bad at synthesizing an overall conclusion, and worse, it did so with characteristic "AI certainty". Also, in the real world just like you found, pictures have glare, and real manufacturers have mistakes. The horrifyingly scary thing is that even if you think Gemini did a fairly good job at feature detection, in the real world people have and will just follow the AI conclusions blindly because they're "mostly" correct, even when they lead to completely wrong outcomes. Heck, a Tesla already killed its passenger when it rammed into the side of a truck a few years ago because it mistook glare on the truck for the sun. Military planners blew up a school of young children based on old data, yet the Pentagon tried to blacklist Anthropic because Anthropic didn't want to provide autonomous kill capabilities. I don't mean to sound over dramatic, but again, to me your study highlights everything I think is wrong with AI and the extreme dangers it will cause if society relies on it too much, which it has already begun to do.

Terr_ 2mo ago

Perhaps a softer critique would be that the author seems to assume that (A) feature-detection will map to (B) accurate and useful conclusions, and that isn't necessarily true.

muppetman 2mo ago

That was my thought too. It failed amazingly at the task yet they were still thinking it was helpful. I had to go back and reread bits because I thought I’d missed a bit where Gemini hadn’t cocked up. Very odd conclusion to an otherwise interesting experiment.

wgrover 2mo ago

I think my "cautious optimism" mostly came from the fact that Gemini found subtle problems with the counterfeit packages that I (someone who looks at fakes as part of my job) didn't notice - like mismatched info between the tube and box, and a malformed Irish postal code. If I didn't notice those, then I seriously doubt that a consumer glancing at a package would notice them. In that case, Gemini's insights would be valuable info for a consumer. Gemini wrongly called an authentic product a fake, but that was mainly because there really were typographical errors in the authentic product's ingredient list. That's more of an indictment of the manufacturer than it is Gemini. Finally, a lot of the things that Gemini got wrong seemed to me like they could reasonably be attributed to things like optical artifacts in the photos (glare, shadows, stuff like that). Better/more photos might improve that. All that being said, this is obviously a tiny study of a single AI tool with a single brand of cosmetic product, so it's probably premature for me to optimistic, even cautiously. I've edited the last section of the writeup accordingly.

hn_throwaway_99 2mo ago
Thanks for the response, and I was overly harsh, I apologize. I did appreciate the detail and thoroughness you went to explain your experiences. Yet I still think your study was a microcosm of the greatest dangers I think about AI. That is, it was astoundingly good at doing small scale feature detection, but astoundingly bad at synthesizing an overall conclusion, and worse, it did so with characteristic "AI certainty". Also, in the real world just like you found, pictures have glare, and real manufacturers have mistakes. The horrifyingly scary thing is that even if you think Gemini did a fairly good job at feature detection, in the real world people have and will just follow the AI conclusions blindly because they're "mostly" correct, even when they lead to completely wrong outcomes. Heck, a Tesla already killed its passenger when it rammed into the side of a truck a few years ago because it mistook glare on the truck for the sun. Military planners blew up a school of young children based on old data, yet the Pentagon tried to blacklist Anthropic because Anthropic didn't want to provide autonomous kill capabilities. I don't mean to sound over dramatic, but again, to me your study highlights everything I think is wrong with AI and the extreme dangers it will cause if society relies on it too much, which it has already begun to do.