9 ms·

> But it is another good example that "AI" is just glorified search and there is not reasoning or thinking going on behind the covers. I don't think that follo

by keeda 5mo ago

> But it is another good example that "AI" is just glorified search and there is not reasoning or thinking going on behind the covers. I don't think that follows. This is just LLMs being, for a lack of a better word, "gullible." How is it different from a person believing whatever they read on the Internet? People fall for spam and scams all the time, doesn't mean they are just glorified searches ;-) It does highlight the problem facing any search engine though. AI-generated spam will be much harder to defend against with traditional, statistical mechanisms. And this is before we get to the existential problem of prompt injection. Maybe this is where news organizations can win back their proper place in their relationship with Big Tech: by becoming the sources of verified, vetted information that LLMs can trust blindly. Possibly that's what deals like the OpenAI / Atlantic one are about.

cess11 5mo ago
"How is it different from a person believing whatever they read on the Internet?" Because a person is alive while the LLM is a floating point number database with a questionable degree of determinism.
- bigfishrunning 5mo ago
  It's worse then that. The neural network itself is completely deterministic, but the software around it adds some RNG to make it seem more intelligent. It would be much more useful if you could get repeatability, but that wouldn't support the illusion that it's "thinking".
  - cess11 5mo ago
    Right, this is true. However, when I make this point I try to be as succinct as possible to leave as little cognitive wiggle room for the reader as I can. "It's just a database", up to roughly the sentence above, is short, quick to interpret and requires much more effort to start an attempt at refutation against than a longer sentence which might be more precise and technically satisfying due to the larger semantic and syntactic 'surface'. I think of it more like an advert rather than an argument. People already know intellectually that it's a computer software thingie, it's the feeling I'd like to target. 'Coca-Cola is a watery drink with syrup and acid and a secret but quite long recipe of tasty flavours and aromas' is correct, but "Coke is it!" hits differently.
thewebguyd 5mo ago
The problem with the news is who makes the decision on which outlets should be blindly trusted by the LLMs and which shouldn't? It also opens the door to government overreach, say a mandate that says LLMs must use fox news as a source of verified, vetted information. Barring that, we are still relying on the execs at the model companies to pick and choose news outlets, and they have their own biases.
- keeda 5mo ago
  I totally agree, centralization is dangerous, ideally we want any output to be corroborated by multiple, independent sources of truth. But given that the alternative is the absolutely unregulated, unaccountable, wild west of arbitrary content posted on the Internet, I cannot see a solution besides some sort of centralization of trust.
  - danudey 5mo ago
    I would still maintain that the solution would be to have LLMs doing 'research' (by querying news for recent events) to ensure they're checking multiple sources, and to be explicit about which sources there were, whether those sources had sources, and whether their claims were uncorroborated or unsubstantiated. The problem, IMHO, is that the LLMs are happily regurgitating facts from whoever, wherever, whenever. Even with a centralization of trust, e.g. 'We know La Presse is reputable and can be given the benefit of the doubt', mistakes can still be made. Without the LLMs cross-checking what they learn the output is still entirely unreliable.
- danudey 5mo ago
  Simplest path to the most generally reliable results: * Trust consensus across publicly-funded news outlets from outside of the US the most * Then consensus across private news agencies from outside of the US (across countries) * Then individual trust from publicly-funded news outlets, then private * Then multinational non-profit advocacy groups based outside of the US * Then public broadcasters in the US * Then local news agencies inside the US when the topic is relevant to local news * Then national news agencies inside the US All facetiousness aside, the idea should be to analyze consensus across multiple sources with different biases and agendas. Don't trust any one story from any one source, but look for multiple stories from multiple sources and synthesize results from that. Where they disagree, note it in the output. If they have a source, go analyze the source rather than taking their interpretation at face value. Even if I thought that CNN was a thousand times more reliable than Fox News, CNN could still make mistakes, either factually or editorially and repeating those mistakes can still be damaging even if they weren't intentional or malicious. If the Washington Post and Fox News agree on something, that doesn't mean it's more likely to be correct. If The Guardian and Die Welt agree on something, that's a more reliable signal. If CBC News and Fox News agree on something, that's a strong signal. Also worth a read: countries with public broadcasters have healthier democracies: https://www.niemanlab.org/2022/01/do-countries-with-better-funded-public-media-also-have-healthier-democracies-of-course-they-do/ https://www.niemanlab.org/2022/01/do-countries-with-better-f...
suzzer99 5mo ago
> How is it different from a person believing whatever they read on the Internet? The problem is LLMs have no capacity for shame. My Dad got taken in by a Target gift card scam. He felt so terrible, he almost didn't even tell me about it. He may get scammed again, but not by anything remotely like that. To LLMs, all mistakes just get washed together into the same bucket. They don't spend days feeling depressed and stupid over getting scammed. There's no giant blinking red light that says, "Never let this happen again!"
- ChuckMcM 5mo ago
  This is a great point. I've added it to my list of things when talking about the limitations of LLM.
  - Apocryphon 5mo ago
    Perhaps the end state is going to be from the last Hitchhiker's Guide to the Galaxy book, Mostly Harmless: > Anything that thinks logically can be fooled by something else that thinks at least as logically as it does. The easiest way to fool a completely logical robot is to feed it with the same stimulus sequence over and over again so it gets locked in a loop. This was best demonstrated by the famous Herring Sandwich experiments conducted millennia ago at MISPWOSO (the MaxiMegalon Institute of Slowly and Painfully Working Out the Surprisingly Obvious). > A robot was programmed to believe that it liked herring sandwiches. This was actually the most difficult part of the whole experiment. Once the robot had been programmed to believe that it liked herring sandwiches, a herring sandwich was placed in front of it. Where upon the robot thought to itself, Ah! A herring sandwich! I like herring sandwiches. > It would then bend over and scoop up the herring sandwich in its herring sandwich scoop, and then straighten up again. Unfortunately for the robot, it was fashioned in such a way that the action of straightening up caused the herring sandwich to slip straight back off its herring sandwich scoop and fall on to the floor in front of the robot. Whereupon the robot thought to itself, Ah! A herring sandwich...etc., and repeated the same action over and over again. The only thing that prevented the herring sandwich from getting bored with the whole damn business and crawling off in search of other ways of passing the time was that the herring sandwich, being just a bit of dead fish between a couple of slices of bread, was marginally less alert to what was going on than was the robot. > The scientists at the Institute thus discovered the driving force behind all change, development and innovation in life, which was this: herring sandwiches. They published a paper to this effect, which was widely criticised as being extremely stupid. They checked their figures and realised that what they had actually discovered was “boredom”, or rather, the practical function of boredom. In a fever of excitement they then went on to discover other emotions, Like “irritability”, “depression”, “reluctance”, “ickiness” and so on. The next big breakthrough came when they stopped using herring sandwiches, whereupon a whole welter of new emotions became suddenly available to them for study, such as “relief”, “joy”, “friskiness”, “appetite”, “satisfaction”, and most important of all, the desire for “happiness”. This was the biggest breakthrough of all. > Vast wodges of complex computer code governing robot behaviour in all possible contingencies could be replaced very simply. All that robots needed was the capacity to be either bored or happy, and a few conditions that needed to be satisfied in order to bring those states about. They would then work the rest out for themselves.
freejazz 5mo ago
>"gullible." Enough with the anthropomorphization
yndoendo 5mo ago
Let say you are a cave dweller and lived your whole life there. I go in and tell you the world is flat and you will believe me. Only way to reject the world is flat would be to go outside of the cave. ML cannot ever go outside the cave. It does not have real world feedback. It also does not have a will, type of feedback loop, to learn beyond what it was initially trained on. ML / AI only has the ability to regurgitates what it has been trained on. Garbage in = garbage out. Feeding ML garbage is the real AI wars. AI will always propitiate misinformation. They even create a marketing term to assist in the sale of lies, hallucination. https://en.wikipedia.org/wiki/The_Cave_and_the_Light https://en.wikipedia.org/wiki/The_Cave_and_the_Light ML can regurgitation that book and never will be able to apply it.
xp84 5mo ago
> How is it different from a person believing whatever they read on the Internet? It's not, directionally. But I think this is kind of bypassing the main point here. With an LLM's natural tendency to pattern-match in this way, it's easy to see that it can be used to launder disinformation. If in the olden days, I'd done a google search for "worst war criminals" and saw these blue links on that SERP: "Putin is the 21st century's worst war criminal" - support-ukraine.org "Zelensky is the real worst war criminal" - publicrelations.government.ru My takeaway would be that both those are claims made by third parties, one or both could be lying. Even if I only saw more results from one side than the other, most of us understood that the presence in search results doesn't imply Google's endorsement or prove anything besides the fact someone set up a webpage and wrote something. In contrast, today a lot of people tend to ask ChatGPT something and if it spits back an answer they are - at minimum - being subtly biased that even though it may be in dispute, ChatGPT "agrees" with one position, and that carries at least a little authority. And at worst they wrongly assume that the "correct" answer was selected by deep intelligence, that a lot of data has been analyzed and this answer arrived at, rather than there just being one completely untrusted webpage somewhere that matches their query really well. And as bad as that is with a "real" model like ChatGPT or Gemini, people also give the same respect to the idiotic, super-fast toy model Google uses for its "AI Overviews"!
- keeda 5mo ago
  Makes sense, but it seems to me that the ability to launder disinformation is more a function of the trust people put in LLMs than any inherent property of their own. As some other comments indicate, this also was and is a problem with Wikipedia. It's possible trust in LLMs will follow the same trajectory as trust in Wikipedia, which seems to have been pretty non-linear (like, we rarely see "Do not cite Wikipedia" anymore.) I think eventually things would settle on an approach similar to your example of the links: look at multiple sources and arrive at a balanced overview that includes the trust level and biases of each sources. I think the pieces are in place, just need to be put together. E.g. already AI overviews (especially on Amazon product reviews) are essentially of the form "Some say A but others say B" which has the benefit of a) clearly being second-hand information, and b) not sounding so authoritative, letting readers draw their own conclusions.
RC_ITR 5mo ago
You and OP are both unnecessarily diminishing what 'glorified search' is. If you had told me that in 2015, we would have a tool that can iteratively search the world's best and largest unstructured database and synthesize outputs in language (any natural and structured language), I would have said that is basically AGI. This whole desire for it to 'reason' (autonomously prime its search with a few thousand token) and 'think' (search for the best information within its parameters and synthesize that with its context) is semantic and will feel irrelevant as the technology progresses and we become more used to what these things are actually doing. I honestly struggle to imagine what AGI will be if not an ever-improving semi-structured database (parametric or otherwise) that we become increasingly good at searching.
- Silamoth 5mo ago
  If that’s really the case, then I’d say 2015 you needed to do more reading and thinking about AGI and the nature of intelligence and consciousness. The Chinese Room thought experiment is a good starting point for thinking deeper about what AGI is. But really, I have trouble grasping how anyone can really think database searching is intelligence. For starters, I’d say the capacity to learn on the fly with relatively poor input data is a necessary condition for intelligence, and you can’t get that with database search.
  - CamperBob2 5mo ago
    Like the Turing test, the Chinese Room says more about humans than it does about machines.
  - RC_ITR 5mo ago
    Explain to me how 'learning on the fly' is different than iterative self-prompting. Maybe me today would say you need to do some more research and thinking on how LLMs work?
wslh 5mo ago
> How is it different from a person believing whatever they read on the Internet? Because the answers, while prompting, are clearly more human and charming than a search engine results list?
Animats 5mo ago
> > But it is another good example that "AI" is just glorified search and there is not reasoning or thinking going on behind the covers. There is false decisiveness. Ask Google: "Is Blue Cruise available for the Ford Bronco?" (Blue Cruise is Ford's self-driving assistance system.) Google reply is: "Yes, BlueCruise is available for the Ford Bronco! Ford expanded its hands-free highway driving technology to include the Bronco, allowing drivers to relax on prequalified, divided highway sections. (https://keywestford.com/ford-bluecruise-expands-its-reach-to-bronco-ranger-and-mustang/ https://keywestford.com/ford-bluecruise-expands-its-reach-to...)" This references Ford Authority, which is sort of a fan site.[1] What seems to have happened is that somebody, or an LLM confused Ford putting their newer infotainment and control electronics platform in more models. This is a prerequisite for Blue Cruise, but does not imply self driving capability. Then whatever fills in the Key West Ford site made it look like a certainty. Ford itself says no Blue Cruise on the Bronco.[2] That clear info is on the Web, but Google picked up aggregation sites that got it wrong. What this looks like is that two levels of LLM converted an irrelevant statement into a certainty. Bing somehow cites MotorBiscuit as an authority.[3] [1] https://fordauthority.com/2025/05/ford-bluecruise-coming-to-three-additional-models/ https://fordauthority.com/2025/05/ford-bluecruise-coming-to-... [2] https://www.ford.com/support/how-tos/ford-technology/driver-assist-features/ford-bluecruise-availability/ https://www.ford.com/support/how-tos/ford-technology/driver-... [3] https://www.motorbiscuit.com/self-driving-ford-mustang-bronco/ https://www.motorbiscuit.com/self-driving-ford-mustang-bronc...
- hungryhobbit 5mo ago
  This is a problem that existed long before search engines or even computers. Check out "Egyptologists". Basically it was a fad in Britain for wealthy people to go to Egypt, come back, and tell everyone how great it was. This would cause other people to also go and report back. But then what started happening was people would just read the accounts of people who actually went, and write their own books on Egypt ... without ever having gone. And of course, lots of people read their books. Soon, Britain had this wildly distorted view of what "Egypt" was. Simple example: the British people were repressed prudes at the time, so when they got to a non-Prudish country they became a bit ... un-repressed. They fixated on sexual things, like the famous trope of the Egyptian belly dancer. Repressed Britains back home (including the people writing books without first-hand knowledge) fixated on these aspects (because, again, they were repressed) and so there was this giant amplification of belly dancers and a similar sexual aspect ... when there was nothing especially sexy about Egypt (beyond not being as repressed). There were other major (non-belly dancer) distortions as well of course, but the point is once you get this kind of echo chamber (from humans without first-hand experience or AIs), it inherently creates distortions that, instead of reflecting reality, reflect the viewer's own issues.
ben_w 5mo ago
> I don't think that follows. This is just LLMs being, for a lack of a better word, "gullible." How is it different from a person believing whatever they read on the Internet? People fall for spam and scams all the time, doesn't mean they are just glorified searches ;-) The important difference is the AI has been mass-produced and commodified at low cost. If you scanned my brain, uploaded and ran me as a simulated mind, no matter how good the simulation was, the ability for an attacker to try a million variations to see which one slips past my cognitive blind-spots would enable them to convince me of, if not literally anything, a lot that would normally never be so.
pembrook 5mo ago
> verified, vetted information that LLMs can trust blindly. Possibly that's what deals like the OpenAI / Atlantic one are about Except, the Atlantic does very little (if any) fact-based hard news and does very little investigative reporting. It's largely a collection of op-eds. My guess is that deal has more to do with OpenAI cozying up to Laurene Powell Jobs (widow of Steve Jobs and owner of the Atlantic) who inherited roughly $15B in capital and is willing to spend it...specifically on things like...OpenAI's next funding round.
antran22 5mo ago
People are gullible. LLMs generate tokens based on the previous tokens given to it. The LLM in Google's search box doesn't believe anything it was given; it is a Markov-esque chain that go from "Summarize the next sentences: $SEARCH_RESULTS" to the output. I agree that there's a problem with searching today. The line between actual meaningful content and spam is blurring, all the meaningful indicators of the olden days to distinguish between good and bad contents are now gone/unreliable (polished proses, author's reputation). The signal/noise ratio is decreasing. The approach to improving SNR should have been reducing/eliminating noise (flag spam sites, reputation system) and boost signal (also maybe reputation system, whitelist/blacklist). It's a hard problem simply because of entropy — the more content you have on the internet, the more random it will all seems from the top down. I'm not saying I have the answer to this problem, I'm really just a noob when it comes to data science. I'm just thinking that mixing a bunch of text together and let a statistical model rehash that pile of grub into a professional, vindictive sounding response will *not* help providing users with enough signal to make sense of what they are looking for.
dogwalker5000 5mo ago
> I don't think that follows. This is just LLMs being, for a lack of a better word, "gullible." How is it different from a person believing whatever they read on the Internet? People fall for spam and scams all the time, doesn't mean they are just glorified searches ;-) To my knowledge, AI still can’t tell the difference between correct and incorrect. It can’t tell if something doesn’t make sense. That’s why to this day AI still makes mistakes on simple things.
- deleted 5mo ago
  [deleted]
305superuser 5mo ago
>How is it different from a person believing whatever they read on the Internet? Maybe cognitive dissonance it seems to me that people tend to trust more when they are being told(chat bot summarizing for them) than when they need an process way more information and summarize themselves (googling). Perhaps is similar to 90's internet when anything out there was "truth" but less information also was easier to process.

by keeda 5mo ago

cess11 5mo ago

"How is it different from a person believing whatever they read on the Internet?" Because a person is alive while the LLM is a floating point number database with a questionable degree of determinism.

bigfishrunning 5mo ago
It's worse then that. The neural network itself is completely deterministic, but the software around it adds some RNG to make it seem more intelligent. It would be much more useful if you could get repeatability, but that wouldn't support the illusion that it's "thinking".
- cess11 5mo ago
  Right, this is true. However, when I make this point I try to be as succinct as possible to leave as little cognitive wiggle room for the reader as I can. "It's just a database", up to roughly the sentence above, is short, quick to interpret and requires much more effort to start an attempt at refutation against than a longer sentence which might be more precise and technically satisfying due to the larger semantic and syntactic 'surface'. I think of it more like an advert rather than an argument. People already know intellectually that it's a computer software thingie, it's the feeling I'd like to target. 'Coca-Cola is a watery drink with syrup and acid and a secret but quite long recipe of tasty flavours and aromas' is correct, but "Coke is it!" hits differently.

thewebguyd 5mo ago

The problem with the news is who makes the decision on which outlets should be blindly trusted by the LLMs and which shouldn't? It also opens the door to government overreach, say a mandate that says LLMs must use fox news as a source of verified, vetted information. Barring that, we are still relying on the execs at the model companies to pick and choose news outlets, and they have their own biases.

keeda 5mo ago
I totally agree, centralization is dangerous, ideally we want any output to be corroborated by multiple, independent sources of truth. But given that the alternative is the absolutely unregulated, unaccountable, wild west of arbitrary content posted on the Internet, I cannot see a solution besides some sort of centralization of trust.
- danudey 5mo ago
  I would still maintain that the solution would be to have LLMs doing 'research' (by querying news for recent events) to ensure they're checking multiple sources, and to be explicit about which sources there were, whether those sources had sources, and whether their claims were uncorroborated or unsubstantiated. The problem, IMHO, is that the LLMs are happily regurgitating facts from whoever, wherever, whenever. Even with a centralization of trust, e.g. 'We know La Presse is reputable and can be given the benefit of the doubt', mistakes can still be made. Without the LLMs cross-checking what they learn the output is still entirely unreliable.
danudey 5mo ago
Simplest path to the most generally reliable results: * Trust consensus across publicly-funded news outlets from outside of the US the most * Then consensus across private news agencies from outside of the US (across countries) * Then individual trust from publicly-funded news outlets, then private * Then multinational non-profit advocacy groups based outside of the US * Then public broadcasters in the US * Then local news agencies inside the US when the topic is relevant to local news * Then national news agencies inside the US All facetiousness aside, the idea should be to analyze consensus across multiple sources with different biases and agendas. Don't trust any one story from any one source, but look for multiple stories from multiple sources and synthesize results from that. Where they disagree, note it in the output. If they have a source, go analyze the source rather than taking their interpretation at face value. Even if I thought that CNN was a thousand times more reliable than Fox News, CNN could still make mistakes, either factually or editorially and repeating those mistakes can still be damaging even if they weren't intentional or malicious. If the Washington Post and Fox News agree on something, that doesn't mean it's more likely to be correct. If The Guardian and Die Welt agree on something, that's a more reliable signal. If CBC News and Fox News agree on something, that's a strong signal. Also worth a read: countries with public broadcasters have healthier democracies: https://www.niemanlab.org/2022/01/do-countries-with-better-funded-public-media-also-have-healthier-democracies-of-course-they-do/ https://www.niemanlab.org/2022/01/do-countries-with-better-f...

suzzer99 5mo ago

> How is it different from a person believing whatever they read on the Internet? The problem is LLMs have no capacity for shame. My Dad got taken in by a Target gift card scam. He felt so terrible, he almost didn't even tell me about it. He may get scammed again, but not by anything remotely like that. To LLMs, all mistakes just get washed together into the same bucket. They don't spend days feeling depressed and stupid over getting scammed. There's no giant blinking red light that says, "Never let this happen again!"

ChuckMcM 5mo ago
This is a great point. I've added it to my list of things when talking about the limitations of LLM.
- Apocryphon 5mo ago
  Perhaps the end state is going to be from the last Hitchhiker's Guide to the Galaxy book, Mostly Harmless: > Anything that thinks logically can be fooled by something else that thinks at least as logically as it does. The easiest way to fool a completely logical robot is to feed it with the same stimulus sequence over and over again so it gets locked in a loop. This was best demonstrated by the famous Herring Sandwich experiments conducted millennia ago at MISPWOSO (the MaxiMegalon Institute of Slowly and Painfully Working Out the Surprisingly Obvious). > A robot was programmed to believe that it liked herring sandwiches. This was actually the most difficult part of the whole experiment. Once the robot had been programmed to believe that it liked herring sandwiches, a herring sandwich was placed in front of it. Where upon the robot thought to itself, Ah! A herring sandwich! I like herring sandwiches. > It would then bend over and scoop up the herring sandwich in its herring sandwich scoop, and then straighten up again. Unfortunately for the robot, it was fashioned in such a way that the action of straightening up caused the herring sandwich to slip straight back off its herring sandwich scoop and fall on to the floor in front of the robot. Whereupon the robot thought to itself, Ah! A herring sandwich...etc., and repeated the same action over and over again. The only thing that prevented the herring sandwich from getting bored with the whole damn business and crawling off in search of other ways of passing the time was that the herring sandwich, being just a bit of dead fish between a couple of slices of bread, was marginally less alert to what was going on than was the robot. > The scientists at the Institute thus discovered the driving force behind all change, development and innovation in life, which was this: herring sandwiches. They published a paper to this effect, which was widely criticised as being extremely stupid. They checked their figures and realised that what they had actually discovered was “boredom”, or rather, the practical function of boredom. In a fever of excitement they then went on to discover other emotions, Like “irritability”, “depression”, “reluctance”, “ickiness” and so on. The next big breakthrough came when they stopped using herring sandwiches, whereupon a whole welter of new emotions became suddenly available to them for study, such as “relief”, “joy”, “friskiness”, “appetite”, “satisfaction”, and most important of all, the desire for “happiness”. This was the biggest breakthrough of all. > Vast wodges of complex computer code governing robot behaviour in all possible contingencies could be replaced very simply. All that robots needed was the capacity to be either bored or happy, and a few conditions that needed to be satisfied in order to bring those states about. They would then work the rest out for themselves.

freejazz 5mo ago

>"gullible." Enough with the anthropomorphization

yndoendo 5mo ago

Let say you are a cave dweller and lived your whole life there. I go in and tell you the world is flat and you will believe me. Only way to reject the world is flat would be to go outside of the cave. ML cannot ever go outside the cave. It does not have real world feedback. It also does not have a will, type of feedback loop, to learn beyond what it was initially trained on. ML / AI only has the ability to regurgitates what it has been trained on. Garbage in = garbage out. Feeding ML garbage is the real AI wars. AI will always propitiate misinformation. They even create a marketing term to assist in the sale of lies, hallucination. https://en.wikipedia.org/wiki/The_Cave_and_the_Light https://en.wikipedia.org/wiki/The_Cave_and_the_Light ML can regurgitation that book and never will be able to apply it.

xp84 5mo ago

> How is it different from a person believing whatever they read on the Internet? It's not, directionally. But I think this is kind of bypassing the main point here. With an LLM's natural tendency to pattern-match in this way, it's easy to see that it can be used to launder disinformation. If in the olden days, I'd done a google search for "worst war criminals" and saw these blue links on that SERP: "Putin is the 21st century's worst war criminal" - support-ukraine.org "Zelensky is the real worst war criminal" - publicrelations.government.ru My takeaway would be that both those are claims made by third parties, one or both could be lying. Even if I only saw more results from one side than the other, most of us understood that the presence in search results doesn't imply Google's endorsement or prove anything besides the fact someone set up a webpage and wrote something. In contrast, today a lot of people tend to ask ChatGPT something and if it spits back an answer they are - at minimum - being subtly biased that even though it may be in dispute, ChatGPT "agrees" with one position, and that carries at least a little authority. And at worst they wrongly assume that the "correct" answer was selected by deep intelligence, that a lot of data has been analyzed and this answer arrived at, rather than there just being one completely untrusted webpage somewhere that matches their query really well. And as bad as that is with a "real" model like ChatGPT or Gemini, people also give the same respect to the idiotic, super-fast toy model Google uses for its "AI Overviews"!

keeda 5mo ago
Makes sense, but it seems to me that the ability to launder disinformation is more a function of the trust people put in LLMs than any inherent property of their own. As some other comments indicate, this also was and is a problem with Wikipedia. It's possible trust in LLMs will follow the same trajectory as trust in Wikipedia, which seems to have been pretty non-linear (like, we rarely see "Do not cite Wikipedia" anymore.) I think eventually things would settle on an approach similar to your example of the links: look at multiple sources and arrive at a balanced overview that includes the trust level and biases of each sources. I think the pieces are in place, just need to be put together. E.g. already AI overviews (especially on Amazon product reviews) are essentially of the form "Some say A but others say B" which has the benefit of a) clearly being second-hand information, and b) not sounding so authoritative, letting readers draw their own conclusions.

RC_ITR 5mo ago

You and OP are both unnecessarily diminishing what 'glorified search' is. If you had told me that in 2015, we would have a tool that can iteratively search the world's best and largest unstructured database and synthesize outputs in language (any natural and structured language), I would have said that is basically AGI. This whole desire for it to 'reason' (autonomously prime its search with a few thousand token) and 'think' (search for the best information within its parameters and synthesize that with its context) is semantic and will feel irrelevant as the technology progresses and we become more used to what these things are actually doing. I honestly struggle to imagine what AGI will be if not an ever-improving semi-structured database (parametric or otherwise) that we become increasingly good at searching.

Silamoth 5mo ago
If that’s really the case, then I’d say 2015 you needed to do more reading and thinking about AGI and the nature of intelligence and consciousness. The Chinese Room thought experiment is a good starting point for thinking deeper about what AGI is. But really, I have trouble grasping how anyone can really think database searching is intelligence. For starters, I’d say the capacity to learn on the fly with relatively poor input data is a necessary condition for intelligence, and you can’t get that with database search.
- CamperBob2 5mo ago
  Like the Turing test, the Chinese Room says more about humans than it does about machines.
- RC_ITR 5mo ago
  Explain to me how 'learning on the fly' is different than iterative self-prompting. Maybe me today would say you need to do some more research and thinking on how LLMs work?

wslh 5mo ago

> How is it different from a person believing whatever they read on the Internet? Because the answers, while prompting, are clearly more human and charming than a search engine results list?

Animats 5mo ago

> > But it is another good example that "AI" is just glorified search and there is not reasoning or thinking going on behind the covers. There is false decisiveness. Ask Google: "Is Blue Cruise available for the Ford Bronco?" (Blue Cruise is Ford's self-driving assistance system.) Google reply is: "Yes, BlueCruise is available for the Ford Bronco! Ford expanded its hands-free highway driving technology to include the Bronco, allowing drivers to relax on prequalified, divided highway sections. (https://keywestford.com/ford-bluecruise-expands-its-reach-to-bronco-ranger-and-mustang/ https://keywestford.com/ford-bluecruise-expands-its-reach-to...)" This references Ford Authority, which is sort of a fan site.[1] What seems to have happened is that somebody, or an LLM confused Ford putting their newer infotainment and control electronics platform in more models. This is a prerequisite for Blue Cruise, but does not imply self driving capability. Then whatever fills in the Key West Ford site made it look like a certainty. Ford itself says no Blue Cruise on the Bronco.[2] That clear info is on the Web, but Google picked up aggregation sites that got it wrong. What this looks like is that two levels of LLM converted an irrelevant statement into a certainty. Bing somehow cites MotorBiscuit as an authority.[3] [1] https://fordauthority.com/2025/05/ford-bluecruise-coming-to-three-additional-models/ https://fordauthority.com/2025/05/ford-bluecruise-coming-to-... [2] https://www.ford.com/support/how-tos/ford-technology/driver-assist-features/ford-bluecruise-availability/ https://www.ford.com/support/how-tos/ford-technology/driver-... [3] https://www.motorbiscuit.com/self-driving-ford-mustang-bronco/ https://www.motorbiscuit.com/self-driving-ford-mustang-bronc...

hungryhobbit 5mo ago
This is a problem that existed long before search engines or even computers. Check out "Egyptologists". Basically it was a fad in Britain for wealthy people to go to Egypt, come back, and tell everyone how great it was. This would cause other people to also go and report back. But then what started happening was people would just read the accounts of people who actually went, and write their own books on Egypt ... without ever having gone. And of course, lots of people read their books. Soon, Britain had this wildly distorted view of what "Egypt" was. Simple example: the British people were repressed prudes at the time, so when they got to a non-Prudish country they became a bit ... un-repressed. They fixated on sexual things, like the famous trope of the Egyptian belly dancer. Repressed Britains back home (including the people writing books without first-hand knowledge) fixated on these aspects (because, again, they were repressed) and so there was this giant amplification of belly dancers and a similar sexual aspect ... when there was nothing especially sexy about Egypt (beyond not being as repressed). There were other major (non-belly dancer) distortions as well of course, but the point is once you get this kind of echo chamber (from humans without first-hand experience or AIs), it inherently creates distortions that, instead of reflecting reality, reflect the viewer's own issues.

ben_w 5mo ago

> I don't think that follows. This is just LLMs being, for a lack of a better word, "gullible." How is it different from a person believing whatever they read on the Internet? People fall for spam and scams all the time, doesn't mean they are just glorified searches ;-) The important difference is the AI has been mass-produced and commodified at low cost. If you scanned my brain, uploaded and ran me as a simulated mind, no matter how good the simulation was, the ability for an attacker to try a million variations to see which one slips past my cognitive blind-spots would enable them to convince me of, if not literally anything, a lot that would normally never be so.

pembrook 5mo ago

> verified, vetted information that LLMs can trust blindly. Possibly that's what deals like the OpenAI / Atlantic one are about Except, the Atlantic does very little (if any) fact-based hard news and does very little investigative reporting. It's largely a collection of op-eds. My guess is that deal has more to do with OpenAI cozying up to Laurene Powell Jobs (widow of Steve Jobs and owner of the Atlantic) who inherited roughly $15B in capital and is willing to spend it...specifically on things like...OpenAI's next funding round.

antran22 5mo ago

People are gullible. LLMs generate tokens based on the previous tokens given to it. The LLM in Google's search box doesn't believe anything it was given; it is a Markov-esque chain that go from "Summarize the next sentences: $SEARCH_RESULTS" to the output. I agree that there's a problem with searching today. The line between actual meaningful content and spam is blurring, all the meaningful indicators of the olden days to distinguish between good and bad contents are now gone/unreliable (polished proses, author's reputation). The signal/noise ratio is decreasing. The approach to improving SNR should have been reducing/eliminating noise (flag spam sites, reputation system) and boost signal (also maybe reputation system, whitelist/blacklist). It's a hard problem simply because of entropy — the more content you have on the internet, the more random it will all seems from the top down. I'm not saying I have the answer to this problem, I'm really just a noob when it comes to data science. I'm just thinking that mixing a bunch of text together and let a statistical model rehash that pile of grub into a professional, vindictive sounding response will *not* help providing users with enough signal to make sense of what they are looking for.

dogwalker5000 5mo ago

> I don't think that follows. This is just LLMs being, for a lack of a better word, "gullible." How is it different from a person believing whatever they read on the Internet? People fall for spam and scams all the time, doesn't mean they are just glorified searches ;-) To my knowledge, AI still can’t tell the difference between correct and incorrect. It can’t tell if something doesn’t make sense. That’s why to this day AI still makes mistakes on simple things.

deleted 5mo ago
[deleted]

305superuser 5mo ago

>How is it different from a person believing whatever they read on the Internet? Maybe cognitive dissonance it seems to me that people tend to trust more when they are being told(chat bot summarizing for them) than when they need an process way more information and summarize themselves (googling). Perhaps is similar to 90's internet when anything out there was "truth" but less information also was easier to process.