9 ms·

AI Responses May Include Mistakes

184 points by userbinator 1y ago

_wire_ 1y ago
Google's Gemini in search just makes up something that arbitrarily appears to support the query without care for context and accuracy. Pure confabulation. Try it for yourself. Ridiculous. It works as memory support if you know the result you're looking for, but if you don't, you can't trust it as far as you can throw it. If you look carefully at Google Veo output, it's similarly full of holes. It's plain there's no reasoning whatsoever informing the output. Veo output with goofy wrongness https://arstechnica.com/ai/2025/05/ai-video-just-took-a-startling-leap-in-realism-are-we-doomed/ https://arstechnica.com/ai/2025/05/ai-video-just-took-a-star... Tesla FSD goes crazy https://electrek.co/2025/05/23/tesla-full-self-driving-veers-off-road-flips-car-scary-crash-driver-couldnt-prevent/ https://electrek.co/2025/05/23/tesla-full-self-driving-veers...
- ImPostingOnHN 1y ago
  gemini is the worst LLM I've used, whether directly or through search. As in your experience, it regularly makes stuff up, like language/application features, or command flags (including regarding google products), and provides helpful references to sources which do not say what is cited from them. in my case, it does so roughly half the time, which is the worst proportion, because that means I can't even slightly rely upon the truth being the opposite of the output.
  - JimDabell 1y ago
    Gemini was underwhelming until 2.5 Pro came along, which is very good. But in my experience all of the Google models are far worse than everything else when it comes to hallucination.
    - MangoToupe 1y ago
      As a corollary, though, the chatbots are probably the most creative.
      - mdp2021 1y ago
        Professional creatives do measure their intuitions against a number of constraints...
        deleted 1y ago
        [deleted]
- flomo 1y ago
  I had a question about my car, so I googled '[year] [make] [model] [feature]'. This seems like the sort of thing that Google had always absolutely nailed. But now, 90% of the page was ai slop about wrong model, wrong year, even the wrong make. (There was one youtube which was sorta informative, so some credit.) But way way down on at the very the bottom of the page, there was the classic google search answer on a totally unrelated car forum. Thanks CamaroZ28.com!
  - camillomiller 1y ago
    This is a very very good point. If this was happening with different queries that we never used, or a new type of questions/queries than I would have some patience. But it happens exactly with the formulations that were giving you the best results in the SERP before!
  - TeMPOraL 1y ago
    That was true before AI too (I know, I did such searches myself). Google results has been drowning in slop for over a decade now - it was just human-generated slop, aka. content marketing and SEO stuff. I'm not defending the AI feature here, just trying to frame the problem: the lies and hallucinations were already there, but nobody cared because apparently people don't mind being constantly lied to by other people.
    - flomo 1y ago
      No, I'm not complaining about SEO shit... The thing is the primordial google had the answer, but Google themselves buried it under 100+ links of Google-generated slopdiarrhea, most of which didn't even factually fit the question, and was not at all relevant to my automobile.
      - ben_w 1y ago
        Indeed, this is part of the current monopoly abuse case they're facing — did Google deliberately choose to make search worse, because that causes people to return to the search results page and spend more time looking at ads, and they knew they could get away with it? e.g. bottom of first page, penultimate paragraph https://www.justice.gov/d9/2023-11/417557.pdf https://www.justice.gov/d9/2023-11/417557.pdf
- MangoToupe 1y ago
  I'm honestly so confused how people use LLMs as a replacement for search. All chatbots can ever find are data tangential to the stuff I want (eg i ask for a source, it gives me a quote). Maybe i was just holding search wrong?
  - Garlef 1y ago
    Maybe that's because we're conditioned by the UX of search. But another thing I find even more surprising is that, at least initially, many expected that the LLMs would give them access to some form of higher truth.
    - MangoToupe 1y ago
      I think you might be on to something. I've found myself irritated that i can't just chuck keywords at LLMs.
  - MaxikCZ 1y ago
    Its good to be shown direction. When I only have a vauge idea of what I want, AI usually helps me frame it into searchable terms I had no clue existed.
  - mdp2021 1y ago
    > LLMs as a replacement for search Some people expect LLMs as part of a better "search". LLMs should be integrated to search, as a natural application: search results can heavily depend on happy phrasing, search engines work through sparse keywords, and LLMs allow to use structured natural language (not "foo bar baz" but "Which foo did a bar baz?" - which should be resistant to terms variation and exclude different semantics related to those otherwise sparse terms). But it has to be done properly - understand the question, find material, verify the material, produce a draft reply, verify the draft vis-a-vis the material, maybe iterate...
    - 1659447091 1y ago
      DuckDuckGo Ai assist is going in the right direction, imo. It will pull info from wikipedia, use math and map tools plus other web sources that has been mostly accurate for me on the search page. The chat option uses gpt-4o with web search and was able to provide links to colonial map resources I was curious about after falling down that rabbit hole. It also gave me general (& proper) present day map links to the places I was looking for in the map sites I asked for. It did get confused a few times when I was trying to get present day names of old places I had forgot; like Charles River in Va that it kept trying to send me to Boston or Charles City Co on the James river and told me to look for it around there... The York river wiki page clearly says it was once Charles River. Maybe I wasn't asking the right questions. For more unique things it was pretty helpful thou and saved the endless searching w/ 100 tabs adventure
- camillomiller 1y ago
  This baffles me like no other tech has done before. Google is betting its own core business on a pivot that relies on a massively faulty piece of technology. And as Ben Evans also says, promising that it will get better only gets you so far, it’s an empty promise. Yesterday AI overview made up an entire album by a dead Italian musician when I searched for a tribute event that was happening at a Berlin venue. It just took the name of the venue and claimed it was the most important work from that artist. Funnily enough (not for Google), I copypasted that answer on chatGPT and it roasted AI Overview so bad on its mistakes and with such sarcasm that it even made me chuckle.
  - DanHulton 1y ago
    It's the unfounded promises that this will be solved because the tech will only get better that really upset me. Because sure, it will get better, I'm pretty certain of that. They'll have additional capabilities, they'll have access to more-recent data, etc. But "better" does not necessarily equate to "will fix the lying problem." That's a problem that is BAKED INTO the technology, and requires some kind of different approach to solve -- you can't just keep making a hammer bigger and bigger in the hopes that one day it'll turn into a screwdriver. Before LLMs really took off, we were in the middle of an "AI winter", where there just weren't any promising techs, at least none with sufficient funding attached to them. And it's WORSE now. LLMs have sucked all the air out of the room, and all of the funding out of other avenues of research. Technologies that were "10-20" years away now might be 30-40, because there's fewer people researching them, with less money, and they might even be completely different people trying to restart the research after the old ones got recruited away to work on LLMs!
    - mountainriver 1y ago
      I really don’t understand the whole AI winter talk all the time. We haven’t had anything of the sort since 2008. There were tons of major RL advancements before ChatGPT that were stunning. I would challenge anyone to find data to actually support any of these claims. ML spending has been up since deep learning year over year and the models just keep getting better
- XorNot 1y ago
  I use ublock to remove Gemini responses from search, because even glancing at then is liable to bias my assumption about whatever I'm looking for. Information hygiene is a skill which started out important but is going to become absolutely critical.
  - justmarc 1y ago
    We can't expect the vast majority of regular users to have any of that skill. What is this going to lead to? fascinating times.
    - jobigoud 1y ago
      It's very easy though, Right click > Block element > Create. Overlays show which blocks you are removing. Sliders can be used to increase/refine. How can we make it even easier and visual? Customizing pages by removing elements should be part of the default browser experience to be honest. Like in the initial web where you would tweak the color of links, visited links, etc.
  - MaxikCZ 1y ago
    Half my browser extensions have sole purpose of removing shit from sites I visit. HN is like a unicorn that havent made me block a single thing yet.
    - alpaca128 1y ago
      Ironically that's an AI tool I would use - one that can dynamically filter content from sites according to my preferences, to counter the algorithmic spam. It wouldn't be generative AI though, and that's the only kind of AI that matters right now apparently.
      - jazzyjackson 1y ago
        Could be a good use of structured output, Llms work okay as one shot classifiers, you could define a schema that's just like, array[]{xpath:true/false} and tell the bot what you want to see and what you don't want to see.
- justmarc 1y ago
  And suddenly this type of quality is becoming "normal" and acceptable now? Nobody really complains. That is very worrying. Normally this would never fly, but nowadays it's kind of OK? Why should false and or inaccurate results be accepted?
  - celdon25 1y ago
    Complain to it enough times, remain resilient and you’ll eventually figure it out (that’s a wild card though). Or find someone who has and take their word for it (except you can’t because they’re probably indistinguishable from the ‘bot’ now according to the contradictory narrative). Iterate. Spiral. No one should have to go through that though. Be merciful.
  - chronid 1y ago
    Suddenly? That's the level of quality that is standard in all software projects I've ever seen since I've started working in IT. Enshittification is all around us and is unstoppable. Because we have deadlines to hit and goals to shows we reached to the VP. We broke everything and the software is just half working? Come on that's an issue for the support and ops teams. On to the next beautiful feature we can put on marketing slides!
    - justmarc 1y ago
      Sadly you are absolutely right.
  - TeMPOraL 1y ago
    We lost that battle back when we collectively decided that sales and marketing is respectable work.
    - bheadmaster 1y ago
      Hah. Good observatoon. I often get in arguments about how I tend to avoid brands that put too much into marketing. Of course, theoretically, the amount of money a company puts into marketing doesn't automatically lower the quality of their products, but in my experience, the correlation is there. Whiskas, Coka Cola, McDonalds, etc.
- Nursie 1y ago
  Yep, I was looking up a hint for the Blue Prince game the other day for the (spoiler alert?) casino room. Google’s AI results proceeded to tell me all about the games available at the Blue Prince Casino down the road from here, where I know for a fact there’s only a prison, a Costco, a few rural properties and a whole lot of fuck-all. It’s amazing to watch it fill in absolute false, fabricated tripe at the top of their search page. It also frequently returns bad information on subjects like employment law and whatever else I look up. It would be hilarious if people weren’t actually relying on it.
  - datavirtue 1y ago
    I the have had a lot of luck with copilot conversations to research stocks and trading strategies. I am always skeptical of results and verify everything with various sources but it does help me find/get on the right track.
- Kwpolska 1y ago
  Google recently started showing me their AI bullshit. This made me pull the trigger and switch to DuckDuckGo as the primary search engine. That said, some niche stuff has significantly better results in Google. But not in the AI bullshit. I searched for a very niche train-related, the bullshit response said condescendingly "this word does not exist, maybe you meant [similarly sounding but completely different word], which in the context of trains means ...". The first real result? Turns out that word does exist.
  - christophilus 1y ago
    What’s the word?
  - dijksterhuis 1y ago
    fyi, you can remove any and all “ai” assistant bs etc from DDG if you use the noai subdomain (in case you wanna avoid their stuff, although it’s much less prominent anyway) https://noai.duckduckgo.com/ https://noai.duckduckgo.com/
  - datavirtue 1y ago
    I switched to DDG over seven years ago and just realized it had been that long when I read your comment. Google started wasting my time and I had to shift.
- christophilus 1y ago
  I’ve had good results with Brave search, which self reports to use: Meta Llama 3, Mistral / Mixtral, and CodeLLM. It’s not always 100% accurate, but it’s almost always done the trick and saved me digging through more docs than necessary.
- veunes 1y ago
  Yeah, it feels like we've crossed into a weird uncanny valley where AI outputs sound smarter than ever, but the underlying logic (or lack thereof) hasn't caught up
  - roywiggins 1y ago
    I think it's just much easier for an LLM to learn how to be convincing than it is to actually be accurate. It just has to convince RLHF trainers that it's right, not actually be right. And the first one is a general skill that can be learned and applied to anything. https://arxiv.org/html/2409.12822v1 https://arxiv.org/html/2409.12822v1
minimaxir 1y ago
The simple "AI responses may include mistakes" disclaimer or ChatGPT's "ChatGPT can make mistakes. Check important info." CYA text at the bottom of the UI are clearly no longer sufficient. After years of news stories about LLM hallucinations in fact-specific domains and people still getting burnt by them, LLM providers should be more aggressive in educating users about their fallability since hallucinations can't ever be fully fixed, even if it means adding friction.
- tbrownaw 1y ago
  > should be more aggressive in educating users about their fallability This might be an "experience is the best teacher" situation. It'd probably be pretty hard to invent a disclaimer that'd be as effective as getting bit.
  - minimaxir 1y ago
    Unfortunately, getting bit in cases such as publishing misinformation or false legal citations waste everyone time, not just their own.
- eddythompson80 1y ago
  That doesn't really make sense. You either make the LLM provider liable for the output of the model, or you have the current model. The friction already exists. All these AI companies and cloud providers are running "censored models" and more censorship is added at every layer. What would more friction be here? more pop-ups? Doing the former basically means killing the model-hosting business. Companies could develop models, use them internally and give them to their employees, but no public APIs exists. Companies strike legally binding contracts to use/license each other models, but the general public doesn't have access to those without something that would mitigate the legal risk. Maybe years down the line, as attitudes soften, some companies would begin to push the boundaries. Automating the legal approval process, opening signups, etc.
  - minimaxir 1y ago
    Yes, more popups, retention metrics be damned. Even 2 years since ChatGPT, many people still think it's omniscent which is what's causing trouble.
Biganon 1y ago
Make up a fake popular wisdom saying that sounds real, search for it, Gemini will gladly tell you it exists and explain it to you
- mdp2021 1y ago
  ...As others have noted elsewhere: which Gemini? There are cheap ones and others proposed as flagship.
  - ghusbands 1y ago
    The one that appears when you search with Google.
elmerfud 1y ago
AI is like that one guy who always can tell you something about anything with total confidence. So really not sure why anyone would trust it beyond a bar conversation.
- vouaobrasil 1y ago
  I think it's psychological. Most people use visual body cues to determine whether someone is lacking in confidence in their answer. AI does not have any cues to show a lack of confidence, and people also have a high trust in machine output because traditional algorithms always give the correct answer. The percentage of people that will look at it critically is negligible.
  - normie3000 1y ago
    > Most people use visual body cues to determine whether someone is lacking in confidence in their answer Do they?
    - vouaobrasil 1y ago
      I can certainly tell when someone is just bull**ing from their tone of voice long before they tell me the information.
      - tonyedgecombe 1y ago
        You obviously haven’t met any of my past bosses. Some people have turned bullshitting into an art form.
      - ghusbands 1y ago
        Which is notably not a "visual body cue"
- JdeBP 1y ago
  No "AI" company has yet had the bravery to name its product Cliff Clavin. Bravery in several ways, that is. There's the risk of being sued by John Ratzenberger. (-:
charcircuit 1y ago
Gemini can handle this fine.
- mucha 1y ago
  Capability != Reliability
- nehal3m 1y ago
  Huh, I thought that AI overview feature is powered by Gemini.
  - charcircuit 1y ago
    Gemini doesn't refer to a specific model so perhaps the one on the search page is weaker than the ones it offers in the app.
    - mdp2021 1y ago
      They may have estimated the volume of replies in those "overview" pages and shrunk the costs routing through a computationally lightweight model. They may this way also have underestimated the reputational loss - the big umbrella of the "Ford Pinto case". -- Edit: I was just now looking at the new Visual Capitalist's "Ranked: 2025’s 10 Largest S&P 500 Stocks". Is it possible that Alphabet being at the top with 7.6% of the weight of the 500-items set is paradoxically allowing to afford more damage?
9x39 1y ago
Gemini appears tuned to try to handle the typical questions people type in, while more traditional things you search for get some confabulated nonsense. I've observed a great deal of people trust the AI Overview as an oracle. IMO, it's how 'normal' people interact with AI if they aren't direct LLM users. It's not even age gated like trusting the news - trusting AI outputs seems to cross most demographics. We love our confident-based-on-nothing computer answers as a species, I think.
- danielbln 1y ago
  We love our confident-based-on-nothing answers period, computer or not.
  - chneu 1y ago
    Most folks just want confirmation. They don't want to have their views/opinions changed. LLM are good at trying to give folks what they're looking for.
  - mdp2021 1y ago
    Repent. You are not there to "love what gives you the kicks". That's a kind of love that should not exit the bedroom (better, the bathroom).
- Llamamoe 1y ago
  I already went through a realization a while ago that you just can't mention something to people anymore and expect them to be able to learn about it by searching the web, like it used to be possible, because everything is just unreliable misleading SEO spam slop. I shudder to think how much worse this is going to be with "AI Overview". Are we entering an era of people googling "how does a printer work" and (possibly) being told that it's built by a system of pulleys and ropes and just trusting it blindly? Because that's the kind of magnitude of errors I've seen in dozens of searches I've made in the domains I'm interested in, and I think everyone has seen the screenshots of even more outlandish - or outright dangerous - answers.
- eddythompson80 1y ago
jimmySixDOF 1y ago
Error control as in LLM-as-a-Judge needs to be integrated into every pipeline there are use at home (sub 7B) sized models and nano's (fit into browser) [1] so push out to the edge if its a Google sized scale/cost problem. Should actually be like VirusTotal where you get a consensus score of x/72 different lookups. [1] see haize labs : https://www.haizelabs.com/product/judges https://www.haizelabs.com/product/judges
- eddythompson80 1y ago
  This is a perfect example of snakeoil nonesense being sold in the AI tech market these days. People offering to wrap your LLM calls with another prompt asking the LLM to "reconsider" or "does this make sense to you", and selling you that artisanally crafted prompt at a premium. This is simply an information retrieval problem. LLMs don't retrieve information perfectly from their context. They are very good at it, but every now and then they'll introduce a change here or there. Changing an "Hello" to "Hi" doesn't really make any difference, but changing an "PS/2 Model 286" to "PS/2 Model 280" makes a huge difference. The LLM "knows"* this at some level because it "knows" that names are important to be reproduced in exact format. But everynow and then even names can change and sill generally mean the same thing, so everynow and then it'll change a name or an identifier for no reason. some of my favorite descriptions of this I have heard from people: - We need to introduce a reflection layer - We need a supervisor tree-like checks on responses - We need an evaluation-feedback mechanism for AI operations - We need to agents that operates and judges/supervisors/evaluators all apparently mean: const response = await getChatResponse([...messages, newMsg]); const {didGood, why} = await getChatResponse([RolePlayLikeAMeanJudgePromot, ...messages, newMsg, response]) if (!didGoog) { response = await getChatResonse([ThreateningMafiaGuyPrompt, [ ...messages, newMsg, response, why[]) } // loop like 4 times, maybe use different models and egg them on each other, like use Grok and tell it you got the response from Claude and that Claude was shit talking Grok all the way through it. Like it was unnecessary tbh. // this makes Grook work extra hard. It's actually been measured by kill-a-watt. *: I say "knows" to indicate just that the model training biases it that way
  - mdp2021 1y ago
rcarmo 1y ago
This is why Google has got search fundamentally wrong. They just don’t care about accuracy of results anymore, and worry mostly about providing a quick answer and a bunch of sponsored links below it.
- Llamamoe 1y ago
  Except that out of 10 answers, the "quick answer" is subtly wrong 6 times, egregiously wrong 2, and outright dangerous once. I've seen screenshots of stuff that would get people killed or in legal trouble.
- dandanua 1y ago
  They just continue the Eric Schmidt idea "More results are better than none". It has evolved to "It's better to hallucinate than produce a negative answer", I guess.
jkuli 1y ago
Humans can also make mistakes. This is the first test I apply every time. Could it be that AI is actually more capable than a human? If industry decides than humans are more reliable, they will choose to use humans. Reliability is part of cost-effectiveness, and it's built in to the business decision process.
- theodric 1y ago
  A human would only make up an answer like the ones in the article if it were a compulsive liar. A human would ideally say "I don't know" or at worst employ the "I'll confirm that and circle back" corpspeak evasion.
- mdp2021 1y ago
  Another case of badly computed reputational loss. That context is not of "this more than that" comparisons, but of threshold: the service must be _adequate_. If you don't have random humans capable of providing the needed service, find them. Do not employ random agents.
- alpaca128 1y ago
  The average Google user expects LLMs to be perfect and treats their responses as answer from an oracle, not just better than the average human. It doesn't matter whether it's better than humans, the one thing that matters are the consequences of its widespread use.
neilv 1y ago
On the Google search Web site, the "AI responses may include mistakes." weak disclaimer small print is also hidden behind the "Show more" button. When OpenAI launched ChatGPT, I had to explain to a non-CS professor that it wasn't AI like they're thinking of, but currently more like a computational parlor trick that looks a lot like AI. But turns out this parlor trick is awesome for cheating on homework. Also good at cheating at many other kinds of work, if you don't care much about quality, nor about copyrights.
- loa_in_ 1y ago
  It's a memory augmentation/information retrieval tool with flexible input and output interface.
- stavros 1y ago
  I really don't understand the view that it's a "parlor trick that looks like AI". If it's not "a thing that can write code", but instead just looks like a thing that can write code (but can actually write code), it can write code. All the "no true Scotsman" stuff about what it's doing behind the scenes is irrelevant, because we have no idea what human brains are doing behind the scenes either.
  - keiferski 1y ago
    It matters if we are making a distinction between essence and output. On the output side, it functionally doesn’t really have a difference. At least in terms of more abstract things like writing code. Although I would argue that the output AI makes still doesn’t match the complexity and nuance of an individual human being, though, and may never do so, simply because the AI is simulating embodiment and existing in the world. It might need to simulate an Earth equivalent to truly simulate a human’s personal output. In the essence side, it’s much more of a clear distinction. We have numerous ways of determining if a thing is human or not - biology, for one. It would take some serious sci-fi until we get to the point where an android is indistinguishable from a human on the cellular level.
    - 1y ago
yusina 1y ago
This heading is so obvious, I hardly see why it warrants an article. The other day I googled "Is it 2025?" and the AI reply was that nope, it's 2024. Such a joke.
- mdp2021 1y ago
  > [current time] You should know that LLMs are very weak in procedural operations. > obvious ... why it warrants an article The phenomenon is part of all the "obvious" things that are not in the awareness of very large masses.
  - yusina 1y ago
    > You should know that LLMs are very weak in procedural operations. Indeed! That's why they are LLMs, not AI! Why are new hypes always re-defining terms? 20 years ago, "AI" was actually about intelligence. Just like "crypto" was about cryptography instead of money scams. > The phenomenon is part of all the "obvious" things that are not in the awareness of very large masses. I can't imagine a single HN reader who is not aware that LLMs make mistakes. This is not the general public. (And even most of the general public has heard about this minor issue by now.)
    - mdp2021 1y ago
      > terms[!] ... ... I can't imagine a single HN reader who is not aware that LLMs make mistakes. This is not the general public So you wrote 'article' but you meant "submission" ;) > And even most of the general public has heard about this minor issue by now And still some public or private enterprises are trying to push LLMs in dangerous parts of the workflow - possibly because drives like "cutting costs" or "chasing waves" are valued more than objective results... > "AI" was actually about intelligence It was (is) about providing outputs intelligently - relevantly.
      - yusina 1y ago
        > So you wrote 'article' but you meant "submission" ;) Ah, true. Sorry.
Dwedit 1y ago
Duckduckgo and Brave Search's AI-generated answers seem to correctly mention that model 280 does not exist, but this is 11 days after the article was first published, and the article is now part of the search results used to generate the AI responses.
deleted 1y ago
[deleted]
Almondsetat 1y ago
"I asked an LLM and this is the wrong answer it gave me" is a genre of content that I'm growing ever more annoyed at. There is nothing constructive or informative about these stories other than the usual adage of not blindly trusting an LLM
- snarf_br 1y ago
  Then don't read it. With Gemini replacing Google search more and more people are blindly trusting those answers, so these stories are needed.
  - Almondsetat 1y ago
    Dismissing my observation with 'then don't read it' sidesteps the core issue. My point isn't about my personal reading habits, but about the low signal-to-noise ratio of this content genre. While you argue these stories are 'needed' because people blindly trust LLMs, especially with integrations like Gemini in search, these posts rarely offer more than the simplistic, already widely understood caution: 'don't blindly trust LLMs.' This is precisely the 'usual adage' I mentioned. The genre often lacks depth, failing to provide nuanced understanding or genuinely new information about why these systems fail in specific ways or how users can develop better critical assessment skills beyond mere distrust. If the goal is genuine education due to increased LLM exposure, the content needs to evolve beyond just showcasing errors.
- mdp2021 1y ago
  > informative That Google uses a faulty assistant in the page is actually informative, not just for people who do not use that search engine, but for those attentive to the progresses in the area - where Google has given massive hits recently. > constructive The - extremely damaging - replacement of experts with "employees wielding an LLM" is ongoing. Some of us have been told nonsense by remote support service staff...
  - Almondsetat 1y ago
    While you argue that showcasing a 'faulty assistant' like Google's is 'informative', particularly for those tracking AI progress, the typical LLM-got-it-wrong post often doesn't provide that deeper insight. It usually presents an isolated error without context or analysis of the system's architecture, training data limitations, or the specific type of reasoning failure. This makes its informative value quite shallow, quickly becoming repetitive rather than truly enlightening about 'progresses in the area' beyond the surface-level observation that LLMs are imperfect. Regarding the 'constructive' aspect and the 'damaging replacement of experts,' I agree this is a critical concern. However, the genre of simply posting screenshots of LLM errors is rarely constructive in addressing this complex socio-technical issue. It highlights a symptom (LLMs making mistakes) but typically fails to constructively engage with the causes or potential solutions for deskilling, corporate responsibility in AI deployment, or the nuances of human-AI collaboration. True constructive engagement would require more than just pointing out a wrong answer; it would demand analysis, discussion of best practices, or calls for better system design and oversight, which this genre seldom provides.
mdp2021 1y ago
The paradox is that that the "overview" is within a search engine, that should provide links to pages containing third party answers to the question; a rational individual knowing to suffer from bad memory would keep documentation at hand to verify its faulty-memory based intuition; the elements for the normal process ("get an intuition - if that is what you do - but verify it through (assessed) available documentation") are there but are not used.
csomar 1y ago
I had this a few months ago with an old man. He said there are 10 billion people in the world; so I told him you are off by 2 billion. He was adamant and challenged me to a Google search. So I did just that and lo and behold there are 10 billion people according to Google. I even took a screenshot: https://imgur.com/a/oQYKmKP https://imgur.com/a/oQYKmKP I really had nothing to say at that moment.
- ekianjo 1y ago
  Note that we don't really know the exact answer because population reporting is shaky at best where there are the most people.
  - mdp2021 1y ago
    And a rational interlocutor, besides conversational shortcuts, replies in the form "The best estimations from sources like S0, S1 and S2, publish a value between V1 and V2".
- deleted 1y ago
  [deleted]
system2 1y ago
The sun is hot, water is wet.
afro88 1y ago
If you look at the sources behind the AI responses for this search, they clearly don't mention the 280. Google are probably using a dirt cheap model for these responses and it's harming user trust
mpweiher 1y ago
In other exciting and groundshaking news: water wet! And I have to admit I thought the title was a joke. However, I loved the detailed description of just how bad it can be. And it puzzles me why people present AI slop as authoritative. Happens a lot in discussions these days. One example was someone presenting me with a grok answer about some aspect of the energy system. It turned out grok was off by a factor of 1000. Of course you can also use that to your advantage with people who believe AI slop, as it is fairly simple to get the AI to produce the answer you want, including the answer you know is right ;-) And I've actually started using AI a bit more in my coding, and it's been helpful as a starter. For example to get my little HTMX-Native project going, I needed to figure out how to configure Android's WebView for local data access. Would I have figured it out eventually? Yes. Was it faster with AI? Yes. Was the AI answer faulty? Yes. Was it still helpful? Yes.
hannob 1y ago
"AI Responses May Include Mistakes" is really the one, single most important thing I want to shout into the whole AI debate. It also should be the central issue - together with the energy/climate impacts - in every debate about AI ethics or AI safety. It's those two things that will harm us most if this hype continues unchecked.
- consp 1y ago
  The problem is not it may, but it will make mistakes. But people do not realize this and treat it as an almighty oracle. It's a statistical model after all, there is a non zero chance of the monkey creating the works Shakespeare.
jll29 1y ago
Language models are not designed to know things, they are designed to say things - that's why they are called language models and not knowledge models. Given a bunch of words have already been generated, it always ads the next words based on how common the sequence is. The reason you get different answers each time is the effect of the pseudo-random number generator on picking the next word. The model looks at the probability distribution of most likely next words, and when the configuration parameter called "temperature" is 0 (and it is actually not possible to set to 0 in the GUI), there is no random influence, and strictly the most likely next word (top-1 MLE) will always be chosen. This leads to output that we would classify as "very boring". So the model knows nothing about IBM, PS/2, 80286 versus 80486, CPUs, 280 or any models per se. -- One of the answers seems to suggest that there is no model 280, I wonder whether that one was generated through another process (there is a way to incorporate user feedback via "reinforcement learning"), or whether that was a consequence of the same randomized next-word picking, just a more lucky attempt.
- otabdeveloper4 1y ago
  > This leads to output that we would classify as "very boring". Not really. I set temperature to 0 for my local models, it works fine. The reason why the cloud UIs don't allow a temperature of 0 is because then models sometimes start to do infinite loops of tokens, and that would break the suspension of disbelief if the public saw it.
  - mdp2021 1y ago
    Which local models are you using, that do not output loop garbage at temperature 0? What do you get at very low temperature values instead of 0?
    - otabdeveloper4 1y ago
      > Which local models are you using, that do not output loop garbage at temperature 0? All of them. I make my own frontends using llama-cpp. Quality goes up with temperature 0 and loops are rare. The temperature setting isn't for improving quality, it's to not break your suspension of disbelief that you're talking to an intelligent entity.
simianwords 1y ago
In such discourse I never see discussion on this: There is no doubt that LLMs have gotten more accurate as newer models were released. At what point should we say "look this is accurate enough to be useful"? We should acknowledge that nothing is ever 100% accurate. You won't go to a doctor expecting 100% accuracy. You know that the doctor's accuracy is high enough for the effort of making an appointment, listening to them to be worth it. Maybe they are 60% accurate? My point is that LLM's are maybe at 20-30% accuracy where the benefit clearly exists even if we collectively acknowledge that 20-30% is not that high. I find it amusing to think about an LLM that is 1% accurate (which could have been achieved way earlier in 2010's). What could have been possible with such an LLM with the right mindset?
- demaga 1y ago
  Realistic-looking lorem ipsum
- gambiting 1y ago
  >>You won't go to a doctor expecting 100% accuracy. The way LLMs work at the moment is equivalent to going to a doctor with a set of symptoms, and the doctor telling you "ah yes, of course, it's illness X and you need to take medicine Y to cure it" and then you check and neither X nor Y exists. That's not "accuracy" that's just straight up fraud? I wouldn't have any problem with Google's AI saying "I don't know" or "I don't have enough sources to provide a good answer, but my best approximation is this". But I literally battle misinformation produced by Google's AI search every single day, because it's telling people actual made up facts that don't exist.
  - simianwords 1y ago
    I have a system prompt that gives me probability estimates to everything the LLM claims. Its a simple fix for your problem.
    - gambiting 1y ago
      My problem is people coming to communities I'm a part of with information they got "from Google" and that information being 100% wrong. Not sure how your prompt helps with that, I need Google to fix their system first.
sspiff 1y ago
I find this phenomenon really frustrating. I understand (or am at least aware of) the probabilistic nature of LLMs and their limitations, but when I point this out to my wife or friends when they are misusing LLMs for tasks they are both unsuited for and unreliable at, they wave their hands and dismiss my concerns as me being an AI cynic. They continue to use AI for math (asking LLMs to split bills, for example) and treat its responses for factual data lookup as 100% reliable and correct.
- osmsucks 1y ago
  > They continue to use AI for math (asking LLMs to split bills, for example) Ah, yes, high tech solutions for low tech problems. Let's use the word machine for this number problem!
  - thaumasiotes 1y ago
    > Let's use the word machine for this number problem! You know, that's a thought process that makes internal sense. You have someone who's terrible at math. They want something else to do math for them. Will they prefer to use a calculator, or a natural language interface? How do you use a calculator without knowing what you're doing?
    - osmsucks 1y ago
      Feels to me like if you can't even use a calculator you have bigger problems to worry about...
      - thaumasiotes 1y ago
        You don't think people who can't use a calculator ever have dinner with their friends?
        osmsucks 1y ago
        I don't know of anyone who can't even operate a calculator, but is at the same buddies with ChatGPT.
BlueTemplar 1y ago
> An expert will immediately notice discrepancies in the hallucinated answers, and will follow for example the List of IBM PS/2 Models article on Wikipedia. Which will very quickly establish that there is no Model 280. 'member when similar blogposts were written about not trusting Wikipedia ? (And Wikipedia is still better than LLMs : while you can trust it less than fixed, specialist-made references, you can improve it yourself, as well as check Talk pages for potential disagreements, and page history for potential shenanigans.)
nickjj 1y ago
I had an experience the other day with ChatGPT and some Python code. I wanted to modify Gunicorn's logger class so I can filter out specific URL paths. Given it's a hot code path (running on every request) I told it I made 3 solutions and was looking to see which one is the fastest. I used a list + loop using startswith, compiled regex and also used startswith while passing in a tuple of paths. It produced me benchmark code and also benchmark results which stated the regex solution was the best and fastest solution using Python's standard library. I didn't believe it so I ran the benchmark myself and the tuple version was over 5x faster than the regex solution. I then told it I ran the benchmark and got different results and it almost word for word said something like "Oh right, thank you for the clarification, the tuple version is indeed the fastest!". It saved me a few minutes writing the benchmark code but yeah, I rarely trust its output for anything I'm not 100% on.
sanbor 1y ago
I tried to reproduce the situation described in the post by Googling "ps/2 model 280". I refreshed the page about 15 times and gives a very similar answer every time. For example: The IBM PS/2 Model 280 was a low-end PS/2 model, often associated with the earlier IBM PC/XT and PC/AT models. It featured an 80286 CPU, a smaller hard drive, and a proprietary 3.5-inch floppy drive. While it used ISA slots for expansion, its proprietary RAM and other components made it less attractive for retro computing compared to the Model 30 286 I tried again after 10 minutes and I got the "The IBM PS/2 Model 280 was not a model in the original Personal System/2 family". It seems the AI response is cached for a few minutes. I tried in Kagi "ps/2 model 280?". This is the first response: ``` While a specific "PS/2 Model 280" is not definitively documented, here's what can be gathered: - It seems that AI models sometimes make errors regarding the PS/2 Model 280, describing it as an ISA-based 286 system. 12 - The IBM PS/2 (Personal System/2) was a line of personal computers from IBM, introduced in 1987 3. - PS/2 computers had two BIOSes: ABIOS (Advanced BIOS) and CBIOS (Compatible BIOS). 4 - One source mentions "CHIPS/280" including undocumented IBM PS/2 Model 80 address recovery logic. 5 - The PS/2 Model 30, released in 1987, used an Intel 8086 processor and had an 8-bit ISA bus. 6 - The PS/2 Model 30 286 had a 10MHz 286 CPU and could have 2 MB of RAM. 7 - The PS/2 Model 80 offered enhanced performance for commercial and compute-intensive tasks. 8 References AI Responses May Include Mistakes | OS/2 Museum www.os2museum.com OS/2 Museum | OS/2, vintage PC computing, and random musings www.os2museum.com List of IBM PS/2 models - Wikipedia en.wikipedia.org IBM PS/2 - Wikipedia en.wikipedia.org Technology ardent-tool.com IBM PS/2 Model 30 - Wikipedia en.wikipedia.org IBM PS/2 Model 30 286 - Computers of Significant History - Userlandia www.userlandia.com IBM PS/2 (Model 80) - Technical specifications ardent-tool.com ``` I waited a few minutes and this is the response now: ``` While information on the IBM PS/2 Model 280 is limited, here's what is available: - There is mention of the PS/2 Model 280, with claims that it "was a significant step forward in IBM's personal computer line" 1. However, it's also noted that claims about the PS/2 Model 280 being an ISA-based 286 system may be inaccurate 2. - The IBM PS/2 series included a variety of models in different form factors like desktops, towers, all-in-ones, portables, laptops, and notebooks 3. Some notable models include: Model 30: Featured an Intel 8086 processor at 8 MHz and an 8-bit ISA bus 4. Model 80: A high-end desktop with a 32-bit Intel 386 processor, initially running at 16 MHz, housed in a tower case 5. References AI Responses May Include Mistakes | OS/2 Museum www.os2museum.com OS/2 Museum | OS/2, vintage PC computing, and random musings www.os2museum.com List of IBM PS/2 models - Wikipedia en.wikipedia.org IBM PS/2 Model 30 - Wikipedia en.wikipedia.org IBM PS/2 Model 80 - Wikipedia en.wikipedia.org ``` I find the way Kagi gives references superior to the way Google does. Anyway, in the second answer Kagi's AI started to get a bit confused by stating "it's also noted that claims about the PS/2 Model 280 being an ISA-based 286 system may be inaccurate" missing the conclusion of the post.
ErrantX 1y ago
There is some parallel here with the message of 1984. In that Orwell saw political entity as the vehicle for mass misinformation. But in some ways Google AI results, with the trust they have, seem just another vehicle for "we've always been at war with Eurasia". For me this is the real risk of AI: developing dependence on it's output
veunes 1y ago
And the worst part is, the more plausible the output sounds, the more dangerous it becomes for casual users who don't know any better.
Hilift 1y ago
"Mistakes"? These are confabulations. There is a reason it is occurring, the code was written by people, who exhibit the same behavior. If you wonder why, what possible disincentive does an AI agent have to not confabulate the truth or facts?
rchaud 1y ago
Of course it can't admit it gets things wrong. That would signal to the user that it can't provide what you want. That's Product Design 101. Would you expect Netflix or Prime to simply show "No results" when you look up a show it doesn't have? Better to fill the screen with a bunch of "we think this is close enough" than to save the user some time.
johnea 1y ago
> AI simply makes stuff up. I do not consider made up, hallucinated answers useful, in fact they are worse than useless. Let's all say it together: The LLM is just WRONG. It's not "hallucinating", it doesn't have a brain, or consciousness. It's just generating a wrong answer.
akomtu 1y ago
If you asked this "AI" to draw a possible continuation of a fictional map, it would draw something very believable and everyone would understand that this map and its continuation is a fiction. No one would try to say that the continuation has a "mistake" in it. "AI" works with text the same way it works with some meaningless ornaments, but we interpret that text to find meaning, and this is why AI is fooling people so much.
YongHaoHu 1y ago
[dead]
410-572-7938 1y ago
[flagged]

184 points by userbinator 1y ago

Google's Gemini in search just makes up something that arbitrarily appears to support the query without care for context and accuracy. Pure confabulation. Try it for yourself. Ridiculous. It works as memory support if you know the result you're looking for, but if you don't, you can't trust it as far as you can throw it. If you look carefully at Google Veo output, it's similarly full of holes. It's plain there's no reasoning whatsoever informing the output. Veo output with goofy wrongness https://arstechnica.com/ai/2025/05/ai-video-just-took-a-startling-leap-in-realism-are-we-doomed/ https://arstechnica.com/ai/2025/05/ai-video-just-took-a-star... Tesla FSD goes crazy https://electrek.co/2025/05/23/tesla-full-self-driving-veers-off-road-flips-car-scary-crash-driver-couldnt-prevent/ https://electrek.co/2025/05/23/tesla-full-self-driving-veers...

ImPostingOnHN 1y ago
gemini is the worst LLM I've used, whether directly or through search. As in your experience, it regularly makes stuff up, like language/application features, or command flags (including regarding google products), and provides helpful references to sources which do not say what is cited from them. in my case, it does so roughly half the time, which is the worst proportion, because that means I can't even slightly rely upon the truth being the opposite of the output.
- JimDabell 1y ago
  Gemini was underwhelming until 2.5 Pro came along, which is very good. But in my experience all of the Google models are far worse than everything else when it comes to hallucination.
  - MangoToupe 1y ago
    As a corollary, though, the chatbots are probably the most creative.
    - mdp2021 1y ago
      Professional creatives do measure their intuitions against a number of constraints...
      - deleted 1y ago
        [deleted]
flomo 1y ago
I had a question about my car, so I googled '[year] [make] [model] [feature]'. This seems like the sort of thing that Google had always absolutely nailed. But now, 90% of the page was ai slop about wrong model, wrong year, even the wrong make. (There was one youtube which was sorta informative, so some credit.) But way way down on at the very the bottom of the page, there was the classic google search answer on a totally unrelated car forum. Thanks CamaroZ28.com!
- camillomiller 1y ago
  This is a very very good point. If this was happening with different queries that we never used, or a new type of questions/queries than I would have some patience. But it happens exactly with the formulations that were giving you the best results in the SERP before!
- TeMPOraL 1y ago
  That was true before AI too (I know, I did such searches myself). Google results has been drowning in slop for over a decade now - it was just human-generated slop, aka. content marketing and SEO stuff. I'm not defending the AI feature here, just trying to frame the problem: the lies and hallucinations were already there, but nobody cared because apparently people don't mind being constantly lied to by other people.
  - flomo 1y ago
    No, I'm not complaining about SEO shit... The thing is the primordial google had the answer, but Google themselves buried it under 100+ links of Google-generated slopdiarrhea, most of which didn't even factually fit the question, and was not at all relevant to my automobile.
    - ben_w 1y ago
      Indeed, this is part of the current monopoly abuse case they're facing — did Google deliberately choose to make search worse, because that causes people to return to the search results page and spend more time looking at ads, and they knew they could get away with it? e.g. bottom of first page, penultimate paragraph https://www.justice.gov/d9/2023-11/417557.pdf https://www.justice.gov/d9/2023-11/417557.pdf
MangoToupe 1y ago
I'm honestly so confused how people use LLMs as a replacement for search. All chatbots can ever find are data tangential to the stuff I want (eg i ask for a source, it gives me a quote). Maybe i was just holding search wrong?
- Garlef 1y ago
  Maybe that's because we're conditioned by the UX of search. But another thing I find even more surprising is that, at least initially, many expected that the LLMs would give them access to some form of higher truth.
  - MangoToupe 1y ago
    I think you might be on to something. I've found myself irritated that i can't just chuck keywords at LLMs.
- MaxikCZ 1y ago
  Its good to be shown direction. When I only have a vauge idea of what I want, AI usually helps me frame it into searchable terms I had no clue existed.
- mdp2021 1y ago
  > LLMs as a replacement for search Some people expect LLMs as part of a better "search". LLMs should be integrated to search, as a natural application: search results can heavily depend on happy phrasing, search engines work through sparse keywords, and LLMs allow to use structured natural language (not "foo bar baz" but "Which foo did a bar baz?" - which should be resistant to terms variation and exclude different semantics related to those otherwise sparse terms). But it has to be done properly - understand the question, find material, verify the material, produce a draft reply, verify the draft vis-a-vis the material, maybe iterate...
  - 1659447091 1y ago
    DuckDuckGo Ai assist is going in the right direction, imo. It will pull info from wikipedia, use math and map tools plus other web sources that has been mostly accurate for me on the search page. The chat option uses gpt-4o with web search and was able to provide links to colonial map resources I was curious about after falling down that rabbit hole. It also gave me general (& proper) present day map links to the places I was looking for in the map sites I asked for. It did get confused a few times when I was trying to get present day names of old places I had forgot; like Charles River in Va that it kept trying to send me to Boston or Charles City Co on the James river and told me to look for it around there... The York river wiki page clearly says it was once Charles River. Maybe I wasn't asking the right questions. For more unique things it was pretty helpful thou and saved the endless searching w/ 100 tabs adventure
camillomiller 1y ago
This baffles me like no other tech has done before. Google is betting its own core business on a pivot that relies on a massively faulty piece of technology. And as Ben Evans also says, promising that it will get better only gets you so far, it’s an empty promise. Yesterday AI overview made up an entire album by a dead Italian musician when I searched for a tribute event that was happening at a Berlin venue. It just took the name of the venue and claimed it was the most important work from that artist. Funnily enough (not for Google), I copypasted that answer on chatGPT and it roasted AI Overview so bad on its mistakes and with such sarcasm that it even made me chuckle.
- DanHulton 1y ago
  It's the unfounded promises that this will be solved because the tech will only get better that really upset me. Because sure, it will get better, I'm pretty certain of that. They'll have additional capabilities, they'll have access to more-recent data, etc. But "better" does not necessarily equate to "will fix the lying problem." That's a problem that is BAKED INTO the technology, and requires some kind of different approach to solve -- you can't just keep making a hammer bigger and bigger in the hopes that one day it'll turn into a screwdriver. Before LLMs really took off, we were in the middle of an "AI winter", where there just weren't any promising techs, at least none with sufficient funding attached to them. And it's WORSE now. LLMs have sucked all the air out of the room, and all of the funding out of other avenues of research. Technologies that were "10-20" years away now might be 30-40, because there's fewer people researching them, with less money, and they might even be completely different people trying to restart the research after the old ones got recruited away to work on LLMs!
  - mountainriver 1y ago
    I really don’t understand the whole AI winter talk all the time. We haven’t had anything of the sort since 2008. There were tons of major RL advancements before ChatGPT that were stunning. I would challenge anyone to find data to actually support any of these claims. ML spending has been up since deep learning year over year and the models just keep getting better
XorNot 1y ago
I use ublock to remove Gemini responses from search, because even glancing at then is liable to bias my assumption about whatever I'm looking for. Information hygiene is a skill which started out important but is going to become absolutely critical.
- justmarc 1y ago
  We can't expect the vast majority of regular users to have any of that skill. What is this going to lead to? fascinating times.
  - jobigoud 1y ago
    It's very easy though, Right click > Block element > Create. Overlays show which blocks you are removing. Sliders can be used to increase/refine. How can we make it even easier and visual? Customizing pages by removing elements should be part of the default browser experience to be honest. Like in the initial web where you would tweak the color of links, visited links, etc.
- MaxikCZ 1y ago
  Half my browser extensions have sole purpose of removing shit from sites I visit. HN is like a unicorn that havent made me block a single thing yet.
  - alpaca128 1y ago
    Ironically that's an AI tool I would use - one that can dynamically filter content from sites according to my preferences, to counter the algorithmic spam. It wouldn't be generative AI though, and that's the only kind of AI that matters right now apparently.
    - jazzyjackson 1y ago
      Could be a good use of structured output, Llms work okay as one shot classifiers, you could define a schema that's just like, array[]{xpath:true/false} and tell the bot what you want to see and what you don't want to see.
justmarc 1y ago
And suddenly this type of quality is becoming "normal" and acceptable now? Nobody really complains. That is very worrying. Normally this would never fly, but nowadays it's kind of OK? Why should false and or inaccurate results be accepted?
- celdon25 1y ago
  Complain to it enough times, remain resilient and you’ll eventually figure it out (that’s a wild card though). Or find someone who has and take their word for it (except you can’t because they’re probably indistinguishable from the ‘bot’ now according to the contradictory narrative). Iterate. Spiral. No one should have to go through that though. Be merciful.
- chronid 1y ago
  Suddenly? That's the level of quality that is standard in all software projects I've ever seen since I've started working in IT. Enshittification is all around us and is unstoppable. Because we have deadlines to hit and goals to shows we reached to the VP. We broke everything and the software is just half working? Come on that's an issue for the support and ops teams. On to the next beautiful feature we can put on marketing slides!
  - justmarc 1y ago
    Sadly you are absolutely right.
- TeMPOraL 1y ago
  We lost that battle back when we collectively decided that sales and marketing is respectable work.
  - bheadmaster 1y ago
    Hah. Good observatoon. I often get in arguments about how I tend to avoid brands that put too much into marketing. Of course, theoretically, the amount of money a company puts into marketing doesn't automatically lower the quality of their products, but in my experience, the correlation is there. Whiskas, Coka Cola, McDonalds, etc.
Nursie 1y ago
Yep, I was looking up a hint for the Blue Prince game the other day for the (spoiler alert?) casino room. Google’s AI results proceeded to tell me all about the games available at the Blue Prince Casino down the road from here, where I know for a fact there’s only a prison, a Costco, a few rural properties and a whole lot of fuck-all. It’s amazing to watch it fill in absolute false, fabricated tripe at the top of their search page. It also frequently returns bad information on subjects like employment law and whatever else I look up. It would be hilarious if people weren’t actually relying on it.
- datavirtue 1y ago
  I the have had a lot of luck with copilot conversations to research stocks and trading strategies. I am always skeptical of results and verify everything with various sources but it does help me find/get on the right track.
Kwpolska 1y ago
Google recently started showing me their AI bullshit. This made me pull the trigger and switch to DuckDuckGo as the primary search engine. That said, some niche stuff has significantly better results in Google. But not in the AI bullshit. I searched for a very niche train-related, the bullshit response said condescendingly "this word does not exist, maybe you meant [similarly sounding but completely different word], which in the context of trains means ...". The first real result? Turns out that word does exist.
- christophilus 1y ago
  What’s the word?
- dijksterhuis 1y ago
  fyi, you can remove any and all “ai” assistant bs etc from DDG if you use the noai subdomain (in case you wanna avoid their stuff, although it’s much less prominent anyway) https://noai.duckduckgo.com/ https://noai.duckduckgo.com/
- datavirtue 1y ago
  I switched to DDG over seven years ago and just realized it had been that long when I read your comment. Google started wasting my time and I had to shift.
christophilus 1y ago
I’ve had good results with Brave search, which self reports to use: Meta Llama 3, Mistral / Mixtral, and CodeLLM. It’s not always 100% accurate, but it’s almost always done the trick and saved me digging through more docs than necessary.
veunes 1y ago
Yeah, it feels like we've crossed into a weird uncanny valley where AI outputs sound smarter than ever, but the underlying logic (or lack thereof) hasn't caught up
- roywiggins 1y ago
  I think it's just much easier for an LLM to learn how to be convincing than it is to actually be accurate. It just has to convince RLHF trainers that it's right, not actually be right. And the first one is a general skill that can be learned and applied to anything. https://arxiv.org/html/2409.12822v1 https://arxiv.org/html/2409.12822v1

minimaxir 1y ago

The simple "AI responses may include mistakes" disclaimer or ChatGPT's "ChatGPT can make mistakes. Check important info." CYA text at the bottom of the UI are clearly no longer sufficient. After years of news stories about LLM hallucinations in fact-specific domains and people still getting burnt by them, LLM providers should be more aggressive in educating users about their fallability since hallucinations can't ever be fully fixed, even if it means adding friction.

tbrownaw 1y ago
> should be more aggressive in educating users about their fallability This might be an "experience is the best teacher" situation. It'd probably be pretty hard to invent a disclaimer that'd be as effective as getting bit.
- minimaxir 1y ago
  Unfortunately, getting bit in cases such as publishing misinformation or false legal citations waste everyone time, not just their own.
eddythompson80 1y ago
That doesn't really make sense. You either make the LLM provider liable for the output of the model, or you have the current model. The friction already exists. All these AI companies and cloud providers are running "censored models" and more censorship is added at every layer. What would more friction be here? more pop-ups? Doing the former basically means killing the model-hosting business. Companies could develop models, use them internally and give them to their employees, but no public APIs exists. Companies strike legally binding contracts to use/license each other models, but the general public doesn't have access to those without something that would mitigate the legal risk. Maybe years down the line, as attitudes soften, some companies would begin to push the boundaries. Automating the legal approval process, opening signups, etc.
- minimaxir 1y ago
  Yes, more popups, retention metrics be damned. Even 2 years since ChatGPT, many people still think it's omniscent which is what's causing trouble.

Biganon 1y ago

Make up a fake popular wisdom saying that sounds real, search for it, Gemini will gladly tell you it exists and explain it to you

mdp2021 1y ago
...As others have noted elsewhere: which Gemini? There are cheap ones and others proposed as flagship.
- ghusbands 1y ago
  The one that appears when you search with Google.

elmerfud 1y ago

AI is like that one guy who always can tell you something about anything with total confidence. So really not sure why anyone would trust it beyond a bar conversation.

vouaobrasil 1y ago
I think it's psychological. Most people use visual body cues to determine whether someone is lacking in confidence in their answer. AI does not have any cues to show a lack of confidence, and people also have a high trust in machine output because traditional algorithms always give the correct answer. The percentage of people that will look at it critically is negligible.
- normie3000 1y ago
  > Most people use visual body cues to determine whether someone is lacking in confidence in their answer Do they?
  - vouaobrasil 1y ago
    I can certainly tell when someone is just bull**ing from their tone of voice long before they tell me the information.
    - tonyedgecombe 1y ago
      You obviously haven’t met any of my past bosses. Some people have turned bullshitting into an art form.
    - ghusbands 1y ago
      Which is notably not a "visual body cue"
JdeBP 1y ago
No "AI" company has yet had the bravery to name its product Cliff Clavin. Bravery in several ways, that is. There's the risk of being sued by John Ratzenberger. (-:

charcircuit 1y ago

Gemini can handle this fine.

mucha 1y ago
Capability != Reliability
nehal3m 1y ago
Huh, I thought that AI overview feature is powered by Gemini.
- charcircuit 1y ago
  Gemini doesn't refer to a specific model so perhaps the one on the search page is weaker than the ones it offers in the app.
  - mdp2021 1y ago
    They may have estimated the volume of replies in those "overview" pages and shrunk the costs routing through a computationally lightweight model. They may this way also have underestimated the reputational loss - the big umbrella of the "Ford Pinto case". -- Edit: I was just now looking at the new Visual Capitalist's "Ranked: 2025’s 10 Largest S&P 500 Stocks". Is it possible that Alphabet being at the top with 7.6% of the weight of the 500-items set is paradoxically allowing to afford more damage?

9x39 1y ago

Gemini appears tuned to try to handle the typical questions people type in, while more traditional things you search for get some confabulated nonsense. I've observed a great deal of people trust the AI Overview as an oracle. IMO, it's how 'normal' people interact with AI if they aren't direct LLM users. It's not even age gated like trusting the news - trusting AI outputs seems to cross most demographics. We love our confident-based-on-nothing computer answers as a species, I think.

danielbln 1y ago
We love our confident-based-on-nothing answers period, computer or not.
- chneu 1y ago
  Most folks just want confirmation. They don't want to have their views/opinions changed. LLM are good at trying to give folks what they're looking for.
- mdp2021 1y ago
  Repent. You are not there to "love what gives you the kicks". That's a kind of love that should not exit the bedroom (better, the bathroom).
Llamamoe 1y ago
I already went through a realization a while ago that you just can't mention something to people anymore and expect them to be able to learn about it by searching the web, like it used to be possible, because everything is just unreliable misleading SEO spam slop. I shudder to think how much worse this is going to be with "AI Overview". Are we entering an era of people googling "how does a printer work" and (possibly) being told that it's built by a system of pulleys and ropes and just trusting it blindly? Because that's the kind of magnitude of errors I've seen in dozens of searches I've made in the domains I'm interested in, and I think everyone has seen the screenshots of even more outlandish - or outright dangerous - answers.
eddythompson80 1y ago

jimmySixDOF 1y ago

Error control as in LLM-as-a-Judge needs to be integrated into every pipeline there are use at home (sub 7B) sized models and nano's (fit into browser) [1] so push out to the edge if its a Google sized scale/cost problem. Should actually be like VirusTotal where you get a consensus score of x/72 different lookups. [1] see haize labs : https://www.haizelabs.com/product/judges https://www.haizelabs.com/product/judges

eddythompson80 1y ago
This is a perfect example of snakeoil nonesense being sold in the AI tech market these days. People offering to wrap your LLM calls with another prompt asking the LLM to "reconsider" or "does this make sense to you", and selling you that artisanally crafted prompt at a premium. This is simply an information retrieval problem. LLMs don't retrieve information perfectly from their context. They are very good at it, but every now and then they'll introduce a change here or there. Changing an "Hello" to "Hi" doesn't really make any difference, but changing an "PS/2 Model 286" to "PS/2 Model 280" makes a huge difference. The LLM "knows"* this at some level because it "knows" that names are important to be reproduced in exact format. But everynow and then even names can change and sill generally mean the same thing, so everynow and then it'll change a name or an identifier for no reason. some of my favorite descriptions of this I have heard from people: - We need to introduce a reflection layer - We need a supervisor tree-like checks on responses - We need an evaluation-feedback mechanism for AI operations - We need to agents that operates and judges/supervisors/evaluators all apparently mean: const response = await getChatResponse([...messages, newMsg]); const {didGood, why} = await getChatResponse([RolePlayLikeAMeanJudgePromot, ...messages, newMsg, response]) if (!didGoog) { response = await getChatResonse([ThreateningMafiaGuyPrompt, [ ...messages, newMsg, response, why[]) } // loop like 4 times, maybe use different models and egg them on each other, like use Grok and tell it you got the response from Claude and that Claude was shit talking Grok all the way through it. Like it was unnecessary tbh. // this makes Grook work extra hard. It's actually been measured by kill-a-watt. *: I say "knows" to indicate just that the model training biases it that way
- mdp2021 1y ago

rcarmo 1y ago

This is why Google has got search fundamentally wrong. They just don’t care about accuracy of results anymore, and worry mostly about providing a quick answer and a bunch of sponsored links below it.

Llamamoe 1y ago
Except that out of 10 answers, the "quick answer" is subtly wrong 6 times, egregiously wrong 2, and outright dangerous once. I've seen screenshots of stuff that would get people killed or in legal trouble.
dandanua 1y ago
They just continue the Eric Schmidt idea "More results are better than none". It has evolved to "It's better to hallucinate than produce a negative answer", I guess.

jkuli 1y ago

Humans can also make mistakes. This is the first test I apply every time. Could it be that AI is actually more capable than a human? If industry decides than humans are more reliable, they will choose to use humans. Reliability is part of cost-effectiveness, and it's built in to the business decision process.

theodric 1y ago
A human would only make up an answer like the ones in the article if it were a compulsive liar. A human would ideally say "I don't know" or at worst employ the "I'll confirm that and circle back" corpspeak evasion.
mdp2021 1y ago
Another case of badly computed reputational loss. That context is not of "this more than that" comparisons, but of threshold: the service must be _adequate_. If you don't have random humans capable of providing the needed service, find them. Do not employ random agents.
alpaca128 1y ago
The average Google user expects LLMs to be perfect and treats their responses as answer from an oracle, not just better than the average human. It doesn't matter whether it's better than humans, the one thing that matters are the consequences of its widespread use.

neilv 1y ago

On the Google search Web site, the "AI responses may include mistakes." weak disclaimer small print is also hidden behind the "Show more" button. When OpenAI launched ChatGPT, I had to explain to a non-CS professor that it wasn't AI like they're thinking of, but currently more like a computational parlor trick that looks a lot like AI. But turns out this parlor trick is awesome for cheating on homework. Also good at cheating at many other kinds of work, if you don't care much about quality, nor about copyrights.

loa_in_ 1y ago
It's a memory augmentation/information retrieval tool with flexible input and output interface.
stavros 1y ago
I really don't understand the view that it's a "parlor trick that looks like AI". If it's not "a thing that can write code", but instead just looks like a thing that can write code (but can actually write code), it can write code. All the "no true Scotsman" stuff about what it's doing behind the scenes is irrelevant, because we have no idea what human brains are doing behind the scenes either.
- keiferski 1y ago
  It matters if we are making a distinction between essence and output. On the output side, it functionally doesn’t really have a difference. At least in terms of more abstract things like writing code. Although I would argue that the output AI makes still doesn’t match the complexity and nuance of an individual human being, though, and may never do so, simply because the AI is simulating embodiment and existing in the world. It might need to simulate an Earth equivalent to truly simulate a human’s personal output. In the essence side, it’s much more of a clear distinction. We have numerous ways of determining if a thing is human or not - biology, for one. It would take some serious sci-fi until we get to the point where an android is indistinguishable from a human on the cellular level.
  - 1y ago

yusina 1y ago

This heading is so obvious, I hardly see why it warrants an article. The other day I googled "Is it 2025?" and the AI reply was that nope, it's 2024. Such a joke.

mdp2021 1y ago
> [current time] You should know that LLMs are very weak in procedural operations. > obvious ... why it warrants an article The phenomenon is part of all the "obvious" things that are not in the awareness of very large masses.
- yusina 1y ago
  > You should know that LLMs are very weak in procedural operations. Indeed! That's why they are LLMs, not AI! Why are new hypes always re-defining terms? 20 years ago, "AI" was actually about intelligence. Just like "crypto" was about cryptography instead of money scams. > The phenomenon is part of all the "obvious" things that are not in the awareness of very large masses. I can't imagine a single HN reader who is not aware that LLMs make mistakes. This is not the general public. (And even most of the general public has heard about this minor issue by now.)
  - mdp2021 1y ago
    > terms[!] ... ... I can't imagine a single HN reader who is not aware that LLMs make mistakes. This is not the general public So you wrote 'article' but you meant "submission" ;) > And even most of the general public has heard about this minor issue by now And still some public or private enterprises are trying to push LLMs in dangerous parts of the workflow - possibly because drives like "cutting costs" or "chasing waves" are valued more than objective results... > "AI" was actually about intelligence It was (is) about providing outputs intelligently - relevantly.
    - yusina 1y ago
      > So you wrote 'article' but you meant "submission" ;) Ah, true. Sorry.

Dwedit 1y ago

Duckduckgo and Brave Search's AI-generated answers seem to correctly mention that model 280 does not exist, but this is 11 days after the article was first published, and the article is now part of the search results used to generate the AI responses.

deleted 1y ago

[deleted]

Almondsetat 1y ago

"I asked an LLM and this is the wrong answer it gave me" is a genre of content that I'm growing ever more annoyed at. There is nothing constructive or informative about these stories other than the usual adage of not blindly trusting an LLM

snarf_br 1y ago
Then don't read it. With Gemini replacing Google search more and more people are blindly trusting those answers, so these stories are needed.
- Almondsetat 1y ago
  Dismissing my observation with 'then don't read it' sidesteps the core issue. My point isn't about my personal reading habits, but about the low signal-to-noise ratio of this content genre. While you argue these stories are 'needed' because people blindly trust LLMs, especially with integrations like Gemini in search, these posts rarely offer more than the simplistic, already widely understood caution: 'don't blindly trust LLMs.' This is precisely the 'usual adage' I mentioned. The genre often lacks depth, failing to provide nuanced understanding or genuinely new information about why these systems fail in specific ways or how users can develop better critical assessment skills beyond mere distrust. If the goal is genuine education due to increased LLM exposure, the content needs to evolve beyond just showcasing errors.
mdp2021 1y ago
> informative That Google uses a faulty assistant in the page is actually informative, not just for people who do not use that search engine, but for those attentive to the progresses in the area - where Google has given massive hits recently. > constructive The - extremely damaging - replacement of experts with "employees wielding an LLM" is ongoing. Some of us have been told nonsense by remote support service staff...
- Almondsetat 1y ago
  While you argue that showcasing a 'faulty assistant' like Google's is 'informative', particularly for those tracking AI progress, the typical LLM-got-it-wrong post often doesn't provide that deeper insight. It usually presents an isolated error without context or analysis of the system's architecture, training data limitations, or the specific type of reasoning failure. This makes its informative value quite shallow, quickly becoming repetitive rather than truly enlightening about 'progresses in the area' beyond the surface-level observation that LLMs are imperfect. Regarding the 'constructive' aspect and the 'damaging replacement of experts,' I agree this is a critical concern. However, the genre of simply posting screenshots of LLM errors is rarely constructive in addressing this complex socio-technical issue. It highlights a symptom (LLMs making mistakes) but typically fails to constructively engage with the causes or potential solutions for deskilling, corporate responsibility in AI deployment, or the nuances of human-AI collaboration. True constructive engagement would require more than just pointing out a wrong answer; it would demand analysis, discussion of best practices, or calls for better system design and oversight, which this genre seldom provides.

mdp2021 1y ago

The paradox is that that the "overview" is within a search engine, that should provide links to pages containing third party answers to the question; a rational individual knowing to suffer from bad memory would keep documentation at hand to verify its faulty-memory based intuition; the elements for the normal process ("get an intuition - if that is what you do - but verify it through (assessed) available documentation") are there but are not used.

csomar 1y ago

I had this a few months ago with an old man. He said there are 10 billion people in the world; so I told him you are off by 2 billion. He was adamant and challenged me to a Google search. So I did just that and lo and behold there are 10 billion people according to Google. I even took a screenshot: https://imgur.com/a/oQYKmKP https://imgur.com/a/oQYKmKP I really had nothing to say at that moment.

ekianjo 1y ago
Note that we don't really know the exact answer because population reporting is shaky at best where there are the most people.
- mdp2021 1y ago
  And a rational interlocutor, besides conversational shortcuts, replies in the form "The best estimations from sources like S0, S1 and S2, publish a value between V1 and V2".
deleted 1y ago
[deleted]

system2 1y ago

The sun is hot, water is wet.

afro88 1y ago

If you look at the sources behind the AI responses for this search, they clearly don't mention the 280. Google are probably using a dirt cheap model for these responses and it's harming user trust

mpweiher 1y ago

In other exciting and groundshaking news: water wet! And I have to admit I thought the title was a joke. However, I loved the detailed description of just how bad it can be. And it puzzles me why people present AI slop as authoritative. Happens a lot in discussions these days. One example was someone presenting me with a grok answer about some aspect of the energy system. It turned out grok was off by a factor of 1000. Of course you can also use that to your advantage with people who believe AI slop, as it is fairly simple to get the AI to produce the answer you want, including the answer you know is right ;-) And I've actually started using AI a bit more in my coding, and it's been helpful as a starter. For example to get my little HTMX-Native project going, I needed to figure out how to configure Android's WebView for local data access. Would I have figured it out eventually? Yes. Was it faster with AI? Yes. Was the AI answer faulty? Yes. Was it still helpful? Yes.

hannob 1y ago

"AI Responses May Include Mistakes" is really the one, single most important thing I want to shout into the whole AI debate. It also should be the central issue - together with the energy/climate impacts - in every debate about AI ethics or AI safety. It's those two things that will harm us most if this hype continues unchecked.

consp 1y ago
The problem is not it may, but it will make mistakes. But people do not realize this and treat it as an almighty oracle. It's a statistical model after all, there is a non zero chance of the monkey creating the works Shakespeare.

jll29 1y ago

Language models are not designed to know things, they are designed to say things - that's why they are called language models and not knowledge models. Given a bunch of words have already been generated, it always ads the next words based on how common the sequence is. The reason you get different answers each time is the effect of the pseudo-random number generator on picking the next word. The model looks at the probability distribution of most likely next words, and when the configuration parameter called "temperature" is 0 (and it is actually not possible to set to 0 in the GUI), there is no random influence, and strictly the most likely next word (top-1 MLE) will always be chosen. This leads to output that we would classify as "very boring". So the model knows nothing about IBM, PS/2, 80286 versus 80486, CPUs, 280 or any models per se. -- One of the answers seems to suggest that there is no model 280, I wonder whether that one was generated through another process (there is a way to incorporate user feedback via "reinforcement learning"), or whether that was a consequence of the same randomized next-word picking, just a more lucky attempt.

otabdeveloper4 1y ago
> This leads to output that we would classify as "very boring". Not really. I set temperature to 0 for my local models, it works fine. The reason why the cloud UIs don't allow a temperature of 0 is because then models sometimes start to do infinite loops of tokens, and that would break the suspension of disbelief if the public saw it.
- mdp2021 1y ago
  Which local models are you using, that do not output loop garbage at temperature 0? What do you get at very low temperature values instead of 0?
  - otabdeveloper4 1y ago
    > Which local models are you using, that do not output loop garbage at temperature 0? All of them. I make my own frontends using llama-cpp. Quality goes up with temperature 0 and loops are rare. The temperature setting isn't for improving quality, it's to not break your suspension of disbelief that you're talking to an intelligent entity.

simianwords 1y ago

In such discourse I never see discussion on this: There is no doubt that LLMs have gotten more accurate as newer models were released. At what point should we say "look this is accurate enough to be useful"? We should acknowledge that nothing is ever 100% accurate. You won't go to a doctor expecting 100% accuracy. You know that the doctor's accuracy is high enough for the effort of making an appointment, listening to them to be worth it. Maybe they are 60% accurate? My point is that LLM's are maybe at 20-30% accuracy where the benefit clearly exists even if we collectively acknowledge that 20-30% is not that high. I find it amusing to think about an LLM that is 1% accurate (which could have been achieved way earlier in 2010's). What could have been possible with such an LLM with the right mindset?

demaga 1y ago
Realistic-looking lorem ipsum
gambiting 1y ago
>>You won't go to a doctor expecting 100% accuracy. The way LLMs work at the moment is equivalent to going to a doctor with a set of symptoms, and the doctor telling you "ah yes, of course, it's illness X and you need to take medicine Y to cure it" and then you check and neither X nor Y exists. That's not "accuracy" that's just straight up fraud? I wouldn't have any problem with Google's AI saying "I don't know" or "I don't have enough sources to provide a good answer, but my best approximation is this". But I literally battle misinformation produced by Google's AI search every single day, because it's telling people actual made up facts that don't exist.
- simianwords 1y ago
  I have a system prompt that gives me probability estimates to everything the LLM claims. Its a simple fix for your problem.
  - gambiting 1y ago
    My problem is people coming to communities I'm a part of with information they got "from Google" and that information being 100% wrong. Not sure how your prompt helps with that, I need Google to fix their system first.

sspiff 1y ago

I find this phenomenon really frustrating. I understand (or am at least aware of) the probabilistic nature of LLMs and their limitations, but when I point this out to my wife or friends when they are misusing LLMs for tasks they are both unsuited for and unreliable at, they wave their hands and dismiss my concerns as me being an AI cynic. They continue to use AI for math (asking LLMs to split bills, for example) and treat its responses for factual data lookup as 100% reliable and correct.

osmsucks 1y ago
> They continue to use AI for math (asking LLMs to split bills, for example) Ah, yes, high tech solutions for low tech problems. Let's use the word machine for this number problem!
- thaumasiotes 1y ago
  > Let's use the word machine for this number problem! You know, that's a thought process that makes internal sense. You have someone who's terrible at math. They want something else to do math for them. Will they prefer to use a calculator, or a natural language interface? How do you use a calculator without knowing what you're doing?
  - osmsucks 1y ago
    Feels to me like if you can't even use a calculator you have bigger problems to worry about...
    - thaumasiotes 1y ago
      You don't think people who can't use a calculator ever have dinner with their friends?
      - osmsucks 1y ago
        I don't know of anyone who can't even operate a calculator, but is at the same buddies with ChatGPT.

BlueTemplar 1y ago

> An expert will immediately notice discrepancies in the hallucinated answers, and will follow for example the List of IBM PS/2 Models article on Wikipedia. Which will very quickly establish that there is no Model 280. 'member when similar blogposts were written about not trusting Wikipedia ? (And Wikipedia is still better than LLMs : while you can trust it less than fixed, specialist-made references, you can improve it yourself, as well as check Talk pages for potential disagreements, and page history for potential shenanigans.)

nickjj 1y ago

I had an experience the other day with ChatGPT and some Python code. I wanted to modify Gunicorn's logger class so I can filter out specific URL paths. Given it's a hot code path (running on every request) I told it I made 3 solutions and was looking to see which one is the fastest. I used a list + loop using startswith, compiled regex and also used startswith while passing in a tuple of paths. It produced me benchmark code and also benchmark results which stated the regex solution was the best and fastest solution using Python's standard library. I didn't believe it so I ran the benchmark myself and the tuple version was over 5x faster than the regex solution. I then told it I ran the benchmark and got different results and it almost word for word said something like "Oh right, thank you for the clarification, the tuple version is indeed the fastest!". It saved me a few minutes writing the benchmark code but yeah, I rarely trust its output for anything I'm not 100% on.

sanbor 1y ago

I tried to reproduce the situation described in the post by Googling "ps/2 model 280". I refreshed the page about 15 times and gives a very similar answer every time. For example: The IBM PS/2 Model 280 was a low-end PS/2 model, often associated with the earlier IBM PC/XT and PC/AT models. It featured an 80286 CPU, a smaller hard drive, and a proprietary 3.5-inch floppy drive. While it used ISA slots for expansion, its proprietary RAM and other components made it less attractive for retro computing compared to the Model 30 286 I tried again after 10 minutes and I got the "The IBM PS/2 Model 280 was not a model in the original Personal System/2 family". It seems the AI response is cached for a few minutes. I tried in Kagi "ps/2 model 280?". This is the first response: ``` While a specific "PS/2 Model 280" is not definitively documented, here's what can be gathered: - It seems that AI models sometimes make errors regarding the PS/2 Model 280, describing it as an ISA-based 286 system. 12 - The IBM PS/2 (Personal System/2) was a line of personal computers from IBM, introduced in 1987 3. - PS/2 computers had two BIOSes: ABIOS (Advanced BIOS) and CBIOS (Compatible BIOS). 4 - One source mentions "CHIPS/280" including undocumented IBM PS/2 Model 80 address recovery logic. 5 - The PS/2 Model 30, released in 1987, used an Intel 8086 processor and had an 8-bit ISA bus. 6 - The PS/2 Model 30 286 had a 10MHz 286 CPU and could have 2 MB of RAM. 7 - The PS/2 Model 80 offered enhanced performance for commercial and compute-intensive tasks. 8 References AI Responses May Include Mistakes | OS/2 Museum www.os2museum.com OS/2 Museum | OS/2, vintage PC computing, and random musings www.os2museum.com List of IBM PS/2 models - Wikipedia en.wikipedia.org IBM PS/2 - Wikipedia en.wikipedia.org Technology ardent-tool.com IBM PS/2 Model 30 - Wikipedia en.wikipedia.org IBM PS/2 Model 30 286 - Computers of Significant History - Userlandia www.userlandia.com IBM PS/2 (Model 80) - Technical specifications ardent-tool.com ``` I waited a few minutes and this is the response now: ``` While information on the IBM PS/2 Model 280 is limited, here's what is available: - There is mention of the PS/2 Model 280, with claims that it "was a significant step forward in IBM's personal computer line" 1. However, it's also noted that claims about the PS/2 Model 280 being an ISA-based 286 system may be inaccurate 2. - The IBM PS/2 series included a variety of models in different form factors like desktops, towers, all-in-ones, portables, laptops, and notebooks 3. Some notable models include: Model 30: Featured an Intel 8086 processor at 8 MHz and an 8-bit ISA bus 4. Model 80: A high-end desktop with a 32-bit Intel 386 processor, initially running at 16 MHz, housed in a tower case 5. References AI Responses May Include Mistakes | OS/2 Museum www.os2museum.com OS/2 Museum | OS/2, vintage PC computing, and random musings www.os2museum.com List of IBM PS/2 models - Wikipedia en.wikipedia.org IBM PS/2 Model 30 - Wikipedia en.wikipedia.org IBM PS/2 Model 80 - Wikipedia en.wikipedia.org ``` I find the way Kagi gives references superior to the way Google does. Anyway, in the second answer Kagi's AI started to get a bit confused by stating "it's also noted that claims about the PS/2 Model 280 being an ISA-based 286 system may be inaccurate" missing the conclusion of the post.

ErrantX 1y ago

There is some parallel here with the message of 1984. In that Orwell saw political entity as the vehicle for mass misinformation. But in some ways Google AI results, with the trust they have, seem just another vehicle for "we've always been at war with Eurasia". For me this is the real risk of AI: developing dependence on it's output

veunes 1y ago

And the worst part is, the more plausible the output sounds, the more dangerous it becomes for casual users who don't know any better.

Hilift 1y ago

"Mistakes"? These are confabulations. There is a reason it is occurring, the code was written by people, who exhibit the same behavior. If you wonder why, what possible disincentive does an AI agent have to not confabulate the truth or facts?

rchaud 1y ago

Of course it can't admit it gets things wrong. That would signal to the user that it can't provide what you want. That's Product Design 101. Would you expect Netflix or Prime to simply show "No results" when you look up a show it doesn't have? Better to fill the screen with a bunch of "we think this is close enough" than to save the user some time.

johnea 1y ago

> AI simply makes stuff up. I do not consider made up, hallucinated answers useful, in fact they are worse than useless. Let's all say it together: The LLM is just WRONG. It's not "hallucinating", it doesn't have a brain, or consciousness. It's just generating a wrong answer.

akomtu 1y ago

If you asked this "AI" to draw a possible continuation of a fictional map, it would draw something very believable and everyone would understand that this map and its continuation is a fiction. No one would try to say that the continuation has a "mistake" in it. "AI" works with text the same way it works with some meaningless ornaments, but we interpret that text to find meaning, and this is why AI is fooling people so much.

YongHaoHu 1y ago

[dead]

410-572-7938 1y ago

[flagged]