7 ms·

I really hope this stays up, despite the politics involvement to a degree. I think this is a situation that is a perfect example of how AI hallucinations/lack o

by AnEro 1y ago

I really hope this stays up, despite the politics involvement to a degree. I think this is a situation that is a perfect example of how AI hallucinations/lack of accuracy could significantly impact our lives going forward. A very nuanced and serious topic with lots of back and forth being distilled down to headlines by any source, it is a terrifying reality. Especially if we aren't able to communicate how these tools work to the public. (if they even will care to learn it) At least when humans did this they knew at some level at least they skimmed the information on the person/topic.

jaccola 1y ago
This story will probably become big enough to drown out the fake video and the AI (which is presumably being fed top n search results) will automatically describe this fake video controversy instead...
- deleted 1y ago
  [deleted]
geerlingguy 1y ago
I've had multiple people copy and paste AI conversations and results in GitHub issues, emails, etc., and there are I think a growing number of people who blindly trust the results of any of these models... including the 'results summary' posted at the top of Google search results. Almost every summary I have read through contains at least one glaring mistake, but if it's something I know nothing about, I could see how easy it would be to just trust it, since 95% of it seems true/accurate. Trust, but verify is all the more relevant today. Except I would discount the trust, even.
- add-sub-mul-div 1y ago
  We all think ourselves as understanding the tradeoffs of this tech and that we know how to use it responsibly. And we here may be right. But the typical person wants to do the least amount of effort and thinking possible. Our society will evolve to reflect this, it won't be great, and it will affect all of us no matter how personally responsible some of us remain.
  - iotku 1y ago
    I consider myself pretty technically literate, and not the worst at programming (though certainly far from the very best). Even so I can spend plenty of time arguing with LLMs which will give me plausible looking but extremely broken answers to some of the programming problems. In the programming domain I can at least run something and see it doesn't compile or work as I expect, but you can't verify that a written statement about someone/something is the correct interpretation without knowing the correct answer ahead of time. To muddy the waters further, things work just well enough on common knowledge that it's easy to believe it could be right about uncommon knowledge which you don't know how to verify. (Or else you wouldn't be asking it in the first place)
    - add-sub-mul-div 1y ago
      Even with code, "seeing" a block of code working isn't a guarantee there's not a subtle bug that will expose itself in a week, in a month, in a year under the right conditions.
      - giantrobot 1y ago
        I've pointed this out a lot and I often get replies along the lines of "people make mistakes too". While this is true, LLMs lack institutional memory leading to decisions. Even good reasoning models can't reliably tell you why they wrote some code they did when asked to review it. They can't even reliably run tests since they'll hardcode passing values for tests. The same code out of an intern or junior programmer you can at least walk through their reasoning on a code review. Even better if they tend to learn and not make that same mistake again. LLMs will happily screw up randomly on every repeated prompt. The hardest code you encounter is code written by someone else. You don't have the same mental model or memories as the original author. So you need to build all that context and then reason through the code. If an LLM is writing a lot of your code you're missing out on all the context you'd normally build writing it.
- Aurornis 1y ago
  > I've had multiple people copy and paste AI conversations and results in GitHub issues, emails, etc., A growing number of Discords, open source projects, and other spaces where I participate now have explicit rules against copying and pasting ChatGPT content. When there aren’t rules, many people are quick to discourage LLM copy and paste. “Please don’t do this”. The LLM copy and paste wall of text that may or may not be accurate is extremely frustrating to everyone else. Some people think they’re being helpful by doing it, but it’s quickly becoming a social faux pas.
  - lawlessone 1y ago
    see it with comments here sometimes , "i asked chatgpt about Y" , really annoying, we all could have asked chatgpt, we didn't.
    - ljm 1y ago
      Have had some conversations where the other person goes into chatgpt to answer a question while I’m in the process of explaining a solution, and then says “GPT says this, look…” Use an agent to help you code or whatever all you want, I don’t care about that. At least listen when I’m trying to share some specific knowledge instead of fobbing me off with GPT. If we’re both stumped, go nuts. But at least put some effort into the prompt to get a better response.
    - buu700 1y ago
      I don't have an issue quoting LLMs in and of itself, but the context and how you present it both matter. "ChatGPT says X" seems roughly equivalent to "some random blog I found claims X". There's a difference between sharing something as a starting point for investigation and passing off unverified information (from any source) as your own well researched/substantiated work which you're willing to stake your professional reputation on standing by. Of course, quoting an LLM is also pretty different from merely collaborating with an LLM on writing content that's substantially your own words or ideas, which no one should care about one way or another, at least in most contexts.
  - tavavex 1y ago
- freeopinion 1y ago
  prompt> use javascript to convert a unix timestamp to a date in 'YYYY-MM-DD' format using Temporal answer> Temporal.Instant.fromEpochSeconds(timestamp).toPlainDate() Trust but verify?
  - nielsbot 1y ago
    what does this mean in this convo?
    - freeopinion 1y ago
      If you were considering purchasing a Biology text book, and spot read two chapters, what if you found the following?: In the first chapter it claimed that most adult humans have 20 teeth. In the second chapter you read that female humans have 22 chromosomes and male humans have 23. You find these claims in the 24 pages you sample. Do you buy the book? Companies are paying huge sums to AI companies with worse track records. Would you put the book in your reference library if somebody gave it to you for free? Services like Google or DuckDuckGo put their AI-generated content at the top of search results with these inaccuracies. [edit: replace paragraph that somehow got deleted, fix typo]
    - freeopinion 1y ago
      Google distinguished itself early with techniques like PageRank that put more relevant content at the top of their search results. Is it too late for a rival to distinguish itself with techniques like "Don't put garbage AI at the top of search results"?
  - eszed 1y ago
    I mean... Yes? That looks correct to me°, but it's been a minute since I worked with Temporal, so I'd run it myself and examine the output before I cut and paste. Or have I missed your point? --- °Missing a TZ assertion, but I don't remember what happens by default. Zulu time? I'd hope so, but that reinforces my point.
    - nosianu 1y ago
- leeoniya 1y ago
  > but if it's something I know nothing about, I could see how easy it would be to just trust it, since 95% of it seems true/accurate. https://en.m.wikipedia.org/wiki/Gell-Mann_amnesia_effect https://en.m.wikipedia.org/wiki/Gell-Mann_amnesia_effect
- userbinator 1y ago
  "Trust, but verify" is an oxymoron. AI is not to be trusted for information.
  - nielsbot 1y ago
    AI = convincing garbage generator
- abustamam 1y ago
  Whenever I use AI in social settings to fact check or do research or get advice, I always trust but verify, but also disclaim it so that people know to trust but verify. I think this is a good habit to get people into, even in casual conversations. Even if someone didn't directly get their info from AI and got it online, the content could have still been generated by AI. Like you said, the trust part of trust but verify is quickly dwindling.
- account42 1y ago
  > I've had multiple people copy and paste AI conversations and results in GitHub issues, emails, etc., and there are I think a growing number of people who blindly trust the results of any of these models... including the 'results summary' posted at the top of Google search results. I like the term "echoborg" for these people. I hope it catches on.
- fennecbutt 1y ago
  It's just mass stupidity really. Technology is just a lever for what already existed. The same people blindly trusting AI nonsense are the same people who trusted nonsense from social media or talking heads on unreputable news channels. Like, who actually reads the output of The Sun, etc? Those people do, have always done and will continue to do so. And they vote, yaaay democracy - if your voter base lives in a fantasy world of fake news and false science is democracy still sancrosact?
outside415 1y ago
[dead]
reaperducer 1y ago
I think this is a situation that is a perfect example of how AI hallucinations/lack of accuracy could significantly impact our lives going forward. This has been a Google problem for decades. I used to run a real estate forum. Someone once wrote a message along the lines of "Joe is a really great real estate agent, but Frank is a total scumbag. Stole all my money." When people would Google Joe, my forum was the first result. And the snippet Google made from the content was "Joe... is a total scumbag. Stole all my money." I found out about it when Joe lawyered up. That was a fun six months.
- dieortin 1y ago
  Sorry but I don’t see how what you mention is a problem. A search engine is just surfacing the content you have in your site. That is very different from it making stuff up.
  - reaperducer 1y ago
    No apology necessary. Read it again. More carefully this time.
ants_everywhere 1y ago
Has anyone independently confirmed the accuracy of his claim?
haswell 1y ago
One of the arguments used to justify the mass-ingestion of copyrighted content to build these models is that the resulting model is a transformative work, and thus fair use. If this is indeed true, it seems like Google et al must be liable for output like this according to their own argument, i.e. if the work is transformative, they can’t claim someone else is liable. These companies can’t have their cake and eat it too. It’ll be interesting to see how this plays out.
- account42 1y ago
  I mean that's what they have been getting already - the average joe had to deal with draconian copyright all this time but not that it's inconvenient for big tech they get to hand-wave it away. The social contract has already been broken. And companies have always been able to get away with relatively minor fines for things that get individuals locked up until they rot.
- pjc50 1y ago
  > These companies can’t have their cake and eat it too I think you're underestimating the effect of billions of dollars on the legal system, and the likely impact of the Have Your Cake And Eat It Act 2026.
  - fennecbutt 1y ago
    Yup, corporations have owned us and the government for a long time. Idk why people still act surprised about this.
fsckboy 1y ago
>a perfect example of how AI hallucinations/lack of accuracy could significantly impact our lives going forward. how about stop forming judgments of people based on their stance on Israel/Hamas, and stop hanging around people who do, and you'll be fine. if somebody misstates your opinion, it won't matter. probably you'll have to drop bluesky and parts of HN (like this political discussion that you urge be left up) but that's necessary because all legitimate opinions about Israel/Hamas are very misinformed/cherry picked, and AI is just flipping a coin which is just as good as an illegitimate opinion. (if anybody would like to convince me that they are well informed on these topics, i'm all ears, but doing it here is imho a bad idea so it's on you if you try)
- slg 1y ago
  >because all legitimate opinions about Israel/Hamas are very misinformed/cherry picked Sure, there is plenty of misinformation being thrown in multiple different directions, but if you think literally "all legitimate opinions" are "misinformed/cherry picked", then odds are you are just looking at the issue through your own misinformed frame of reference.
  - fsckboy 1y ago
    >but if you think literally "all legitimate opinions" are "misinformed/cherry picked", then odds are yes, i literally do think that, so there are no odds. i think i am well informed on the related subjects to the extent that whatever point someone might want to make i'll probably have a counterpoint
    - deleted 1y ago
      [deleted]
    - dotancohen 1y ago
      Whenever someone asks my opinion on something related to the conflict in the holy land, I ask them if they want the anti-Jewish answer, the anti-Muslim answer, the pro-Jewish answer, of the pro-Muslim answer. And it took me decades of studying this to determine what to call the two sides.
Gigachad 1y ago
Google should be held liable for this. They are the ones who published and hosted this. And should be accountable for every bit of libel they publish.
- pjc50 1y ago
  American attitudes to free speech mean that only the most dramatic, damaging libels can be held accountable, years after the effect. I think the only one I can think of that got justice was Alex Jones libelling Sandy Hook victims. (no easy answers: UK libel law errs in the other direction)

by AnEro 1y ago

jaccola 1y ago

This story will probably become big enough to drown out the fake video and the AI (which is presumably being fed top n search results) will automatically describe this fake video controversy instead...

deleted 1y ago
[deleted]

geerlingguy 1y ago

I've had multiple people copy and paste AI conversations and results in GitHub issues, emails, etc., and there are I think a growing number of people who blindly trust the results of any of these models... including the 'results summary' posted at the top of Google search results. Almost every summary I have read through contains at least one glaring mistake, but if it's something I know nothing about, I could see how easy it would be to just trust it, since 95% of it seems true/accurate. Trust, but verify is all the more relevant today. Except I would discount the trust, even.

add-sub-mul-div 1y ago
We all think ourselves as understanding the tradeoffs of this tech and that we know how to use it responsibly. And we here may be right. But the typical person wants to do the least amount of effort and thinking possible. Our society will evolve to reflect this, it won't be great, and it will affect all of us no matter how personally responsible some of us remain.
- iotku 1y ago
  I consider myself pretty technically literate, and not the worst at programming (though certainly far from the very best). Even so I can spend plenty of time arguing with LLMs which will give me plausible looking but extremely broken answers to some of the programming problems. In the programming domain I can at least run something and see it doesn't compile or work as I expect, but you can't verify that a written statement about someone/something is the correct interpretation without knowing the correct answer ahead of time. To muddy the waters further, things work just well enough on common knowledge that it's easy to believe it could be right about uncommon knowledge which you don't know how to verify. (Or else you wouldn't be asking it in the first place)
  - add-sub-mul-div 1y ago
    Even with code, "seeing" a block of code working isn't a guarantee there's not a subtle bug that will expose itself in a week, in a month, in a year under the right conditions.
    - giantrobot 1y ago
      I've pointed this out a lot and I often get replies along the lines of "people make mistakes too". While this is true, LLMs lack institutional memory leading to decisions. Even good reasoning models can't reliably tell you why they wrote some code they did when asked to review it. They can't even reliably run tests since they'll hardcode passing values for tests. The same code out of an intern or junior programmer you can at least walk through their reasoning on a code review. Even better if they tend to learn and not make that same mistake again. LLMs will happily screw up randomly on every repeated prompt. The hardest code you encounter is code written by someone else. You don't have the same mental model or memories as the original author. So you need to build all that context and then reason through the code. If an LLM is writing a lot of your code you're missing out on all the context you'd normally build writing it.
Aurornis 1y ago
> I've had multiple people copy and paste AI conversations and results in GitHub issues, emails, etc., A growing number of Discords, open source projects, and other spaces where I participate now have explicit rules against copying and pasting ChatGPT content. When there aren’t rules, many people are quick to discourage LLM copy and paste. “Please don’t do this”. The LLM copy and paste wall of text that may or may not be accurate is extremely frustrating to everyone else. Some people think they’re being helpful by doing it, but it’s quickly becoming a social faux pas.
- lawlessone 1y ago
  see it with comments here sometimes , "i asked chatgpt about Y" , really annoying, we all could have asked chatgpt, we didn't.
  - ljm 1y ago
    Have had some conversations where the other person goes into chatgpt to answer a question while I’m in the process of explaining a solution, and then says “GPT says this, look…” Use an agent to help you code or whatever all you want, I don’t care about that. At least listen when I’m trying to share some specific knowledge instead of fobbing me off with GPT. If we’re both stumped, go nuts. But at least put some effort into the prompt to get a better response.
  - buu700 1y ago
    I don't have an issue quoting LLMs in and of itself, but the context and how you present it both matter. "ChatGPT says X" seems roughly equivalent to "some random blog I found claims X". There's a difference between sharing something as a starting point for investigation and passing off unverified information (from any source) as your own well researched/substantiated work which you're willing to stake your professional reputation on standing by. Of course, quoting an LLM is also pretty different from merely collaborating with an LLM on writing content that's substantially your own words or ideas, which no one should care about one way or another, at least in most contexts.
- tavavex 1y ago
freeopinion 1y ago
prompt> use javascript to convert a unix timestamp to a date in 'YYYY-MM-DD' format using Temporal answer> Temporal.Instant.fromEpochSeconds(timestamp).toPlainDate() Trust but verify?
- nielsbot 1y ago
  what does this mean in this convo?
  - freeopinion 1y ago
    If you were considering purchasing a Biology text book, and spot read two chapters, what if you found the following?: In the first chapter it claimed that most adult humans have 20 teeth. In the second chapter you read that female humans have 22 chromosomes and male humans have 23. You find these claims in the 24 pages you sample. Do you buy the book? Companies are paying huge sums to AI companies with worse track records. Would you put the book in your reference library if somebody gave it to you for free? Services like Google or DuckDuckGo put their AI-generated content at the top of search results with these inaccuracies. [edit: replace paragraph that somehow got deleted, fix typo]
  - freeopinion 1y ago
    Google distinguished itself early with techniques like PageRank that put more relevant content at the top of their search results. Is it too late for a rival to distinguish itself with techniques like "Don't put garbage AI at the top of search results"?
- eszed 1y ago
  I mean... Yes? That looks correct to me°, but it's been a minute since I worked with Temporal, so I'd run it myself and examine the output before I cut and paste. Or have I missed your point? --- °Missing a TZ assertion, but I don't remember what happens by default. Zulu time? I'd hope so, but that reinforces my point.
  - nosianu 1y ago
leeoniya 1y ago
> but if it's something I know nothing about, I could see how easy it would be to just trust it, since 95% of it seems true/accurate. https://en.m.wikipedia.org/wiki/Gell-Mann_amnesia_effect https://en.m.wikipedia.org/wiki/Gell-Mann_amnesia_effect
userbinator 1y ago
"Trust, but verify" is an oxymoron. AI is not to be trusted for information.
- nielsbot 1y ago
  AI = convincing garbage generator
abustamam 1y ago
Whenever I use AI in social settings to fact check or do research or get advice, I always trust but verify, but also disclaim it so that people know to trust but verify. I think this is a good habit to get people into, even in casual conversations. Even if someone didn't directly get their info from AI and got it online, the content could have still been generated by AI. Like you said, the trust part of trust but verify is quickly dwindling.
account42 1y ago
> I've had multiple people copy and paste AI conversations and results in GitHub issues, emails, etc., and there are I think a growing number of people who blindly trust the results of any of these models... including the 'results summary' posted at the top of Google search results. I like the term "echoborg" for these people. I hope it catches on.
fennecbutt 1y ago
It's just mass stupidity really. Technology is just a lever for what already existed. The same people blindly trusting AI nonsense are the same people who trusted nonsense from social media or talking heads on unreputable news channels. Like, who actually reads the output of The Sun, etc? Those people do, have always done and will continue to do so. And they vote, yaaay democracy - if your voter base lives in a fantasy world of fake news and false science is democracy still sancrosact?

outside415 1y ago

[dead]

reaperducer 1y ago

I think this is a situation that is a perfect example of how AI hallucinations/lack of accuracy could significantly impact our lives going forward. This has been a Google problem for decades. I used to run a real estate forum. Someone once wrote a message along the lines of "Joe is a really great real estate agent, but Frank is a total scumbag. Stole all my money." When people would Google Joe, my forum was the first result. And the snippet Google made from the content was "Joe... is a total scumbag. Stole all my money." I found out about it when Joe lawyered up. That was a fun six months.

dieortin 1y ago
Sorry but I don’t see how what you mention is a problem. A search engine is just surfacing the content you have in your site. That is very different from it making stuff up.
- reaperducer 1y ago
  No apology necessary. Read it again. More carefully this time.

ants_everywhere 1y ago

Has anyone independently confirmed the accuracy of his claim?

haswell 1y ago

One of the arguments used to justify the mass-ingestion of copyrighted content to build these models is that the resulting model is a transformative work, and thus fair use. If this is indeed true, it seems like Google et al must be liable for output like this according to their own argument, i.e. if the work is transformative, they can’t claim someone else is liable. These companies can’t have their cake and eat it too. It’ll be interesting to see how this plays out.

account42 1y ago
I mean that's what they have been getting already - the average joe had to deal with draconian copyright all this time but not that it's inconvenient for big tech they get to hand-wave it away. The social contract has already been broken. And companies have always been able to get away with relatively minor fines for things that get individuals locked up until they rot.
pjc50 1y ago
> These companies can’t have their cake and eat it too I think you're underestimating the effect of billions of dollars on the legal system, and the likely impact of the Have Your Cake And Eat It Act 2026.
- fennecbutt 1y ago
  Yup, corporations have owned us and the government for a long time. Idk why people still act surprised about this.

fsckboy 1y ago

>a perfect example of how AI hallucinations/lack of accuracy could significantly impact our lives going forward. how about stop forming judgments of people based on their stance on Israel/Hamas, and stop hanging around people who do, and you'll be fine. if somebody misstates your opinion, it won't matter. probably you'll have to drop bluesky and parts of HN (like this political discussion that you urge be left up) but that's necessary because all legitimate opinions about Israel/Hamas are very misinformed/cherry picked, and AI is just flipping a coin which is just as good as an illegitimate opinion. (if anybody would like to convince me that they are well informed on these topics, i'm all ears, but doing it here is imho a bad idea so it's on you if you try)

slg 1y ago
>because all legitimate opinions about Israel/Hamas are very misinformed/cherry picked Sure, there is plenty of misinformation being thrown in multiple different directions, but if you think literally "all legitimate opinions" are "misinformed/cherry picked", then odds are you are just looking at the issue through your own misinformed frame of reference.
- fsckboy 1y ago
  >but if you think literally "all legitimate opinions" are "misinformed/cherry picked", then odds are yes, i literally do think that, so there are no odds. i think i am well informed on the related subjects to the extent that whatever point someone might want to make i'll probably have a counterpoint
  - deleted 1y ago
    [deleted]
  - dotancohen 1y ago
    Whenever someone asks my opinion on something related to the conflict in the holy land, I ask them if they want the anti-Jewish answer, the anti-Muslim answer, the pro-Jewish answer, of the pro-Muslim answer. And it took me decades of studying this to determine what to call the two sides.

Gigachad 1y ago

Google should be held liable for this. They are the ones who published and hosted this. And should be accountable for every bit of libel they publish.

pjc50 1y ago
American attitudes to free speech mean that only the most dramatic, damaging libels can be held accountable, years after the effect. I think the only one I can think of that got justice was Alex Jones libelling Sandy Hook victims. (no easy answers: UK libel law errs in the other direction)