5 ms·

I wonder how long it will be until we hear about a company accidentally doing something because the Slack AI summary / AI search incorrectly paraphrased a reque

by pfych 2y ago

I wonder how long it will be until we hear about a company accidentally doing something because the Slack AI summary / AI search incorrectly paraphrased a request. I really wish AI companies would call hallucinations what they really are - Mistakes. Using wishy-washy words like that make non-techy people trust outputs of LLMs much much more than they should. "Oh it just hallucinated - how silly!" instead of "It got it wrong - we should be more careful". Can be scary hearing how much trust the average joe puts into LLM outputs...

deleted 2y ago
[deleted]
39896880 2y ago
I actually think "hallucination" sounds worse than "mistake." A hallucination is a fundamental break from the shared reality.
- madeofpalk 2y ago
  The problem is that everything from an LLM is a hallucination, because LLMs don't have a reality. LLM's inventing new text is the whole point of them. It's just that sometimes they invent text that's also wrong.
  - MostlyStable 2y ago
    This is simply false. LLMs have a world model. Is it perfect? Obviously not. But neither is yours. Saying that "everything" from an LLM is a hallucination, to whatever extent it can be said to be true is overly broad to the point of being useless. It is very possible to learn which kinds of things LLMs are likely to be correct at and which kinds of things they are likely to be wrong about. Neither of those categories is going to be perfect, but, again, neither will it be for any particular human. I mostly agree that current LLMs should not be used in any kind of a critical role that does not have a human constantly checking it. And even with a human, probably not, because humans are lazy, and are likely to stop checking. But they can be very useful if one takes the time to learn what kinds of outputs they usually give. The problem with the vast majority of articles on them is that they usually go in one of two directions: Look at this amazing task that an LLM did really well or Look at this incredibly basic task that an LLM did very poorly. Both of those ignore the totality that LLMs are shockingly, amazingly, good at certain kinds of tasks and also shockingly, embarrassingly bad at other kinds of tasks. If one can learn to recognize which is which, one can get a lot of value and, if one is careful, avoid the pitfalls.
- neilv 2y ago
  If an employee was hallucinating, you'd call them an ambulance.
SystemOut 2y ago
I think it's only going to get worse as the LLMs get just that much better that people stop thinking more critically about what they generate. Right now I think many people still verify or read through the results to make sure they sound right. What happens when people get lulled into complacency and then it really goes off on a bender.
- iknowSFR 2y ago
  Nothing will have changed. How many times have you seen HN comments from people that never read the article? How many times have you read flat out lies on Reddit? The last 40 years of of cable news have banked on this theory. At least the last 80 years of American politics as well. There’s no degradation of critical thinking, either. It’s just a cohort that does it and a cohort that doesn’t. For all of time.
  - dzhiurgis 2y ago
    IMO future is government neutering of anything related to safety, physical world or food...
LeoPanthera 2y ago
> I really wish AI companies would call hallucinations what they really are - Mistakes. At the bottom of every ChatGPT conversation: "ChatGPT can make mistakes. Check important info."
- sadops 2y ago
  [flagged]
  - danielmarkbruce 2y ago
    "I don't work, please hire me anyway"
- sadops 2y ago
  "Our product doesn't work, but please use it anyway."
duskwuff 2y ago
> because the Slack AI summary / AI search incorrectly paraphrased a request. Or because it misinterpreted a question (or a joke!) as a factual statement.
sadops 2y ago
I really wish we'd stop anthropomorphizing these things. They're not hallucinating, the program simply returned an error. Remember, y'all, this stuff is just Markov chains with extra steps. An incorrect answer from any other computer program is called an "error", and apparently LLMs are the only piece of software in which significant amounts of them are tolerated, because they look just like a mechanical Turk, and are sufficiently convincing as to be a really neat parlor trick. But write a program that generates random integers and returns -1 for even numbers, and it's the exact same thing. LLMs are crap, because programs aren't supposed to return errors. "On two occasions I have been asked, — 'Pray, Mr. Babbage, if you put into the machine wrong figures, will the right answers come out?'... I am not able rightly to apprehend the kind of confusion of ideas that could provoke such a question.
- danielmarkbruce 2y ago
  They aren't markov chains. Even GPT-4 knows this.
  - sadops 2y ago
    I'm critiquing their functionality. I know they aren't literally Markov chains... Ask ChatGPT about humor as a tool of criticism.
ldjkfkdsjnv 2y ago
What is it with this constant boogeyman of hallucinations, Its like people cannot understand that this is a new technology that is rapidly improving
- Kiro 2y ago
  GPT3.5 used to hallucinate a lot but it was drastically improved with 4, to a point where I can't even remember the last time it did it. It feels like people are exaggerating or basing it on the experience with old models.
- viraptor 2y ago
  Because majority of people don't understand how LLMs work, what hallucinations are, or the capabilities of AI tech. Like lawyers https://www.abc.net.au/news/2023-06-24/us-lawyer-uses-chatgpt-to-research-case-with-embarrassing-result/102490068 https://www.abc.net.au/news/2023-06-24/us-lawyer-uses-chatgp... airline website devs https://bc.ctvnews.ca/air-canada-s-chatbot-gave-a-b-c-man-the-wrong-information-now-the-airline-has-to-pay-for-the-mistake-1.6769454 https://bc.ctvnews.ca/air-canada-s-chatbot-gave-a-b-c-man-th... law enforcement https://gizmodo.com/deepfake-cheer-mom-sues-for-defamation-1850884295 https://gizmodo.com/deepfake-cheer-mom-sues-for-defamation-1... No, people in general cannot understand changes in technology.
add-sub-mul-div 2y ago
A mistake suggests competence and overall reliability. Hallucination suggests something is more seriously and systemically wrong. I think this way is appropriate.
hi_hi 2y ago
Whatever name is used will inevitably become normalised. Changing the name isn't going to move the needle on people accepting the magic AI fairy dust. I wish I knew what would. I'm sick of seeing it thrown into everything due to corporate FOMO. This stuff isn't cheap to run, and seeing it used to automatically summarise peoples pointless meetings kills me, almost as much as the meetings.

by pfych 2y ago

deleted 2y ago

[deleted]

39896880 2y ago

I actually think "hallucination" sounds worse than "mistake." A hallucination is a fundamental break from the shared reality.

madeofpalk 2y ago
The problem is that everything from an LLM is a hallucination, because LLMs don't have a reality. LLM's inventing new text is the whole point of them. It's just that sometimes they invent text that's also wrong.
- MostlyStable 2y ago
  This is simply false. LLMs have a world model. Is it perfect? Obviously not. But neither is yours. Saying that "everything" from an LLM is a hallucination, to whatever extent it can be said to be true is overly broad to the point of being useless. It is very possible to learn which kinds of things LLMs are likely to be correct at and which kinds of things they are likely to be wrong about. Neither of those categories is going to be perfect, but, again, neither will it be for any particular human. I mostly agree that current LLMs should not be used in any kind of a critical role that does not have a human constantly checking it. And even with a human, probably not, because humans are lazy, and are likely to stop checking. But they can be very useful if one takes the time to learn what kinds of outputs they usually give. The problem with the vast majority of articles on them is that they usually go in one of two directions: Look at this amazing task that an LLM did really well or Look at this incredibly basic task that an LLM did very poorly. Both of those ignore the totality that LLMs are shockingly, amazingly, good at certain kinds of tasks and also shockingly, embarrassingly bad at other kinds of tasks. If one can learn to recognize which is which, one can get a lot of value and, if one is careful, avoid the pitfalls.
neilv 2y ago
If an employee was hallucinating, you'd call them an ambulance.

SystemOut 2y ago

I think it's only going to get worse as the LLMs get just that much better that people stop thinking more critically about what they generate. Right now I think many people still verify or read through the results to make sure they sound right. What happens when people get lulled into complacency and then it really goes off on a bender.

iknowSFR 2y ago
Nothing will have changed. How many times have you seen HN comments from people that never read the article? How many times have you read flat out lies on Reddit? The last 40 years of of cable news have banked on this theory. At least the last 80 years of American politics as well. There’s no degradation of critical thinking, either. It’s just a cohort that does it and a cohort that doesn’t. For all of time.
- dzhiurgis 2y ago
  IMO future is government neutering of anything related to safety, physical world or food...

LeoPanthera 2y ago

> I really wish AI companies would call hallucinations what they really are - Mistakes. At the bottom of every ChatGPT conversation: "ChatGPT can make mistakes. Check important info."

sadops 2y ago
[flagged]
- danielmarkbruce 2y ago
  "I don't work, please hire me anyway"
sadops 2y ago
"Our product doesn't work, but please use it anyway."

duskwuff 2y ago

> because the Slack AI summary / AI search incorrectly paraphrased a request. Or because it misinterpreted a question (or a joke!) as a factual statement.

sadops 2y ago

I really wish we'd stop anthropomorphizing these things. They're not hallucinating, the program simply returned an error. Remember, y'all, this stuff is just Markov chains with extra steps. An incorrect answer from any other computer program is called an "error", and apparently LLMs are the only piece of software in which significant amounts of them are tolerated, because they look just like a mechanical Turk, and are sufficiently convincing as to be a really neat parlor trick. But write a program that generates random integers and returns -1 for even numbers, and it's the exact same thing. LLMs are crap, because programs aren't supposed to return errors. "On two occasions I have been asked, — 'Pray, Mr. Babbage, if you put into the machine wrong figures, will the right answers come out?'... I am not able rightly to apprehend the kind of confusion of ideas that could provoke such a question.

danielmarkbruce 2y ago
They aren't markov chains. Even GPT-4 knows this.
- sadops 2y ago
  I'm critiquing their functionality. I know they aren't literally Markov chains... Ask ChatGPT about humor as a tool of criticism.

ldjkfkdsjnv 2y ago

What is it with this constant boogeyman of hallucinations, Its like people cannot understand that this is a new technology that is rapidly improving

Kiro 2y ago
GPT3.5 used to hallucinate a lot but it was drastically improved with 4, to a point where I can't even remember the last time it did it. It feels like people are exaggerating or basing it on the experience with old models.
viraptor 2y ago
Because majority of people don't understand how LLMs work, what hallucinations are, or the capabilities of AI tech. Like lawyers https://www.abc.net.au/news/2023-06-24/us-lawyer-uses-chatgpt-to-research-case-with-embarrassing-result/102490068 https://www.abc.net.au/news/2023-06-24/us-lawyer-uses-chatgp... airline website devs https://bc.ctvnews.ca/air-canada-s-chatbot-gave-a-b-c-man-the-wrong-information-now-the-airline-has-to-pay-for-the-mistake-1.6769454 https://bc.ctvnews.ca/air-canada-s-chatbot-gave-a-b-c-man-th... law enforcement https://gizmodo.com/deepfake-cheer-mom-sues-for-defamation-1850884295 https://gizmodo.com/deepfake-cheer-mom-sues-for-defamation-1... No, people in general cannot understand changes in technology.

add-sub-mul-div 2y ago

A mistake suggests competence and overall reliability. Hallucination suggests something is more seriously and systemically wrong. I think this way is appropriate.

hi_hi 2y ago

Whatever name is used will inevitably become normalised. Changing the name isn't going to move the needle on people accepting the magic AI fairy dust. I wish I knew what would. I'm sick of seeing it thrown into everything due to corporate FOMO. This stuff isn't cheap to run, and seeing it used to automatically summarise peoples pointless meetings kills me, almost as much as the meetings.