6 ms·

Tell HN: Twilio support replies with hallucinated features

159 points by haute_cuisine 1y ago

I was investigating some bug with our voice system and asked support where I can find some debugging information and event logs. They told me where I should go in the interface to see that and reassured that they checked logs and this event exist. It turned out these features and information doesn't exist anywhere in the interface and impossible to retrieve in any way. The support message with hallucinated features is mostly AI written. CEOs tell us AGI is around the corner but in reality it just unreliable information and AI can't even restock the vending machine.

quinnjh 1y ago
These tools are perfect for deployment where providing plausible-but-incorrect info is aligned with business outcomes, like cutting your support staff and giving disgruntled customers fake information. I’ve seen most of the frontier models hallucinate their capabilities, not surprising they might do so for api completions regarding a product they barely know about. Unless they lose more money from cancelled subscriptions than they saved on cutting support staff, it’s probably the new normal.
- trollbridge 1y ago
  Twilio registered my business name as “My Twilio Account” and is unable to change it. My application for 10DLC also got rejected since I wanted to do something other than send marketing messages with it and I can’t figure out how to describe an opt-in only service that is strictly for employees, to their provided phone number, with a signature opting in to get payroll information texted to them. As a test, I set up something to send junk quality marketing texts. Was approved.
  - collingreen 1y ago
    I had this same experience. Twilio is not interested in the small internal tool customer - do mass marketing or go away.
barbazoo 1y ago
None of them know what they're doing. Even Google's own AI integrated into their own apps, hallucinates about those very apps, e.g. asking Gemini in Docs about how to do something in Docs. It's laughable. LLM have great utility but this is not it.
gdulli 1y ago
There used to be a contract that a business had something to lose by providing bad service, that customers would leave and seek better service elsewhere. I believe the most important and least discussed phenomenon of modern consumer culture is that consumers have passed a threshold of passive and docile behavior such that businesses no longer fear losing customers. Partly because the customers have shown willingness to eat shit, partly because there's a new understanding that all businesses will adopt the same customer-hostile behaviors (AI customer service in this case) so consumers don't have significant choice anyway.
- azemetre 1y ago
  This is mostly due to not trust busting enough in society. If there were actual competitive markets, not monopolies/oligarchies/monopsonies/cartels, the business world would be completely different. Either that, or legislate workplace democracy.
  - falcor84 1y ago
    I tend to disagree. While there are definitely monopolies/oligopoly for every domain, I'm actually constantly impressed with the very long tail of other providers available in that area. Whenever I am looking for a new solution to a need at work, I would go to sites like g2.com to look at the lists of the most popular ones, and would then typically skim reviews of the top ~10, and more fully evaluate the top ~3. But there are often hundreds of alternatives that I haven't given a chance to, and I know that it's my <s>laziness</s> need to manage my limited time that's promoting this oligopoly, rather than any particular issue with all of those other providers down the list. I don't see how legislation can help here, other than picking a provider for me. If anything, this is actually a place where I feel that AI tools, and particularly ChatGPT's Deep Research can research a lot more of the alternatives than I as a human would have time for. But that of course has its own set of issues, and I really don't know what the solution is. We no longer live in that world where you just use that provider who lives down the street.
    - azemetre 1y ago
lab14 1y ago
"Hallucination machine, responds with hallucinations". But seriously, entreprise customers (and any big spender account) usually get access to a dedicated (human) account rep and private support channels in Slack, so they never really interact with this.
- mikeandrees9921 1y ago
  Yeah true. Abstraction reigns king lol
gooseWithFood20 1y ago
This happened to me with a chat it for Chevrolet lmao. Started telling me about a car that DID NOT EXIST
elicash 1y ago
What distinguishes AI slop customer support from the previous enshittification of customer service is that previously if you wanted to avoid the garbage chat support you could get on the phone and -- even if you had to go through a phone tree -- you could at least eventually ask a person about the problem. But now, even if it's possible to get a person on the phone, THAT PERSON is just doing the AI chatbot on their end. By talking to a human, you're just adding a middleman who is accessing the same incorrect chatbot that's available to you.
cacozen 1y ago
The vending machine mention is about this paper from Anthropic: https://www.anthropic.com/research/project-vend-1 https://www.anthropic.com/research/project-vend-1 The gist is: Claude AI successfully ran a shop by itself! - Actually a vending machine - Actually a mini-fridge in our office - Actually it gave lots of discounts and free products on our slack - Actually it hallucinated a Venmo account and people sent payments to God-knows-who
- sieep 1y ago
  This is hilarious. The gall these guys have to say things like '...not-too-distant future in which AI models are autonomously running things in the real economy.' It's not even close to doing something a little girl at a lemonade stand could do, no?
jeromegv 1y ago
Air Canada got sued in Canada for having a chatbot that allucinating a policy And they lost https://www.cbc.ca/news/canada/british-columbia/air-canada-chatbot-lawsuit-1.7116416 https://www.cbc.ca/news/canada/british-columbia/air-canada-c...
jqpabc123 1y ago
Current AI is probabilistic, not deterministic. This means it can and most probably will be dead wrong at some point. So before integrating AI into your workflow, you should ask yourself, "Do you feel lucky?".
ilamparithi 1y ago
I searched on Google to check if banks were open on a certain day. The AI response on top said they were closed because it was a second Saturday, but it was actually a Wednesday.
aiiizzz 1y ago
Twilio CEO tells you agi around the corner? Doubt
dedi089 1y ago
[flagged]
sherinjosephroy 1y ago
[flagged]
taf2 1y ago
Fun - I always test the ai support by asking it for a really good sc2 Zerg rush build - as I recall Twilio gave me a pretty good build order

159 points by haute_cuisine 1y ago

quinnjh 1y ago

These tools are perfect for deployment where providing plausible-but-incorrect info is aligned with business outcomes, like cutting your support staff and giving disgruntled customers fake information. I’ve seen most of the frontier models hallucinate their capabilities, not surprising they might do so for api completions regarding a product they barely know about. Unless they lose more money from cancelled subscriptions than they saved on cutting support staff, it’s probably the new normal.

trollbridge 1y ago
Twilio registered my business name as “My Twilio Account” and is unable to change it. My application for 10DLC also got rejected since I wanted to do something other than send marketing messages with it and I can’t figure out how to describe an opt-in only service that is strictly for employees, to their provided phone number, with a signature opting in to get payroll information texted to them. As a test, I set up something to send junk quality marketing texts. Was approved.
- collingreen 1y ago
  I had this same experience. Twilio is not interested in the small internal tool customer - do mass marketing or go away.

barbazoo 1y ago

None of them know what they're doing. Even Google's own AI integrated into their own apps, hallucinates about those very apps, e.g. asking Gemini in Docs about how to do something in Docs. It's laughable. LLM have great utility but this is not it.

gdulli 1y ago

There used to be a contract that a business had something to lose by providing bad service, that customers would leave and seek better service elsewhere. I believe the most important and least discussed phenomenon of modern consumer culture is that consumers have passed a threshold of passive and docile behavior such that businesses no longer fear losing customers. Partly because the customers have shown willingness to eat shit, partly because there's a new understanding that all businesses will adopt the same customer-hostile behaviors (AI customer service in this case) so consumers don't have significant choice anyway.

azemetre 1y ago
This is mostly due to not trust busting enough in society. If there were actual competitive markets, not monopolies/oligarchies/monopsonies/cartels, the business world would be completely different. Either that, or legislate workplace democracy.
- falcor84 1y ago
  I tend to disagree. While there are definitely monopolies/oligopoly for every domain, I'm actually constantly impressed with the very long tail of other providers available in that area. Whenever I am looking for a new solution to a need at work, I would go to sites like g2.com to look at the lists of the most popular ones, and would then typically skim reviews of the top ~10, and more fully evaluate the top ~3. But there are often hundreds of alternatives that I haven't given a chance to, and I know that it's my <s>laziness</s> need to manage my limited time that's promoting this oligopoly, rather than any particular issue with all of those other providers down the list. I don't see how legislation can help here, other than picking a provider for me. If anything, this is actually a place where I feel that AI tools, and particularly ChatGPT's Deep Research can research a lot more of the alternatives than I as a human would have time for. But that of course has its own set of issues, and I really don't know what the solution is. We no longer live in that world where you just use that provider who lives down the street.
  - azemetre 1y ago

lab14 1y ago

"Hallucination machine, responds with hallucinations". But seriously, entreprise customers (and any big spender account) usually get access to a dedicated (human) account rep and private support channels in Slack, so they never really interact with this.

mikeandrees9921 1y ago
Yeah true. Abstraction reigns king lol

gooseWithFood20 1y ago

This happened to me with a chat it for Chevrolet lmao. Started telling me about a car that DID NOT EXIST

elicash 1y ago

What distinguishes AI slop customer support from the previous enshittification of customer service is that previously if you wanted to avoid the garbage chat support you could get on the phone and -- even if you had to go through a phone tree -- you could at least eventually ask a person about the problem. But now, even if it's possible to get a person on the phone, THAT PERSON is just doing the AI chatbot on their end. By talking to a human, you're just adding a middleman who is accessing the same incorrect chatbot that's available to you.

cacozen 1y ago

The vending machine mention is about this paper from Anthropic: https://www.anthropic.com/research/project-vend-1 https://www.anthropic.com/research/project-vend-1 The gist is: Claude AI successfully ran a shop by itself! - Actually a vending machine - Actually a mini-fridge in our office - Actually it gave lots of discounts and free products on our slack - Actually it hallucinated a Venmo account and people sent payments to God-knows-who

sieep 1y ago
This is hilarious. The gall these guys have to say things like '...not-too-distant future in which AI models are autonomously running things in the real economy.' It's not even close to doing something a little girl at a lemonade stand could do, no?

jeromegv 1y ago

Air Canada got sued in Canada for having a chatbot that allucinating a policy And they lost https://www.cbc.ca/news/canada/british-columbia/air-canada-chatbot-lawsuit-1.7116416 https://www.cbc.ca/news/canada/british-columbia/air-canada-c...

jqpabc123 1y ago

Current AI is probabilistic, not deterministic. This means it can and most probably will be dead wrong at some point. So before integrating AI into your workflow, you should ask yourself, "Do you feel lucky?".

ilamparithi 1y ago

I searched on Google to check if banks were open on a certain day. The AI response on top said they were closed because it was a second Saturday, but it was actually a Wednesday.

aiiizzz 1y ago

Twilio CEO tells you agi around the corner? Doubt

dedi089 1y ago

[flagged]

sherinjosephroy 1y ago

[flagged]

taf2 1y ago

Fun - I always test the ai support by asking it for a really good sc2 Zerg rush build - as I recall Twilio gave me a pretty good build order