4 ms·

The air Canada chatbot that mistakenly told someone they can cancel and be refunded for a flight due to a bereavement is a good example of this. It went to cour

The air Canada chatbot that mistakenly told someone they can cancel and be refunded for a flight due to a bereavement is a good example of this. It went to court and they had to honour the chatbot’s response. It’s quite funny that a chatbot has more humanity than its corporate human masters.

shinycode 11mo ago
What a nice side effect, unfortunately they’ll lock chatbots with more barriers in the future but that’s ironic.
- danaris 11mo ago
  ...And under pressure, those barriers will fail, too. It is not possible, at least with any of the current generations of LLMs, to construct a chatbot that will always follow your corporate policies.
  - Loughla 11mo ago
    That's what people aren't understanding, it seems. You are providing people with an endlessly patient, endlessly novel, endlessly naive employee to attempt your social engineering attacks on. Over and over and over. Hell, it will even provide you with reasons for its inability to answer your question, allowing you to fine-tune your attacks faster and easier than with a person. Until true AI exists, there are no actual hard-stops, just guardrails that you can step over if you try hard enough. We recently cancelled a contract with a company because they implemented student facing AI features that could call data from our student information and learning management systems. I was able to get it to give me answers to a test for a class I wasn't enrolled in and PII for other students, even though the company assured us that, due to their built-in guardrails, it could only provide general information for courses that the students are actively enrolled in (due dates, time limits, those sorts of things). Had we allowed that to go live (as many institutions have), it was just a matter of time before a savvy student figured that out. We killed the connection with that company the week before finals, because the shit-show of fixing broken features was less of a headache than unleashing hell on our campus in the form of a very friendly chatbot.
  - PunchyHamster 11mo ago
    With chat ai + guardrail AI it probably will get to the point of it being sure enough that the amount of mistakes won't hit the bottom line. ...and we will find a way to turn it into malicious compliance where rules are not broken but stuff corporation wanted to happen doesn't.
RobotToaster 11mo ago
Chatbots have no fear of being fired, most humans would do the same in a similar position.
- actionfromafar 11mo ago
  "... adding fear into system prompt"
- roughly 11mo ago
  More to the point, most humans loudly declare they would do the right thing, so all the chatbot’s training data is on people doing the right thing. There’s comparatively fewer loud public pronunciations of personal cowardice, so if the bot’s going to write a realistic completion, it’s more likely to conjure an author acting heroically.
- SoftTalker 11mo ago
  Do they not? If a chatbot isn't doing what its owners want, won't they just shut it down? Or switch to a competitor's chatbot?
kebman 11mo ago
Not AI, but similar sounding incident in Norway. Some traders found a way to exploit another company's trading bot at the Oslo Stock Exchange. The case went to court. And the court's ruling? "Make a better trading bot."
- Marazan 11mo ago
  I am so glad to read this. Last I had read on the case was that the traders were (outrageously) convicted of market manipulation: https://www.cnbc.com/2010/10/14/norwegians-convicted-for-outwitting-trading-robots.html https://www.cnbc.com/2010/10/14/norwegians-convicted-for-out... But you are right, they appealed and had their appeal upheld by the Supreme Courts: https://www.finextra.com/newsarticle/23677/norwegian-court-acquits-day-traders-in-algo-cracking-case https://www.finextra.com/newsarticle/23677/norwegian-court-a... I am so glad at the result.
delichon 11mo ago
That policy would be fraudulently exploited immediately. So is it more humane or more gullible? I suppose it would hallucinate a different policy if it includes in the context window the interests of shareholders, employees and other stakeholders, as well as the customer. But it would likely be a more accurate hallucination.
butlike 11mo ago
Efficiency, not money, seems to be the currency of chatbots

shinycode 11mo ago

What a nice side effect, unfortunately they’ll lock chatbots with more barriers in the future but that’s ironic.

danaris 11mo ago
...And under pressure, those barriers will fail, too. It is not possible, at least with any of the current generations of LLMs, to construct a chatbot that will always follow your corporate policies.
- Loughla 11mo ago
  That's what people aren't understanding, it seems. You are providing people with an endlessly patient, endlessly novel, endlessly naive employee to attempt your social engineering attacks on. Over and over and over. Hell, it will even provide you with reasons for its inability to answer your question, allowing you to fine-tune your attacks faster and easier than with a person. Until true AI exists, there are no actual hard-stops, just guardrails that you can step over if you try hard enough. We recently cancelled a contract with a company because they implemented student facing AI features that could call data from our student information and learning management systems. I was able to get it to give me answers to a test for a class I wasn't enrolled in and PII for other students, even though the company assured us that, due to their built-in guardrails, it could only provide general information for courses that the students are actively enrolled in (due dates, time limits, those sorts of things). Had we allowed that to go live (as many institutions have), it was just a matter of time before a savvy student figured that out. We killed the connection with that company the week before finals, because the shit-show of fixing broken features was less of a headache than unleashing hell on our campus in the form of a very friendly chatbot.
- PunchyHamster 11mo ago
  With chat ai + guardrail AI it probably will get to the point of it being sure enough that the amount of mistakes won't hit the bottom line. ...and we will find a way to turn it into malicious compliance where rules are not broken but stuff corporation wanted to happen doesn't.

RobotToaster 11mo ago

Chatbots have no fear of being fired, most humans would do the same in a similar position.

actionfromafar 11mo ago
"... adding fear into system prompt"
roughly 11mo ago
More to the point, most humans loudly declare they would do the right thing, so all the chatbot’s training data is on people doing the right thing. There’s comparatively fewer loud public pronunciations of personal cowardice, so if the bot’s going to write a realistic completion, it’s more likely to conjure an author acting heroically.
SoftTalker 11mo ago
Do they not? If a chatbot isn't doing what its owners want, won't they just shut it down? Or switch to a competitor's chatbot?

kebman 11mo ago

Not AI, but similar sounding incident in Norway. Some traders found a way to exploit another company's trading bot at the Oslo Stock Exchange. The case went to court. And the court's ruling? "Make a better trading bot."

Marazan 11mo ago
I am so glad to read this. Last I had read on the case was that the traders were (outrageously) convicted of market manipulation: https://www.cnbc.com/2010/10/14/norwegians-convicted-for-outwitting-trading-robots.html https://www.cnbc.com/2010/10/14/norwegians-convicted-for-out... But you are right, they appealed and had their appeal upheld by the Supreme Courts: https://www.finextra.com/newsarticle/23677/norwegian-court-acquits-day-traders-in-algo-cracking-case https://www.finextra.com/newsarticle/23677/norwegian-court-a... I am so glad at the result.

delichon 11mo ago

That policy would be fraudulently exploited immediately. So is it more humane or more gullible? I suppose it would hallucinate a different policy if it includes in the context window the interests of shareholders, employees and other stakeholders, as well as the customer. But it would likely be a more accurate hallucination.

butlike 11mo ago

Efficiency, not money, seems to be the currency of chatbots