5 ms·
Yeah I see things like "AI Firewalls" as both, firstly ridiculously named, but also, the idea you can slap an applicance (thats sometimes its own LLM) onto anot
by protocolture 11mo ago
Yeah I see things like "AI Firewalls" as both, firstly ridiculously named, but also, the idea you can slap an applicance (thats sometimes its own LLM) onto another LLM and pray that this will prevent errors to be lunacy.
For tasks that arent customer facing, LLMs rock. Human in the loop. Perfectly fine. But whenever I see AI interacting with someones customer directly I just get sort of anxious.
Big one I saw was a tool that ingested a humans report on a safety incident, adjusted them with an LLM, and then posted the result to an OHS incident log. 99% of the time its going to be fine, then someones going to die and the the log will have a recipe for spicy noodles in it, and someones going to jail.
- jonplackett 11mo agoThe air Canada chatbot that mistakenly told someone they can cancel and be refunded for a flight due to a bereavement is a good example of this. It went to court and they had to honour the chatbot’s response. It’s quite funny that a chatbot has more humanity than its corporate human masters.
- shinycode 11mo agoWhat a nice side effect, unfortunately they’ll lock chatbots with more barriers in the future but that’s ironic.
- danaris 11mo ago...And under pressure, those barriers will fail, too. It is not possible, at least with any of the current generations of LLMs, to construct a chatbot that will always follow your corporate policies.
- Loughla 11mo agoThat's what people aren't understanding, it seems. You are providing people with an endlessly patient, endlessly novel, endlessly naive employee to attempt your social engineering attacks on. Over and over and over. Hell, it will even provide you with reasons for its inability to answer your question, allowing you to fine-tune your attacks faster and easier than with a person. Until true AI exists, there are no actual hard-stops, just guardrails that you can step over if you try hard enough. We recently cancelled a contract with a company because they implemented student facing AI features that could call data from our student information and learning management systems. I was able to get it to give me answers to a test for a class I wasn't enrolled in and PII for other students, even though the company assured us that, due to their built-in guardrails, it could only provide general information for courses that the students are actively enrolled in (due dates, time limits, those sorts of things). Had we allowed that to go live (as many institutions have), it was just a matter of time before a savvy student figured that out. We killed the connection with that company the week before finals, because the shit-show of fixing broken features was less of a headache than unleashing hell on our campus in the form of a very friendly chatbot.
- PunchyHamster 11mo agoWith chat ai + guardrail AI it probably will get to the point of it being sure enough that the amount of mistakes won't hit the bottom line. ...and we will find a way to turn it into malicious compliance where rules are not broken but stuff corporation wanted to happen doesn't.
- RobotToaster 11mo agoChatbots have no fear of being fired, most humans would do the same in a similar position.
- actionfromafar 11mo ago"... adding fear into system prompt"
- roughly 11mo agoMore to the point, most humans loudly declare they would do the right thing, so all the chatbot’s training data is on people doing the right thing. There’s comparatively fewer loud public pronunciations of personal cowardice, so if the bot’s going to write a realistic completion, it’s more likely to conjure an author acting heroically.
- SoftTalker 11mo agoDo they not? If a chatbot isn't doing what its owners want, won't they just shut it down? Or switch to a competitor's chatbot?
- kebman 11mo agoNot AI, but similar sounding incident in Norway. Some traders found a way to exploit another company's trading bot at the Oslo Stock Exchange. The case went to court. And the court's ruling? "Make a better trading bot."
- Marazan 11mo agoI am so glad to read this. Last I had read on the case was that the traders were (outrageously) convicted of market manipulation: https://www.cnbc.com/2010/10/14/norwegians-convicted-for-outwitting-trading-robots.html https://www.cnbc.com/2010/10/14/norwegians-convicted-for-out... But you are right, they appealed and had their appeal upheld by the Supreme Courts: https://www.finextra.com/newsarticle/23677/norwegian-court-acquits-day-traders-in-algo-cracking-case https://www.finextra.com/newsarticle/23677/norwegian-court-a... I am so glad at the result.
- delichon 11mo agoThat policy would be fraudulently exploited immediately. So is it more humane or more gullible? I suppose it would hallucinate a different policy if it includes in the context window the interests of shareholders, employees and other stakeholders, as well as the customer. But it would likely be a more accurate hallucination.
- butlike 11mo agoEfficiency, not money, seems to be the currency of chatbots
- ben_w 11mo ago> 99% of the time its going to be fine, then someones going to die and the the log will have a recipe for spicy noodles in it, and someones going to jail. I agree, and also I am now remembering Terry Pratchett's (much lower stakes) reason for getting angry with his German publisher: https://gmkeros.wordpress.com/2011/09/02/terry-pratchett-and-the-maggi-soup-adverts/ https://gmkeros.wordpress.com/2011/09/02/terry-pratchett-and... Which is also the kind of product placement that comes up at least once in every thread about how LLMs might do advertising.
- antonvs 11mo ago> … LLMs might do advertising. It’s no longer “might”. There was very recently a leak that OpenAI is actively working on this.
- ben_w 11mo agoIt's "how LLMs might do" it right up until we see what they actually do. There's lots of other ways they might do it besides this way.
- PunchyHamster 11mo ago"I see you're annoyed with that problem, did you ate recently ? There is that restaurant that gets great reviews near you, and they have a promotion!"
- mikkupikku 11mo ago> the idea you can slap an applicance (thats sometimes its own LLM) onto another LLM and pray that this will prevent errors to be lunacy It usually works though. There are no guarantees of course, but sanity checking an LLMs output with another instance of itself usually does work because LLMs usually aren't reliably wrong in the same way. For instance if you ask it something it doesn't know and it hallucinates a plausible answer, another instance of the same LLM is unlikely to hallucinate the same exact answer, it'll probably give you another answer, which is your heads up that probably both are wrong.
- phatskat 11mo agoSure, and then you can throw another LLM in and make them come to a consensus, of course that could be wrong too so have another three do the same and then compare, and then…
- bsenftner 11mo agoI have an ongoing and endless debate with a PhD that insists consensus of multiple LLMs is a valid proof check. The guy is a neuroscientist, not at all a developer tech head, and is just stubborn, continually projecting a sentient being perspective on his LLM usage.
- mikkupikku 11mo agoThis, but unironically. It's not much different from the way human unreliability is accounted for. Add more until you're satisfied a suitable ratio of mistakes will be caught.
- SoftTalker 11mo agoOr maybe it will be a circle of LLMs all coming up with different responses and all telling each other "You're absolutely right!"
- protocolture 11mo agoYeah but, real firewalls are deterministic. Hoping that a second non deterministic thing, will make something more deterministic is weird. Probably usually it will work, like probably usually the LLM can be unsupervised. but that 1% error rate in production is going to add up fast.
- PunchyHamster 11mo agoIt's "wonderfully" human way. Just like sometimes you need senior/person at power to tell the junior "no, you can't just promise the project manager shorter deadline with no change in scope, and if PM have problem with that they can talk with me", now we need Judge Dredd AI to keep the law when other AIs are bullied into misbehaving
- littlestymaar 11mo ago> For tasks that arent customer facing, LLMs rock. Human in the loop. Perfectly fine. But whenever I see AI interacting with someones customer directly I just get sort of anxious Especially since every mainstream model has been human preference-tuned to obey the requests of the user… I think you may be able to have an LLM customer facing, but it would have to be a purpose-trained one from a base model, not a repurposed sycophantic chat assistant.