8 ms·
Kinney Drugs pulls back AI phone assistant after hundreds of customer complaints
- kotaKat 2mo agoRelated: A pharmacy chain in Vermont implemented AI for efficiency (11 days ago, 40 points) https://news.ycombinator.com/item?id=49105190 https://news.ycombinator.com/item?id=49105190
- recursive 2mo agoWow, it lasted over a week.
- nubg 2mo ago> Marraffa says Burt is fully HIPAA compliant and that the AI is not open-source and does not generate or manipulate data. We need to fight the talking point that open source is bad!
- aswegs8 2mo agoHow could something like open source ever be perceived as bad?
- freeone3000 2mo agoThe idea, within regulated circles, is that it lacks provenance - there’s no vendor to sue. Many open source licenses also carry an explicit disclaimer of merchantability and fitness for purpose. These are taken seriously, and open-source software is viewed akin to car parts found on the side of the road. They could work but it’s better to let someone else verify.
- altmanaltman 2mo agoBut in the context of privacy it makes no sense though right? Open source and close source models (depending on vendor) both can be configured in a way that protects privacy of the user. The entire quote is wonky despite your point about companies preferring vendor-based system rather than raw dog oss
- stronglikedan 2mo agobut if the source is open, the ay-eyes can find the zero days! /s
- altmanaltman 2mo agohow can it not manipulate data and still work? does it do it by magic? absolute L quote by the spokesperson
- nazgulsenpai 2mo agoIt will probably sound nice to laypersons but you're right, it has to have access to, at a minimum, a data extract with relevant information to perform it's tasks, although in my experience in the pharmacy IT world, a database view.
- cmiles8 2mo agoCompanies will pawn off thier customers to AI at their peril. In many ways this is a repeat of the India call center train wrecks of the 00s. On paper, letting someone in Bangalore vs onshore handle incoming customer service calls looked like a path to amazing savings. In practice the customer experience was horrendous and companies CTRL-Zed these decisions and rapidly brought customer service back onshore again. AI is just that story of shortsighted decisions by weak leadership playing out all over again.
- graemep 2mo agoIts still a huge and growing business and one that AI helps https://tech.yahoo.com/ai/articles/call-centres-using-ai-whiten-090000847.html https://tech.yahoo.com/ai/articles/call-centres-using-ai-whi...
- moomoo11 2mo agolol at the title
- munk-a 2mo agoThis, I suspect, is exactly like offshoring - and a lot of companies died during the offshoring fad because their software products become full of low quality merges and feature improvements that the company couldn't afford to fix or remove by the time they realized there was a problem. That remains my primary concern with AI code generation - well established codebases might become full of slop to the point of being unusable without a clear path back to maintainability leading to company death.
- jdw64 2mo agoI think of this as the kiosk economy. It's like how businesses introduce kiosks even when having a human employee would be more convenient, just for the sake of greater profit. The biggest problem with AI customer service is that a human employee would've let a minor issue slide without escalating it. But a chatbot often inflames the situation, and by the time the customer reaches a human agent, they're already furious. Most people aren't rational or logical. Non verbal feedback, like acknowledging someone's anger and showing empathy, is incredibly important.
- dqv 2mo agoAI customer service doesn't fucking listen. A few years ago, I noticed that my bank had an old address on file for only one of my accounts. When I clicked edit next to the old address, it would bring me to a page that had the correct address. At the time, they had a real person handling their support. The real person told me this was just a glitch and that my correct address was on file and everything is fine. That wasn't actually true. A new debit card to replace my expiring one ended up being sent to the old address a couple of months ago. So I go back to the website to try to edit it one last time. Same problem. I call them and get their fucking stupid AI slopbot. Still, I explain the whole situation: > I'm trying to update my address on the website. When I click the edit link next to the old address, it shows my new address instead. I need to remove the old address. What did the slopbot decide to say? Oh well, I should go on the website and edit my address of course! I gave it two more chances by trying to explain it a different way and how this didn't work. Did it accept that and escalate the issue to a human? No. It kept telling me to go to the website. I had to say > Hey you fucking bitch, how many times do I have to tell you that doesn't fucking work? Escalate this to a fucking human before I get the attorney general involved. to finally get a live person to fix it (I actually had to do this twice, the first set of fucks, the slopbot forgot what problem I was having and needed a refresher!). Suffice it to say, I will never trust any large amount of money with this bank again. I'm highly skeptical of any company using AI CS now, because this is one of my many experiences where the slopbot is incapable of recognizing a complex situation outside of its capabilities. I mean look at this situation: their AI slopbot turned this one mistake into 5 mistakes instead of just allowing it to be one that another human corrects. It's gotten to the point where whenever I see someone lauding AI customer support, I just assume they have an investment/job that hinges on people liking it and/or their only experiences are using it as a glorified refund button.
- newswasboring 2mo agoThe desired speed of AI adoption is what is hindering AI adoption. Big companies are trying to sell this as magic, and it's not. It requires proper use to get anything useful out of this system. But that takes away the magic of it and the investors can't have that as the eventual share price only works if it is magic. Frontier labs will smother their product with their timelines.
- zucked 2mo agoThis is probably the most pragmatic take here and the overall responses support it. There's a handful of practitioners who are seeing some implementation success. They probably had significant domain expertise and are several iterations deep. Then there's a bunch of people who are experiencing the "AI iS mAgIc" approach where LLMs are being shoehorned into workflows with very little effort given to make the process actually… work (well)… to satisfy investor-driven mania. This growing-pains stage will pass eventually and we'll hit some equilibrium.
- taurath 2mo agoAnd even if the AI worked well enough (a claim to be proven by its proponents), companies wouldn’t give it access to be actually helpful. It’s all to save costs on routine calls, not enhance problem solving, the same way they’ve outsourced CS operators to body shops with a strict script.
- eithed 2mo agoIt's interesting that I had to close 3 popups just to read this article, two of them were for tracking cookies
- phyzix5761 2mo agoGDPR popups remind me of the spam popups of the 90s.
- cjs_ac 2mo agoGDPR requires websites to get user consent before doing sketchy shit. You don’t need a cookie popup if you’re not a scumbag.
- anon48293 2mo agoNo, maliciously compliant scumbag companies did that. You don’t need a cookie banner for functional cookies, and opt out should be a single button.
- karim79 2mo agoAgreed. It's horrible and stupid. I actually run a service in the EU and I don't present that nonsense even with the risk of being sued.
- asib 2mo agoWhat is your service and what data are you collecting?
- asdff 2mo agoAlways surprising to me to find people in the HN comments who don't use adblockers in $CURRENTYEAR
- asdev 2mo agoif you've tried to legitimately productionize LLMs to deliver a specific repeatable outcome, rather than human editable output(text, code), you know the models still aren't there yet and the whole product experience feels hacky
- 27183 2mo agoNot to mention insecure.
- exizt88 2mo agoAs someone who's been doing AI call centres for a while, I'm not surprised. It's really, really hard to build voice AI that works. There are no open source solutions that work out of the box. But the biggest problem is ASR. WER is still atrocious even with SOTA models. When you add drug names and regional accents, it's a recipe for disaster.
- fn-mote 2mo agoASR = Automatic Speech Recognition WER = Word Error Rate Too many acronyms; not everybody is in your field.
- 8n4vidtmkvmk 2mo agoJust curious, why are drug names such a disaster? Do LLMs not understand pronunciation keys?
- thunderfork 2mo agoPeople pronounce drug names all sorts of ways, and there's a lot of them (with a steady trickle of new ones)
- munk-a 2mo agoDrug names are frequently BS loaded with strange consonants without singular pronunciation guidelines (sadly English isn't German) so humans already have a lot of variation in how things are pronounced - models then need to comprehend those variations, regional dialects (like those fuggin' ahs from Boston). It ends up being quite a lot of variation. As some examples: Wegovy, Ixempra, Qvar, Keflex
- recursive 2mo agoWow, those names are actually real. They sound like fly-by-night compliance "brand" names on amazon sellers. How did those get approved by a multi-billion dollar marketing team?
- 2mo ago
- pessimizer 2mo agoIt's important to note while people are pumping the AI bubble that a near-moron with a checklist can do this job. They replaced AI with a regular old phone tree. AI couldn't do this job with a checklist. It would forget the entire checklist after two responses, then get fixated on the words that the customer used to complain about how dumb the AI was and loop until they hung up and called back to hopefully clear the context. edit: Unless Kinney had been sold a product that used caller ID to carry over context, then the customer's best chances would be to scream "Agent!" over and over again (to get put on hold for 45 minutes, then transferred to the most screamed-at Philippine minimum-wage gig worker on the planet), or to show up to the drugstore and start screaming at people.
- SalaevAl 2mo agoPharmacy phone lines are a terrible first use case for voice AI. Older callers, drug names, insurance mess, zero patience. Automating "store hours" is fine. Automating refill exceptions is how you get this outcome.
- barbazoo 2mo agoI just don't get why we're not doing it vertically like you said here. Implement a part of the cognitive work using AI but let the hard parts be done by a person. They're still gonna be more rich, just less so.
- DANmode 2mo agoYou figured it out, there, at the end!
- AndrewKemendo 2mo agoExplain to me then why pharmacies have almost exclusively used IVR based phone systems for pharmacy refills and prescription management for the last decade? Like they don’t even say the full name of the prescription they just tell you the first three letters and they’ll say “for the patient born on…” Current pharmacy IT systems are currently almost 100% automated so how are you claiming it’s the first use?
- superb_dev 2mo agoDo you actually use these systems? They can’t at all handle my case where half the time the drug is on back order or there some insurance non sense in the way. I need to talk to a human every time
- AndrewKemendo 2mo agoCorrect they fail in any out of boundary condition which is like 50% of cases in my experience Even worse is that the human is usually just as worthless because the problem is often upstream at the insurance provider or regulation This is something I’m finding with people complaining about AI systems the AI system in this case is not any worse than the previous IVR system was it just expanded the use cases and now people say oh it doesn’t work for those expanded use cases There’s no mention whatsoever of the current absolute devastatingly bad state of all software across the world It’s like people are mad when there’s an application that already sucks, someone attempts to use genAI, that fails and then they blame AI The whole thing is absurd
- jonathan-adly 2mo agoSo - I run a company that does AI agents for pharmacy. We are at Series B now and growing well - series C in 9-12 months given growth. We just acquired a competitor of that vendor so very much an insider. First, The technology works, and it scales, but the whole bottleneck is domain expertise and implementation. These are expensive, and hard to scale. We hire pharmacists as project managers, that's how important domain expertise and implementations are. Second, the amount of noise of "Voice AI for <industry>" is incredible. Most of them are completely clueless about the industry & basically "YC-striver" type who can only sell to other yc companies. They fail hard the moment they touch critical functions of the real world. We have hundreds of pharmacies with us now and they mostly successful and happy. Our biggest friction is customers being burned by vendors like this and writing off AI altogether as "not ready".
- barbazoo 2mo agoMake it so the parts that are done by AI are rock solid, until then, they shouldn't be near a customer and be done by humans. Resist the greed!
- lagrange77 2mo agoSo you want him to make a piece of software that models a probability distribution rock solid.
- pasc1878 2mo agoYes
- drdeca 2mo agoMaybe “rock solid” could be understood as slightly hyperbolic (though it is metaphorical, so hard to say whether it is really hyperbole), but, it is possible to predict some things probabilistically with a high degree of accuracy. Depending on the task, a high enough degree of accuracy sounds doable, and if it offers a sufficiently convenient fallback to an actual human when it fails, that sounds like it could be quite a bit nicer than navigating a phone tree menu? Classical voice to text combined with NN based voice to text, I would imagine can be highly accurate, and that’s probabilistic.
- criddell 2mo agoThis is my favorite AI customer service story: https://arstechnica.com/tech-policy/2024/02/air-canada-must-honor-refund-policy-invented-by-airlines-chatbot/ https://arstechnica.com/tech-policy/2024/02/air-canada-must-... > Air Canada essentially argued that “the chatbot is a separate legal entity that is responsible for its own actions”
- impendia 2mo agoLOL, pleased to read that Air Canada got smacked down. Yeah, if I'm calling or chatting with a company, I expect them to stand by whatever commitments they make, and I will judge them harshly if they don't. This is true whether an AI or a human is speaking on the company's behalf.
- uncertainrhymes 2mo agoThat event happened in 2022. It keeps being tagged as "AI" when it was nothing more than a dumb front end for a document search. Yes, it still screwed up. And yes, Air Canada tried to weasel out of it, but even though it was a chatbot it had no LLM behind it.
- criddell 2mo agoIt may not have been an LLM (nobody said it was) but it was definitely more than just a document search. AI systems were in use for decades before LLMs rolled around.
- digitist 2mo ago[dead]
- crystaln 2mo agoI guess everything is ok these days as long as the President says "We screwed up [our most basic responsibility to customers], and we own that."
- dj_axl 2mo agoFWIW I know of a pharmacy network that's implementing AI. Not only does the AI make egregious errors like speaking in Spanish when the conversation is in English. It also has a limited context window for ground rules of 5000 tokens or less. So correcting all the various errors goes over the instructions limit. As a result if the implementation goes through it will be of poor quality. All I can think is that people with no technical expertise are making these decisions, it is easy for any techie to predict that AI of this level won't be capable.
- PhilipRoman 2mo agoThis is an underappreciated point when people talk about AI/chat integrations. If they put a claude-level model behind it, I'm sure it could resolve 99% of the situations faster and better than a human could, or at least realize its limitations when it can't. But of course that is too expensive so they often use extremely weak models that are literally worse than a good incremental full text search box.
- HPsquared 2mo agoI wonder what the price level (per token) needs to be, to match a minimum wage employee.
- themgt 2mo ago100x higher than it is now?
- asdff 2mo agoAnother aspect of the human version is that the human version is deterministic. Call center workers have a flow that is pretty much a series of if else statements with appropriate edge cases already carved out and places to break out and escalate to another provider. These flows have been iterated over for some time and refined to this point. AI unfortunately is not deterministic. It would be nice if you could actually use it deterministically in this way, set a seed such that the logic flow are preserved across all sessions well into the future unless the logic flow instructions are changed. They just don't work like that though. Even if you have a good harness they can go off the rails pretty easily and need coaxing to get back to appropriate context. That is a nonstarter for the typical caller of a call center. Functionally speaking just an automated call where you touch type through options would be equivocal to the human operator and that paradigm has replaced a lot of call center work already, with the human operator serving to smooth things over if the person is old or can't comprehend the options for whatever reason and starts yelling incoherently into the automated line (my actual strategy for getting human operators on the line). Everything becomes a nail though when you have a hammer though, even more so when you are selling hammers. So expect to see AI in plenty of places where it makes no sense to belong given the supremacy of existing tooling. There is also no advocate for a lot of existing tooling unlike AI that has sales teams active in pursuit of new customers integrating this into their product.
- fn-mote 2mo ago> wrong dosages, and missed prescription notifications. BS. How is the AI touching that information? Prescription notification is outbound communication. It’s almost a recorded message. A little text to speech if it is supposed to read the drug name. Wrong dosages?? This must be handled by the “enroll a new scrip” functionality, right? So basically B2B doctor interaction. You let AI touch that? Aren’t the vast majority of the costs from customers not doctors? None of this makes sense.
- laughingcow 2mo agoI happened to be in Vermont last week and was unaware of this change. I heard complaints from three separate people unprompted. It was a total dumpster fire put on top of a critical service.
- sicktriple 2mo agoDo we think that as a society there exists some meaningful lever by which we can apply force against this type of carelessness and exploitation? I think we need to take a good hard look at what we want out of this world and ask ourselves. Do we really need to automate this? Like really. What benefit does the world receive having an AI chatbot tell grandma the wrong doage for her meds? This is the naked extraction of value from a service that's already been offshored. Companies like these are gleeful to make their service as shitty as they can because they know their users are doing so against their will. Infact, that's become the goal.
- designium 2mo agoHow open source makes it less private?!?
- coffeefirst 2mo agoThis doesn’t surprise me. Set up whatever online portal or app you want. I’d prefer to just sign in and click a button. If I’m picking up the phone or sending an email it’s because I have a weird problem and I need to talk to a person.
- wolfi1 2mo agomy experience with AI assistants has not been that great so far, they ask for customer id numbers which are long as ... with several digits and if you group them they don't understand, for example if the number begins with two 1s and you say eleven instead of one one they don't understand you, then they are usually not trained for the local pronunciation or dialect which makes it very hard communicating, my blood pressure rises into unhealthy ranges, but if I use expletives enough I get a human on the line
- vanuatu 2mo agowonder how much of the ai CX hate is availability bias where people really notice it when it breaks but you never see the whole picture on total impact it seems very implementation dependant, I know a startup with incredibly realistic voice agents that most people don’t detect is an ai that outperforms humans. But ai for CX seems like an easy project to do for companies looking to cut costs so the average agent a user encounters in the wild is poor quality
- Glyptodon 2mo agoNot Pharmacy, but I literally had Shopify checkout button clickjack me and then AI agent and human agents (chat, email, and phone) refuse to cancel the resulting order today. Despite contacting within 5 minutes. Given how hostile so many companies are, I question why they even bother appearing to have support. But this doesn't seem likely to translate well to pharmacy, where there will be consequences for treating customers like vermin, as opposed to actual incentives to do it. I wouldn't be shocked if the tech needed in places like pharmacy or other sectors where there's some level forced responsibility leads to the version that's actually palatable elsewhere.
- dkarl 2mo agoCases like this raise a question. As an engineer, the errors that coding AIs make are an annoyance, and reassuring in terms of my job prospect. As a consumer, the errors that AIs make are more than an annoyance. I have yet to meet an AI phone assistant that can do anything more than an explicitly programmed phone tree, and several, including the one for the pharmacy I use, that do far less. They are undoubtedly much more expensive to create and test. Clearly there's a bubble going on, and unprincipled people chasing funding and promotions as usual, but why are companies like CVS paying more to get less? I think it is literally to waste our time. It's one more layer of defense to stop you from talking to a person. There are two big wins in health care. One is to increase the scope of health care that gets paid for. More and more expensive treatments, to bring the money in. The second big win is to stop people from getting treatment. Deny coverage, confuse them and wear them out with bureaucracy, and in this case stop them from talking to an expensively educated human pharmacist. That's what an AI does better than a phone tree, because it more effectively creates a sense of helplessness and valuelessness in the customer. Expanding the scope of care you're entitled to and then fighting against your ability to get it isn't a contradiction any more than it's a contradiction when a person breathes in and breathes out. It's two synergistic parts of a system whose goal ultimately has nothing to do with health care.
- ericmcer 2mo agoAll of these conversational AIs should either confidently detect a users intent or apologize and offer to forward call to a real person. LLMs perform pretty poorly when you start try to restrict them too much (e.g. detect intents and control responses based on intent detection), the replies gets disjointed and weird feeling. An open ended prompt and chat completion gives a much better feeling most of the time but can go off the rails. If you don't let LLMs handle the conversation entirely you get into insanely complex traditional engineering problem: (if user is complaining do x, if user asks a relevant question do y, if user asks an irrelevant question do z), which is an endless can of worms.
- pseudohadamard 2mo agoThat's been my experience with AI "support" chatbots as well, in every single case it's been an optimistic attempt to get the bot to fix the problem I'm calling with followed almost immediately by an even longer attempt to get it to connect me to a human that can actually fix the problem. If the problem was simple enough that a fixed-program clanker can fix it then I wouldn't be having to call support in the first place.
- drewg123 2mo agoActual personal service from a human being is the main reason that I'd choose a small regional pharmacy over a local CVS or mail-order. It seems to me like personal service is the ONLY advantage a regional chain like this has. If I were them, I'd lean into it rather than burning customer's goodwill with AI.
- CrzyLngPwd 2mo agoMy partner got trapped in an email loop with a support agent and my goodness was she pissed. Still, I got it to send a chocolate chip cookies recipe to us. These things are horrible, but then coding agents were dumb as dogshite not too long ago, so maybe in a few years AI support agents will become useful.
- benmorris 2mo agoI'm testing it in my business. I'm in a unique position of running a business totally unrelated to software dev, handled customer service personally for 15+ years. I developed a AI backed live chat app first a little over a year ago. It has handled ~6,000 chats since then so I have a ton of valuable data on what does and doesn't work (spoiler alert, it works really well). I'm taking that information and building a voice version. I'm taking a lot of time to make sure the phone version works naturally and that is can address most common things people call in about. Just like my chat version, GPT 4o was barely good enough to do what I needed. Since GPT 5 and on it has been more than adequate. I expect the same thing over time with voice, Open AI is going to release the GPT-live to the API which is a noticeable improvement. Some people are just offended at the idea of talking to an AI chatbot voice or phone, no matter how good it is. I don't know how good this Kinney Drugs bot was, but it's very easy to make something that works but ends up being very frustrating for customers.
- dmix 2mo agoThe first versions of these were always going to be terrible. Like all new technology. It's usually the big old companies who fail to realize this early on.
- autoexec 2mo agoIt's amazing how many times AI has failed to perform even the most basic jobs. Tasks often handled by a teenager or a phone tree are too difficult for AI to manage, but somehow we're supposed to worried that this same AI is going to put doctors and lawyers and engineers out of a job.
- josefritzishere 2mo agoMost AI is an annoyance. I'd complain.
- gloosx 2mo ago>Customers also raised concerns about data privacy - Marraffa says that the AI is not open-source and does not generate or manipulate data. Marraffa should hire some better PR people, closed-source AI which doesn't generate or manipulate data is just a hilarious thing to say adressing data-privacy concerns;)
- rafsan_rafsan 2mo ago[dead]
- deleted 2mo ago[deleted]
- rafsan_rafsan 2mo ago[flagged]