6 ms·
Salesforce study finds LLM agents flunk CRM and confidentiality tests
- toomuchtodo 1y agoPaper: CRMArena-Pro: Holistic Assessment of LLM Agents Across Diverse Business Scenarios and Interactions - https://arxiv.org/abs/2505.18878 https://arxiv.org/abs/2505.18878 | https://doi.org/10.48550/arXiv.2505.18878 https://doi.org/10.48550/arXiv.2505.18878
- worldsayshi 1y agoThis makes me realize something: The internet has very little training data for "when to shut up". The bias is always towards more yapping.
- tempodox 1y ago+1. Actually, the infinitely many things that have never been posted would be such training data, but how do you count how much nothing you hoovered up while stealing data?
- danielbln 1y agoStealing implies the original is no longer there. I'm no fan of the large AI labs hoovering up the Internet, but let's keep our terminology accurate. We don't even know if this sort of crawling and training on public data constitutes infringement.
- dylan604 1y agoPedantry is so boring. In conversational parlance, stealing is often the meaning without paying for. So yes, pedantically, this would be unlicensed use of vs the removal of the original from the owner's possession. But what else do you want us to think when even the FBI pushed the copying is stealing bit with their logos at the head of DVDs/VHS tapes?
- chii 1y ago> this would be unlicensed use which is exactly what the parent poster is implying - the hoovering up of data off the internet may not be unlicensed use. After all, the information is not what's copyrighted, but the expression of it only. By calling it stealing, it already presupposes the idea that such hoovering is unlawful, before it is made clear that it is unlawful. And it prejudices the "jury" so to speak - the language for which you call the subject can influence other people's perception.
- notahacker 1y agoWe know for a fact that some LLM developers made digital copies of lots of copyrightable material for the purpose of training a system to create [unattributed] derivative works which had licenses expressly forbidding ingesting the content into an information retrieval system for the purpose of creating derivative works [without attribution], and that derivative works were produced, some of them containing substantial portions of content recognisably identical to copyrighted material. LLM providers are free to argue in and outside court that EULAs or software licences are not applicable to them or enforceable at all, or that their specific actions fell short of violations but it's far more prejudicial to wade into conversations to try to shut down any suggestion that it might be possible to do anything unlawful with an LLM.
- meepmorp 1y ago> Stealing implies the original is no longer there. It really doesn't, and I'm pretty sure even you regularly use the word 'steal' in a context where there's clearly no such implication.
- falcor84 1y agoNow that much of the input to AI systems is from the search tool, maybe post-training should indeed be treating the lack of a result as a signal, perhaps a bit like in TF-IDF, where something being more rare in the corpus as a whole implies that it's more unique and potentially meaningful to the current document.
- deleted 1y ago[deleted]
- themanmaran 1y agoThis is a big problem when it comes to conversational agents. Sometimes users ask questions that are really prying, potentially misleading, or just annoying repeats (like asking for a cheaper price 50 times). In these situations a real person would just ignore them. But most LLMs will cheerfully continue the conversation, and potentially make false promises or give away information they shouldn't.
- jotux 1y ago> But most LLMs will cheerfully continue the conversation, and potentially make false promises Example: https://www.bbc.com/travel/article/20240222-air-canada-chatbot-misinformation-what-travellers-should-know https://www.bbc.com/travel/article/20240222-air-canada-chatb...
- notahacker 1y agoIndeed I suspect if anything the weighting is the opposite (being annoyingly persistent weights and LLM towards spitting out text that approximates what the annoyingly persistent person wants to get), whereas with humans it weights then towards being less helpful...
- esafak 1y agoIf you value brevity, don't ask Gemini.
- el_benhameen 1y agoExcellent point! You’ve stumbled upon something fundamental about Gemini—it’s exceedingly verbose, even when answering the most mundane of queries. Let’s dig deeper …
- j45 1y agoOn one hand if responses were concise and perfectly clear (more than the human interacting with it), could it be unnerving? Prompting with clarity seems to help alleviate any accumulated response pressure where it's having to reach beyond what it has readily available. When it comes up short, it seems to dig deeper and come up with more than intended, or over respond. Jumping to solutions remains one of the biggest challenges.
- detourdog 1y agoThe generous interpretation is that the internet is a communication medium and everyone is just tying g to understand and be understood. The back forth is a continuous effort of clarification of the points being made. The process can break down resulting in no gain in clarity.
- zihotki 1y agoIs that the Salesforce that had recently announced that they are going to replace a lot of its staff with AI agents?
- lubujackson 1y agoLikely a political statement. Likewise, this seems to be a political pushback, as others have said they used a bad agent and got bad results - I am assuming some head of IT is trying to save some jobs (or pave a saner path). Not sure there is much of a real world takeaway from this.
- bionhoward 1y agolol, might have been good to conduct this study BEFORE making that decision
- onlyrealcuzzo 1y ago> lol, might have been good to conduct this study BEFORE making that decision Why? First, they wanted to do a layoff for financial reasons (and they did), secondly they came up with a reason for the layoffs (aside from the truth, which is needing to make more profit per employee, because growth). LLMs are a convenient scapegoat for firing decent employees just because you want your other ones to work harder so you can return more cash to shareholders.
- xnx 1y agoColor me "not-surprised" that a made-up benchmark by Salesforce shows that using a CRM is good.
- paxys 1y agoSo Salesforce spent a couple years hyping itself up as an "AI agents" company, failed at becoming a player in the space (because it was all marketing and no substance, as is their MO), and is now turning around and saying "LLMs are bad actually...". Sure bud.
- AstroBen 1y agoSaying they're biased isn't a good argument against their claim. You actually have to disprove the claim
- bitzun 1y agoI think it’s an argument against paying attention to anything Salesforce publishes, regardless of what they claim.
- hobs 1y agoThat would be the definition of ad hom then, anyone can publish science - the important part is if you take off the name is it reproducible and falsifiable. You hope it also is somewhat useful or tells us something we don't already know.
- bwfan123 1y agoFinally some real pushback to the whole agentic mania - from an actor who is incentivized to push the narrative. Following the recent apple paper - some realism is being injected into the hype. 58% success rate on a task is close to a coin flip. and 35% success rate on multiturn. >80% success rate on workflows could make that a reasonable usecase (eg, form filling) with some human supervision.
- bigbuppo 1y agoIf it were an employee it would have been fired already, unless it were a nepo hire, and in someways, it is.
- sieabahlpark 1y ago[dead]
- onlyrealcuzzo 1y agoIt might depend how much this employee costs. Your incentive to fire an employee who isn't great and costs $1 per day is much less than an incentive to fire one who isn't great and costs $1000 per day...
- bigbuppo 1y agoThere's a reason why I post the entire script to Bee Movie in every single AI-powered chat out there...
- AbstractH24 1y agoAs someone who works in and around tons of nepo babies, this comment led to a lot of reflection on the similarities/differences. Wins the award for "deep comment of the week"
- onlyrealcuzzo 1y ago> 58% success rate on a task is close to a coin flip. Why does a single-step task imply a coinflip to you? There are more than two possible choices for an instruction like: "Lookup the status of order X".
- anshumankmr 1y agoCan this not be solved by RBAC? But I am not sure what all questions were asked and what the setting was, what database was used, what prompts etc.
- morgango 1y agoFair question, slightly nuanced answer. If going against a datasource (like with Retrieval Augmented Generation), yes. If the information is just part of the context window, no.
- anshumankmr 1y agoIdeally I would not let anything in the context which is not authorized for the user or the bot is not authorized to do.
- simonw 1y agoPaper: https://arxiv.org/abs/2505.18878 https://arxiv.org/abs/2505.18878 Code: https://github.com/SalesforceAIResearch/CRMArena https://github.com/SalesforceAIResearch/CRMArena Data: https://huggingface.co/datasets/Salesforce/CRMArenaPro https://huggingface.co/datasets/Salesforce/CRMArenaPro (8,614 rows) Here's one of those JSON files loaded in Datasette Lite (15MB page load): https://lite.datasette.io/?json=https://huggingface.co/datasets/Salesforce/CRMArenaPro/raw/main/tasks_b2b.json#/data/tasks_b2b?_facet=reward_metric&_facet_array=answer https://lite.datasette.io/?json=https://huggingface.co/datas... I had Gemini 2.5 Pro extract the prompts they used from the code: llm install llm-gemini llm install llm-fragments-github llm -m gemini/gemini-2.5-pro-preview-06-05 \ -f github:SalesforceAIResearch/CRMArena \ -s 'Markdown with a comprehensive list of all prompts used and how they are used' Result here: https://gist.github.com/simonw/33d51edc574dbbd9c7e3fa9c9f79e9b4#response https://gist.github.com/simonw/33d51edc574dbbd9c7e3fa9c9f79e...
- heymijo 1y agoYou are a perpetual motion machine. Truly prolific.
- jzelinskie 1y agoI recommend folks check out the linked paper -- it's discussing more than just confidentiality tests as a benchmark for being ready for B2B AI usage. But when it comes to confidentiality, having fine-grained authorization securing your RAG layer is the only valid solution that I've seen in used in industry. Injecting data into the context window and relying on prompting will never be secure.
- sausagefeet 1y agoIs that sufficient? I'm not very adept at modern AI but it feels to me like the only reliable solution is to not have the data in the model at all. Is that what you're saying accomplishes?
- rafaelmn 1y agoYes. It's basically treat the model as another frontend approach - that way the model has the same scopes as any frontend app would.
- CityOfThrowaway 1y agoThis paper doesn't make any sense. They are claiming LLMs are bad at this set of tasks, but the reality is that they built a bad agent. I bet it's possible to nearly ace this using existing LLMs by designing a better agent. Better tool structure, better scaffolding, better prompting. LLMs are not gods, they are tools that require good engineering to achieve good outcomes.
- contagiousflow 1y agoHow is that an argument at all? Of course if you could build a better agent that could solve every problem the outcome of the paper would be "this tool performs well at this"
- handfuloflight 1y agoIt is an argument: a flawed agent lead to flawed results. A flawed agent does not speak for all agents.
- oblio 1y agoThe "sufficiently smart compiler" debate, 50 years later :-p
- handfuloflight 1y agohttps://en.wikipedia.org/wiki/Faulty_generalization https://en.wikipedia.org/wiki/Faulty_generalization
- contagiousflow 1y agoBut the argument should be showing an agent that does in fact pass these tests. You can't just assert that "this one failed, but surely there must be some agent that is perfect, therefore you can't generalize".
- handfuloflight 1y ago
- einrealist 1y agoRemember that increasing the accuracy/correctness does not solve the problem. It only increases the cost of identifying cases where the LLM has failed. That's why I am highly sceptical about using LLMs in situations where accuracy matters. And that's even if humans are kept in the loop (we are lazy and are biased towards trusting computations).
- cycomanic 1y agoI was posting this the other day. I find that all llms no matter their benchmark scores make enough mistakes that I always have to check their work, so pretty much any chat with an llm ends up like this: Me: question... Llm: certainly the answer is... Me: that answer can't be correct because of some test case... Llm: Certainly, my previous answer was obviously incorrect (if it was obviously wrong why give it to me?), here is the correct solution The same pattern continues for a couple of iterations until I get the correct solution. The problem is, the llm responses are so slow that I could just work out the problem myself in the time (I typically ask questions that I know I can solve, it just takes too much time at the moment, e.g. Just yesterday I asked a question about some interlocked indeces, which I was to lazy to work out myself at the time). Instead of the llms with increasing benchmark scores I want an llm that is of similar level to the current ones, but answers instantaneously so I can iterate quickly.
- rjst01 1y agoThe headline here makes it sound (to me) like Salesforce did the study.
- profstasiak 1y agojudging from the comments most of the people read it like Salesforce did the study
- burningChrome 1y agoIt sure sounds like it in the article: A team led by Kung-Hsiang Huang, a Salesforce AI researcher, showed that using a new benchmark relying on synthetic data, LLM agents achieve around a 58 percent success rate on tasks that can be completed in a single step without needing follow-up actions or more information. and The Salesforce AI Research team argued that existing benchmarks failed to rigorously measure the capabilities or limitations of AI agents, and largely ignored an assessment of their ability to recognize sensitive information and adhere to appropriate data handling protocols.
- 0xffff2 1y agoThe article also makes it sound like that. Are you saying they didn't? I don't see any reference in the article to any other organization that could have done the research. Edit: Unless "Salesforce AI Research" is not a part of Salesforce, I think Salesforce did do the research.
- b0a04gl 1y ago[dead]