13 ms·
I don't trust LLMs to do the kind of precise deterministic work you need in a spreadsheet. It's one thing to fudge the language in a report summary, it can be
by cube00 1y ago
I don't trust LLMs to do the kind of precise deterministic work you need in a spreadsheet.
It's one thing to fudge the language in a report summary, it can be subjective, however numbers are not subjective. It's widely known LLMs are terrible at even basic maths.
Even Google's own AI summary admits it which I was surprised at, marketing won't be happy.
Yes, it is true that LLMs are often bad at math because they don't "understand" it as a logical system but rather process it as text, relying on pattern recognition from their training data.
- extr 1y agoSeems like you're very confused about what this work typically entails. The job of these employees is not mental arithmatic. It's closer to: - Log in to the internal system that handles customer policies - Find all policies that were bound in the last 30 days - Log in to the internal system that manages customer payments - Verify that for all policies bound, there exists a corresponding payment that roughly matches the premium. - Flag any divergences above X% for accounting/finance to follow up on. Practically this involves munging a few CSVs, maybe typing in a few things, setting up some XLOOKUPs, IF formulas, conditional formatting, etc. Will AI replace the entire job? No...but that's not the goal. Does it have to be perfect? Also no...the existing employees performing this work are also not perfect, and in fact sometimes their accuracy is quite poor.
- Ntrails 1y agoChecking someone elses spreadsheet is a fucking nightmare. If your company has extremely good standards it's less miserable because at least the formatting etc will be consistent... The one thing LLMs should consistently do is ensure that formatting is correct. Which will help greatly in the checking process. But no, I generally don't trust them to do sensible things with basic formulation. Not a week ago GPT 5 got confused whether a plus or a minus was necessary in a basic question of "I'm 323 days old, when is my birthday?"
- szundi 1y ago[dead]
- xmprt 1y agoI think you have a misunderstanding of the types of things that LLMs are good at. Yes you're 100% right that they can't do math. Yet they're quite proficient at basic coding. Most Excel work is similar to basic coding so I think this is an area where they might actually be pretty well suited. My concern would be more with how to check the work (ie, make sure that the formulas are correct and no columns are missed) because Excel hides all that. Unlike code, there's no easy way to generate the diff of a spreadsheet or rely on Git history. But that's different from the concerns that you have.
- collingreen 1y agoI've built spreadsheet diff tools on Google sheets multiple times. As the needs grows I think we will see diffs and commits and review tools reach customers
- break_the_bank 1y agohey Collin! I am working on an AI agent on Google Sheets, I am curious if any of your designs are out in the public. We are trying to re-think how diffs should look like and want to make something nicer than what we currently have, so curious.
- collingreen 1y agoHi! Nothing public nor generic enough to be a good building block. I found myself often frustrated by the tools that came out of the box but I believe better apis could make this slightly easier to solve. The UX of spreadsheet diffs is a hard one to solve because of how weird the calculation loops are and how complicated the relationship between fields might be. I've never tried to solve this for a real end user before in a generic way - all my past work here was for internal ability to audit changes and rollback catastrophes. I took a lot of shortcuts by knowing which cells are input data vs various steps of calculations -- maybe part of your ux is being able to define that on a sheet by sheet basis? Then you could show how different data (same formulas) changed outputs or how different formulas (same data) did differently? Spreadsheets are basically weird app platforms at this point so you might not be able to create a single experience that is both deep and generic. On the other hand maybe treating it as an app is the unlock? Get your AI to noodle on what the whole thing is for, then show diff between before and after stable states (after all calculation loops stabilize or are killed) side by side with actual diffs of actual formulas? I feel like Id want to see a diff as a live final spreadsheet and be able to click on changed cells and see up the chain of their calculations to the ancestors that were modified. Fun problem that sounds extremely complicated. Good luck distilling it!
- lossolo 1y agoLast time, I gave claude an invoice and asked it to change one item on it, it did so nicely and gave me the new invoice. Good thing I noticed it had also changed the bank account number.. The more complicated the spreadsheet and the more dependencies it has, the greater the room for error. These are probabilistic machines. You can use them, I use them all the time for different things, but you need to treat them like employees you can't even trust to copy a bank account number correctly.
- mikeyouse 1y agoWe’ve tried to gently use them to automate some of our report generation and PDF->Invoice workflows and it’s a nightmare of silent changes and absence of logic.. basic things like specifically telling it “debits need to match credits” and “balance sheets need to balance” that are ignored.
- wholinator2 1y agoYeah, asking llm to edit one specific thing in a large or complex document/ codebase is like those repeated "give me the exact same image" gifs. It's fundamentally a statistical model so the only thing we can be _certain_ of is that _it's not_. It might get the desired change 100% correct but it's only gonna get the entire document 99 5%
- onion2k 1y agoSomething that Claude Sonnet does when you use it to code is write scripts to test whether or not something is working. If it does that for Excel (e.g. some form of verification) it should be fine. Besides, using AI is an exercise in a "trust but verify" approach to getting work done. If you asked a junior to do the task you'd check their output. Same goes for AI.
- dpoloncsak 1y agoSysadmin of a small company. I get asked pretty often to help with a pivot table, vlookup, or just general excel functions (and smartsheet, these users LOVE smartsheet)
- toomuchtodo 1y agoIndeed, in a small enough org, the sysadmin/technologist becomes support of last resort for all the things.
- JumpCrisscross 1y ago> these users LOVE smartsheet I hate smartsheet… Excel or R. (Or more often, regex followed by pen and paper followed by more regex.)
- dpoloncsak 1y agoThey're coming to me for pivot tables.... Handing them regex would be like giving a monkey a bazooka
- AvAn12 1y ago> “Does it have to be perfect?” Actually, yes. This kind of management reporting is either (1) going to end up in the books and records of the company - big trouble if things have to be restated in the future or (2) support important decisions by leadership — who will be very much less than happy if analysis turns out to have been wrong. A lot of what ties up the time of business analysts is ticking and tying everything to ensure that mistakes are not made and that analytics and interpretations are consistent from one period to the next. The math and queries are simple - the details and correctness are hard.
- 2b3a51 1y agoThere is another aspect to this kind of activity. Sometimes there can be an advantage in leading or lagging some aspects of internal accounting data for a time period. Basically sitting on credits or debits to some accounts for a period of weeks. The tacit knowledge to know when to sit on a transaction and when to action it is generally not written down in formal terms. I'm not sure how these shenanigans will translate into an ai driven system.
- AvAn12 1y agoThat’s the kind of thing that can get a company into a lot of trouble with its auditors and shareholders. Not that I am offering accounting advice of course. And yeah, one can not “blame” and ai system or try to ai-wash any dodgy practices.
- iamacyborg 1y ago> Sometimes there can be an advantage in leading or lagging some aspects of internal accounting data for a time period. This worked famously well for Enron.
- extr 1y agoSpeak for yourself and your own use cases. There are a huge diversity of workflows with which to apply automation in any medium to large business. They all have differing needs. Many excel workflows I'm personally familiar with already incoporate a "human review" step. Telling a business leader that they can now jump straight to that step, even if it requires 2x human review, with AI doing all of the most tediuous and low-stakes prework, is a clear win.
- next_xibalba 1y agoThe use cases for spreadsheets are much more diverse than that. In my experience, spreadsheets just as often used for calculation. Many of them do require high accuracy, rely on determinism, and necessitate the understanding of maths ranging from basic arithmetic to statistics and engineering formulas. Financial models, for example, must be built up from ground truth and need to always use the right formulas with the right inputs to generate meaningful outputs. I have personally worked with spreadsheet based financial models that use 100k+ rows x dozens of columns and involve 1000s of formulas that transform those data into the desired outputs. There was very little tolerance for mistakes. That said, humans, working in these use cases, make mistakes >0% of the time. The question I often have with the incorporation of AI into human workflows is, will we eventually come to accept a certain level of error from them in the way we do for humans?
- jay_kyburz 1y ago>Does it have to be perfect? Also no. Yeah, but it could be perfect, why are there humans in the loop at all? That is all just math!
- mrcwinn 1y agoI couldn’t agree more. I get all my perfectly deterministic work output from human beings!
- goatlover 1y agoIf only we had created some device that could perform deterministic calculations and then wrote software that made it easy for humans to use such calculations.
- bryanrasmussen 1y agook but humans are idiots, if only we could make some sort of Alternate Idiot, a non-human but every bit as generally stupid as humans are! This A.I would be able to do every stupid thing humans did with the device that performed deterministic calculations only many times faster!
- baconbrand 1y agoYes and when the AI did that all the stupid humans could accept its output without question. This would save the humans a lot of work and thought and personal responsibility for any mistakes! See also Israel’s Lavender for an exciting example of this in action.
- laweijfmvo 1y agoI don't trust humans to do the kind of precise deterministic work you need in a spreadsheet!
- baconbrand 1y agoRight, we shouldn’t use humans or LLMs. We should use regular deterministic computer programs. For cases where that is not available, we should use a human and never an LLM.
- extr 1y ago"regular deterministic computer programs" - otherwise known as the SUM function in Microsoft Excel
- szundi 1y ago[dead]
- davidpolberger 1y agoI like to use Claude Code to write deterministic computer programs for me, which then perform the actual work. It saves a lot of time. I had a big backlog of "nice to have scripts" I wanted to write for years, but couldn't find the time and energy for. A couple of months after I started using Claude Code, most of them exist.
- baconbrand 1y agoThat’s great and the only legitimate use case here. I suspect Microsoft will not try to limit customers to just writing scripts and will instead allow and perhaps even encourage them to let the AI go ham on a bunch of raw data with no intermediary code that could be reviewed. Just a suspicion.
- doug_durham 1y agoSure, but this isn't requiring that the LLM do any math. The LLM is writing formulas and code to do the math. They are very good at that. And like any automated system you need to review the work.
- causal 1y agoExactly, and if it can be done in a way that helps users better understand their own spreadsheets (which are often extremely complex codebases in a single file!) then this could be a huge use case for Claude.
- deleted 1y ago[deleted]
- bg24 1y ago"I don't trust LLMs to do the kind of precise deterministic work" => I think LLM is not doing the precise arithmetic. It is the agent with lots of knowledge (skills) and tools. Precise deterministic work is done by tools (deterministic code). Skills brings domain knowledge and how to sequence a task. Agent executes it. LLM predicts the next token.
- deleted 1y ago[deleted]
- zarmin 1y ago>I don't trust LLMs to do the kind of precise deterministic work not just in a spreadsheet, any kind of deterministic work at all. find me a reliable way around this. i don't think there is one. mcp/functions are a band aid and not consistent enough when precision is important. after almost three years of using LLMs, i have not found a single case where i didn't have to review its output, which takes as long or longer than doing it by hand. ML/AI is not my domain, so my knowledge is not deep nor technical. this is just my experience. do we need a new architecture to solve these problems?
- baconbrand 1y agoML/AI is not my domain but you don’t have to get all that technical to understand that LLMs run on probability. We need a new architecture to solve these problems.
- CodeNest 1y ago[dead]
- chpatrick 1y agoThey're not great at arithmetic but at abstract mathematics and numerical coding they're pretty good actually.
- mhh__ 1y agoIf LLMs can replace mathematica for me when I'm doing affine yield curve calculations they can do a DCF for some banker idiots
- sdeframond 1y ago> I don't trust LLMs to do the kind of precise deterministic work you need in a spreadsheet. Rightly so! But LLMs can still make you faster. Just don't expect too much from it.
- mbreese 1y agoI don’t see the issue so much as the deterministic precision of an LLM, but the lack of observability of spreadsheets. Just looking at two different spreadsheets, it’s impossible to see what changes were made. It’s not like programming where you can run a `git diff` to see what changes an LLM agent made to a source code file. Or even a word processing document where the text changes are clear. Spreadsheets work because the user sees the results of complex interconnected values and calculations. For the user, that complexity is hidden away and left in the background. The user just sees the results. This would be a nightmare for most users to validate what changes an LLM made to a spreadsheet. There could be fundamental changes to a formula that could easily be hidden. For me, that the concern with spreadsheets and LLMs - which is just as much a concern with spreadsheets themselves. Try collaborating with someone on a spreadsheet for modeling and you’ll know how frustrating it can be to try and figure out what changes were made.
- informal007 1y agoyou might trust when the precision is extremely high and others agree with that. high precision is possible because they can realize that by multiple cross validations
- prisonguard 1y agoChatGPT is actively being used as a calculator.
- game_the0ry 1y ago> I don't trust LLMs to do the kind of precise deterministic work you need in a spreadsheet. I was thinking along the same lines, but I could not articulate as well as you did. Spreadsheet work is deterministic; LLM output is probabilistic. The two should be distinguished. Still, its a productivity boost, which is always good.
- Kiro 1y agoMost real-world spreadsheets I've worked with were fragile and sloppy, not precise and deterministic. Programmers always get shocked when they realize how many important things are built on extremely messy spreadsheets, and that people simply accept it. They rather just spend human hours correcting discrepancies than trying to build something maintainable.
- bonoboTP 1y agoUsually this is very hard because the tasks and the job often subtly shifts in somewhat unpredictable and unforeseen ways and there is no neat clean abstraction that you can just implement as an application. Too hererogeneous, too messy, too many exceptions. If you develop some clean elegant solution, next week there will be something that your shiny app doesn't allow and they'd have to submit a feature request or whatever. In Excel, it's possible to just ad hoc adjust things and make it up as you go. It's not clean but very adaptable and flexible.
- MangoCoffee 1y agoLLMs are just a tool, though. Humans still have to verify them, like with very other tools out there
- A4ET8a8uTh0_v2 1y agoEh, yes. In theory. In practice, and this is what I have experienced personally, bosses seem to think that you now have interns so you should be able to do 5x the output.. guess what that means. No verification or rubber stamp.
- brookst 1y agoDo you trust humans to be precise and deterministic, or even to be especially good at math? This is talking about applying LLMs to formula creation and references, which they are actually pretty good at. Definitely not about replacing the spreadsheet's calculation engine.
- amrocha 1y agoI trust humans to not be able to shoot the company on the foot without even realizing it. Why are we suddenly ok with giving every underpaid and exploited employee a foot gun and expect them to be responsible with it???
- brookst 1y agoMaybe my experience is unique, but I have seen so many security issues from underpaid and exploited employees being successfully phished, among many other abuses. If your experience with the lowest, most-abused employees is better than mine, I envy you.
- amrocha 1y agoNow imagine your accountant can ask AI to write a formula that calculates EBITDA so he can go back to gambling on basketball faster
- onion2k 1y agoIt's widely known LLMs are terrible at even basic maths. Claude for Excel isn't doing maths. It's doing Excel. If the llm is bad at maths then teaching it to use a tool that's good at maths seems sensible.