5 ms·

This shouldn't be ignored in the discussion here: The job performed by the humans was broader than what was requested of the model in this benchmark: humans

This shouldn't be ignored in the discussion here: The job performed by the humans was broader than what was requested of the model in this benchmark: humans also had to find the relevant invoices (searching through mailboxes, or requesting them from providers) and reason through any circumstances which cannot be inferred from the bank feed and invoices/receipts on their own. In the benchmark these circumstances are presented to the model as “user notes." This is precisely the kind of fine print on white-collar AI capability that companies keep running into: pretty much any non-entry office job worth having involves a lot of undocumented (even undocumentable) problems requiring judgment and experience. And I would be pretty nervous about asking any of the frontier LLMs to retrieve invoices: "cool, Claude logged that it found the May 6th bill from the paper supplier, I am sure it didn't just make something up arbitrary, then compound on the error by agentically iterating over the made-up invoice lurking in its reasoning traces. I checked the first 30 times and there were no problems!"

adamkurkiewicz 3mo ago
Hey, the author of the benchmark here. The benchmark data was prepared in April 2026 (when I was manually doing our VAT return with my co-founder). The invoices were indeed found manually. Currently we're using a custom "invoice searcher" built on Kimi 2.6 (in our testing several weeks ago it outperformed Opus 4.7; it was just more persistent). Ultimately, I still verify everything manually after the model is finished fetching invoices for the month -- but it's a great help to have all the invoices already found (usually correctly).
- breadislove 3mo ago
  adam, i'd like to get in touch and would love to run the benachmark with mixedbread as a search backend. we are doing this right now with a lot of compliance companies. would be very curious how it improves quality/cost e2e
  - adamkurkiewicz 3mo ago
    Sure, I'm at adam@vineyard-finance.com
CodingJeebus 3mo ago
> And I would be pretty nervous about asking any of the frontier LLMs to retrieve invoices: I watched an accountant YouTuber reviewing a new AI-driven personal finance app the other day (I really need to touch grass), and it started out just fine. He had seeded the account with a bunch of his data and was able to ask questions about which categories had the most spend, etc. About half a dozen questions in, he asked it to calculate a certain segment of his spend (and being an accountant, he had his numbers memorized), and he immediately got back a calculation that he did not expect. So he asked for an itemized response and it hallucinated line items that never appeared in his account data, which he pointed out to viewers. He followed up with the chatbot with "where did line item X come from?" and the bot acknowledged that it wasn't legit. He immediately noped out after that, and who could blame him?
sublinear 3mo ago
AI helps automate things that didn't already have rigorous formatting and structures available as input... and that's really all it does (99% of the time). Doesn't matter how many more nines you add, rigorous formatting is still required. In some cases, it has teeth with compliance standards. Those standards cannot be compromised because there are already a lot of other layers contributing inaccuracy. It all adds up. In most situations, you could just hire a junior dev (or an intern! remember those?) write some CSV scripts and call it a day. Cheaper and auditable too. Those scripts can't change anyway until standards are revised. I'm still not seeing the benefit outside of solopreneur efforts and shady businesses wanting to launder blame.
- paytonjjones 3mo ago
  > doesn't matter how many more nines you add I don't get this argument. People say the same about autonomous driving. But humans also have some number of nines. If you can get it better than humans, that's better!
  - sublinear 3mo ago
    Manual data entry and other tedious chores are definitely unreliable. However, running a script that a human wrote according to committee specs is the most reliable part. You're conflating the different aspects of human work. We are much better at understanding our needs and arguing about them than doing the manual part. So, I don't get your argument either. I hear yours often enough and so much louder that I feel it's a deliberate muddying of waters. What cannot be obsoleted by automation becomes bureaucracy. To my ears, it sounds like you're afraid of ending the tech wild west. That bureaucracy was always the most valuable part, and the demand for experienced programmers over at that table is very high.
walrus01 3mo ago
If and when a large number of companies blindly turn over their accounts payable workflow to some AI agent system, it'll be very interesting to see the "social engineer the LLM" methods that fraud people use to get money sent to them. Basically the same idea as the ancient "send a fax with a bill for an unsolicited delivery of copier toner to 30,000 businesses" but taken into the modern era. edit: There's already a number of LLM which are intended for outgoing data loss protection to redact or prevent PII from escaping. Is anyone specifically working on a training set and agent that is specialized in reviewing "is this legit to pay", as a sub-task or filtering step in an AP workflow? I suppose it's a GIGO problem, as it would work best only if you have suppliers enrolled in some kind of existing db, with a specific contracted format for invoices, and correlating with project numbers/cost codes.
- mediaman 3mo ago
  You can fix this simply by using normal controls. That's why we have purchase orders that can only be entered by buyers. Product is received and approved by buyer. Invoice goes to accounting, who can't approve it unless there's a matching purchase order and receiver. Yes, letting agents do whatever they want leads to disaster. But humans are gullible stochastic token generators as well. And that's why the problem is already solved.
  - walrus01 3mo ago
    Indeed so, a fairly mundane RFP, RFQ, buyer, receiver, accounts payable process will stop a lot of problems. If an agent is inserted at some stage in the process with a clear path to make a ticket/escalate to a human if it sees something it doesn't understand, the risk isn't absurdly high, in my opinion. I've seen so many reports of humans with the authority/ability to execute an outgoing SWIFT transfer who've been social engineered into sending money to fraudsters... Or even just the basic low level "Hey I'm your boss sending you an SMS, please go buy some gift cards and scratch them off and send me the codes". No AI involved whatsoever. The danger exists where some true believer AI evangelist type of management person tries to fully automate the entire purchasing and AP workflow, which I'm sure some people will attempt soon, with varying degrees of success.
ofjcihen 3mo ago
Hahaha non-deterministic accounting probably won’t fly well with the IRS
- frereubu 3mo ago
  That made me laugh, thanks. I remember talking to my accountant in the UK a long time ago when I was newly self-employed, asking if I could pass something off as a business expense that was sort-of-related, but I knew probably not really OK. Her reply has stuck with me ever since: "HMRC [the UK equivalent of the IRS] are interested in matters of fact, not interpretation."
  - jimnotgym 3mo ago
    As an accountant at a large Corp, I can tell you that there are cases daily where two senior accountants argue over how to interpret accounting and tax rules, and often each agrees that the others interpretation is valid. I regularly see rulings from area specialists, and have to challenge them, only to be told, 'well in that case, you would be right' I sometimes ask an AI to comment, and usually their answer is couched is ifs and maybes, just like the humans. I often consult with auditors who tell me x is wrong, and they go away agreeing that y and z would be valid too, and x is also fine. There are often second order effects that need to be thought through. Good luck AI
    - ofjcihen 3mo ago
      This is the problem with the majority of AI push coming from devs. They look at something as complex as accounting and only think in terms of what they see on the surface and then say “oh that looks easy enough, there couldn’t possibly be more under the hood”.
    - frereubu 3mo ago
      Yeah, my query was an open-and-shut case, I know there are grey areas too. The Arctic Systems case in the UK was an interesting one for IT contractors.
- 3mo ago
Calazon 3mo ago
My wife (head of accounting for a small business) has been working on automating large parts of her job using AI. It's not completely reliable and the human cannot be taken out of the loop, but the number of menial tasks she's been able to automate has been really cool. A lot of processing data that arrives in non-standard formats, generating documents based on that data, etc. She still has to review everything, but her workload is way down, and when her assistant quit she automated away his whole position.
- alexjimenez99 3mo ago
  Curious what she automated? Invoices/receipt document processing been a round for a while. Did she go beyond that
  - PebblesRox 3mo ago
    Wife here. Pre-agentic coding I had automated a bunch of low-hanging fruit via low code tools (mainly Pipedream and Airtable) but I never had enough time to do everything that I wanted to. Lately I've really been getting into agentic coding (first with Antigravity and now with Claude Code) and I'm really excited about what I'm now able to accomplish on a very small budget of both money and time! It's been a big boon to my ability to automate things that really ought to be the computer's job. Plus with LLMs, the scope of what is properly the computer's job has increased by a lot! I've created a couple of Pipedream workflows that send freight bills and customer POs to an LLM to extract data. Now I just have to review and occasionally make a few tweaks vs. having to enter everything by hand. Since building those, my main focus has been on building better tools for myself and the team, such as a little web-app for sales reps to create and send their own purchase orders vs. having to go through accounting (aka me). The nice thing about vibe coding vs. hooking together a bunch of low-code tools is that I can customize the UI to make it exactly what we need. My current project is building a back-office app to help streamline the process of managing incoming shipping paperwork. This is a much fuzzier problem than regular invoice processing because it's a bunch of different types of documents (BOLs, packing lists, scale tickets) and a lot of judgment calls to make. I don't expect to hand it all off to the LLM but I plan to have the LLM do the first pass of transcribing the data. Then I'll make the human-review portion as ergonomic as possible for myself. I have a first draft up-and-running but the default interface that Claude Code came up with is not very good. I'm currently discussing the UI design with Claude in chat mode, narrating my manual process as I do it to help flesh out all the details of all the different sources of info and what decisions need to be made.[0] Once we're done I'll ask for a spec that I can take back to Claude Code for implementation. [0](https://x.com/CBancos/status/2077793484115755282 https://x.com/CBancos/status/2077793484115755282)
elevate_ 3mo ago
[flagged]
Haven880 3mo ago
AI is a glorified calculator in this case. It frees up human to do other things. It shouldn't be viewed as human replacer.
- cudgy 3mo ago
  Even when humans used to do the calculating?

adamkurkiewicz 3mo ago

Hey, the author of the benchmark here. The benchmark data was prepared in April 2026 (when I was manually doing our VAT return with my co-founder). The invoices were indeed found manually. Currently we're using a custom "invoice searcher" built on Kimi 2.6 (in our testing several weeks ago it outperformed Opus 4.7; it was just more persistent). Ultimately, I still verify everything manually after the model is finished fetching invoices for the month -- but it's a great help to have all the invoices already found (usually correctly).

breadislove 3mo ago
adam, i'd like to get in touch and would love to run the benachmark with mixedbread as a search backend. we are doing this right now with a lot of compliance companies. would be very curious how it improves quality/cost e2e
- adamkurkiewicz 3mo ago
  Sure, I'm at adam@vineyard-finance.com

CodingJeebus 3mo ago

> And I would be pretty nervous about asking any of the frontier LLMs to retrieve invoices: I watched an accountant YouTuber reviewing a new AI-driven personal finance app the other day (I really need to touch grass), and it started out just fine. He had seeded the account with a bunch of his data and was able to ask questions about which categories had the most spend, etc. About half a dozen questions in, he asked it to calculate a certain segment of his spend (and being an accountant, he had his numbers memorized), and he immediately got back a calculation that he did not expect. So he asked for an itemized response and it hallucinated line items that never appeared in his account data, which he pointed out to viewers. He followed up with the chatbot with "where did line item X come from?" and the bot acknowledged that it wasn't legit. He immediately noped out after that, and who could blame him?

sublinear 3mo ago

AI helps automate things that didn't already have rigorous formatting and structures available as input... and that's really all it does (99% of the time). Doesn't matter how many more nines you add, rigorous formatting is still required. In some cases, it has teeth with compliance standards. Those standards cannot be compromised because there are already a lot of other layers contributing inaccuracy. It all adds up. In most situations, you could just hire a junior dev (or an intern! remember those?) write some CSV scripts and call it a day. Cheaper and auditable too. Those scripts can't change anyway until standards are revised. I'm still not seeing the benefit outside of solopreneur efforts and shady businesses wanting to launder blame.

paytonjjones 3mo ago
> doesn't matter how many more nines you add I don't get this argument. People say the same about autonomous driving. But humans also have some number of nines. If you can get it better than humans, that's better!
- sublinear 3mo ago
  Manual data entry and other tedious chores are definitely unreliable. However, running a script that a human wrote according to committee specs is the most reliable part. You're conflating the different aspects of human work. We are much better at understanding our needs and arguing about them than doing the manual part. So, I don't get your argument either. I hear yours often enough and so much louder that I feel it's a deliberate muddying of waters. What cannot be obsoleted by automation becomes bureaucracy. To my ears, it sounds like you're afraid of ending the tech wild west. That bureaucracy was always the most valuable part, and the demand for experienced programmers over at that table is very high.

walrus01 3mo ago

If and when a large number of companies blindly turn over their accounts payable workflow to some AI agent system, it'll be very interesting to see the "social engineer the LLM" methods that fraud people use to get money sent to them. Basically the same idea as the ancient "send a fax with a bill for an unsolicited delivery of copier toner to 30,000 businesses" but taken into the modern era. edit: There's already a number of LLM which are intended for outgoing data loss protection to redact or prevent PII from escaping. Is anyone specifically working on a training set and agent that is specialized in reviewing "is this legit to pay", as a sub-task or filtering step in an AP workflow? I suppose it's a GIGO problem, as it would work best only if you have suppliers enrolled in some kind of existing db, with a specific contracted format for invoices, and correlating with project numbers/cost codes.

mediaman 3mo ago
You can fix this simply by using normal controls. That's why we have purchase orders that can only be entered by buyers. Product is received and approved by buyer. Invoice goes to accounting, who can't approve it unless there's a matching purchase order and receiver. Yes, letting agents do whatever they want leads to disaster. But humans are gullible stochastic token generators as well. And that's why the problem is already solved.
- walrus01 3mo ago
  Indeed so, a fairly mundane RFP, RFQ, buyer, receiver, accounts payable process will stop a lot of problems. If an agent is inserted at some stage in the process with a clear path to make a ticket/escalate to a human if it sees something it doesn't understand, the risk isn't absurdly high, in my opinion. I've seen so many reports of humans with the authority/ability to execute an outgoing SWIFT transfer who've been social engineered into sending money to fraudsters... Or even just the basic low level "Hey I'm your boss sending you an SMS, please go buy some gift cards and scratch them off and send me the codes". No AI involved whatsoever. The danger exists where some true believer AI evangelist type of management person tries to fully automate the entire purchasing and AP workflow, which I'm sure some people will attempt soon, with varying degrees of success.

ofjcihen 3mo ago

Hahaha non-deterministic accounting probably won’t fly well with the IRS

frereubu 3mo ago
That made me laugh, thanks. I remember talking to my accountant in the UK a long time ago when I was newly self-employed, asking if I could pass something off as a business expense that was sort-of-related, but I knew probably not really OK. Her reply has stuck with me ever since: "HMRC [the UK equivalent of the IRS] are interested in matters of fact, not interpretation."
- jimnotgym 3mo ago
  As an accountant at a large Corp, I can tell you that there are cases daily where two senior accountants argue over how to interpret accounting and tax rules, and often each agrees that the others interpretation is valid. I regularly see rulings from area specialists, and have to challenge them, only to be told, 'well in that case, you would be right' I sometimes ask an AI to comment, and usually their answer is couched is ifs and maybes, just like the humans. I often consult with auditors who tell me x is wrong, and they go away agreeing that y and z would be valid too, and x is also fine. There are often second order effects that need to be thought through. Good luck AI
  - ofjcihen 3mo ago
    This is the problem with the majority of AI push coming from devs. They look at something as complex as accounting and only think in terms of what they see on the surface and then say “oh that looks easy enough, there couldn’t possibly be more under the hood”.
  - frereubu 3mo ago
    Yeah, my query was an open-and-shut case, I know there are grey areas too. The Arctic Systems case in the UK was an interesting one for IT contractors.
3mo ago

Calazon 3mo ago

My wife (head of accounting for a small business) has been working on automating large parts of her job using AI. It's not completely reliable and the human cannot be taken out of the loop, but the number of menial tasks she's been able to automate has been really cool. A lot of processing data that arrives in non-standard formats, generating documents based on that data, etc. She still has to review everything, but her workload is way down, and when her assistant quit she automated away his whole position.

alexjimenez99 3mo ago
Curious what she automated? Invoices/receipt document processing been a round for a while. Did she go beyond that
- PebblesRox 3mo ago
  Wife here. Pre-agentic coding I had automated a bunch of low-hanging fruit via low code tools (mainly Pipedream and Airtable) but I never had enough time to do everything that I wanted to. Lately I've really been getting into agentic coding (first with Antigravity and now with Claude Code) and I'm really excited about what I'm now able to accomplish on a very small budget of both money and time! It's been a big boon to my ability to automate things that really ought to be the computer's job. Plus with LLMs, the scope of what is properly the computer's job has increased by a lot! I've created a couple of Pipedream workflows that send freight bills and customer POs to an LLM to extract data. Now I just have to review and occasionally make a few tweaks vs. having to enter everything by hand. Since building those, my main focus has been on building better tools for myself and the team, such as a little web-app for sales reps to create and send their own purchase orders vs. having to go through accounting (aka me). The nice thing about vibe coding vs. hooking together a bunch of low-code tools is that I can customize the UI to make it exactly what we need. My current project is building a back-office app to help streamline the process of managing incoming shipping paperwork. This is a much fuzzier problem than regular invoice processing because it's a bunch of different types of documents (BOLs, packing lists, scale tickets) and a lot of judgment calls to make. I don't expect to hand it all off to the LLM but I plan to have the LLM do the first pass of transcribing the data. Then I'll make the human-review portion as ergonomic as possible for myself. I have a first draft up-and-running but the default interface that Claude Code came up with is not very good. I'm currently discussing the UI design with Claude in chat mode, narrating my manual process as I do it to help flesh out all the details of all the different sources of info and what decisions need to be made.[0] Once we're done I'll ask for a spec that I can take back to Claude Code for implementation. [0](https://x.com/CBancos/status/2077793484115755282 https://x.com/CBancos/status/2077793484115755282)

elevate_ 3mo ago

[flagged]

Haven880 3mo ago

AI is a glorified calculator in this case. It frees up human to do other things. It shouldn't be viewed as human replacer.

cudgy 3mo ago
Even when humans used to do the calculating?