4 ms·

> I helped one recently almost one-shot[3] converting a 30 sheet mind numbingly complicated Excel financial model to Python with Claude Code. I'm sure Claude C

> I helped one recently almost one-shot[3] converting a 30 sheet mind numbingly complicated Excel financial model to Python with Claude Code. I'm sure Claude Code will happily one-shot that conversion. It's also virtually guaranteed to have messed up vital parts of the original logic in the process.

linsomniac 9mo ago
It depends on how easily testable the Excel is. If Claude has the ability to run both the Excel and the Python with different inputs, and check the outputs, it's stunningly likely to be able to one-shot it.
- martinald 9mo ago
  That's exactly what it did (author here).
  - majormajor 9mo ago
    I'm having trouble reconciling "30 sheet mind numbingly complicated Excel financial model" and "Two or three prompts got it there, using plan mode to figure out the structure of the Excel sheet, then prompting to implement it. It even added unit tests to the Python model itself, which I was impressed with!" "1 or 2 plan mode prompts" to fully describe a 30-sheet complicated doc suggests a massively higher level of granularity than Opus initial plans on existing codebases give me or a less-than-expected level of Excel craziness. And the tooling harnesses have been telling the models to add testing to things they make for months now, so why's that impressive or suprising?
    - martinald 9mo ago
      No it didn't make a giant plan of every detail. It made a plan of the core concepts and then when it was in implementation mode it kept checking the excel file to get more info. It took around ~30 mins in implementation mode to build it. I was impressed because the prompt didn't ask it to do that. It doesn't normally add tests for me without asking, YMMV.
      - majormajor 9mo ago
        Ah, I see. Did it build a test suite for the Excel side? A fuzzer or such? It's the cross-concern interactions that still get me. 80% of what I think about these days when writing software is how to test more exhaustively without build times being absolute shit (and not necessarily actually being exhaustive anyway).
- AlotOfReading 9mo ago
  Something being simultaneously described as a "30 sheet, mind-numbingly complex Excel model" and "testable" seems somewhat unlikely, even before we get into whether Claude will be able to test such a thing before it runs into context length issues. I've seen Claude hallucinate running test suites before.
  - martinald 9mo ago
    It compacted at least twice but continued with no real issues. Anyway, please try it if you find it unbelievable. I didn't expect it to work FWIW like it did. Opus 4.5 is pretty amazing at long running tasks like this.
    - moregrist 9mo ago
      I think the skepticism here is that without tests or a _lot_ of manual QA how would you know that it did it correctly? Maybe you did one or the other , but “nearly one-shotted” doesn’t tend to mean that. Claude Code more than occasionally likes to make weird assumptions, and it’s well known that it hallucinates quite a bit more near the context length, and that compaction only partially helps this issue.
      - skybrian 9mo ago
        If you’re porting some formulas from one language to another, “correct” can be defined as “gets the same answers as before.” Assuming you can run both easily, this is easy to write a property test for. Sure, maybe that’s just building something that’s bug-for-bug compatible, but it’s something Claude can work with.
        gregoryl 9mo ago
        For starters, Python uses IEEE 754, and Excel uses IEEE 754 (with caveats). I wonder if that's being emulated.
    - stavros 9mo ago
      I generally agree with you, but I tried to get it to modernize a fairly old SaaS codebase, and it couldn't. It had all the code right there, all it had to do was change a few lines, upgrade a few libraries, etc, but it kept getting lots of things wrong. The HTML was wrong, the CSS was completely missing, basic views wouldn't work, things like that. I have no idea why it had so much trouble with this generally easy task. Bizarre.
- datsci_est_2015 9mo ago
  And also - who understands the system now? Does anyone know Python at this shop? Is it someone’s implicit duty to now learn Python, or is the LLM now the de facto interface for modifying the system? When shit hits the fan and execs need answers yesterday, will they jump to using the LLM to probabilistically make modifications to the system, or will they admit it was a mistake and pull Excel back up to deterministically make modifications the way they know how?
  - theshrike79 9mo ago
    At least you can literally buy books that tell you how to test Python code and confirm it works as expected. When shit hits the fan, execs need answers yesterday and the 30 sheet Excel monstrosity is producing the wrong numbers - who fixes it? It was done by Sue, who left the company 4 years ago, people have been using it since and nobody really understands it.
    - datsci_est_2015 9mo ago
      Alright so it is implicit that it is someone’s duty to learn Python. Don’t get me wrong, Python is better than Excel in a million ways, but not many companies are competent at hiring people who are competent at Python - including software companies from my personal experience hah. I’ve also heard plenty of horror stories of bus factor employees leaving (or threatening to leave) behind an excel monstrosity and companies losing 6 months of sales, so maybe there’s a win for AI somewhere in there.
- rk06 9mo ago
  where exactly have you seen excel forumalas to have tests? I have, in my early careers, gone knee deep into Excel macros and worked on c# automation that will create excel sheet run excel macros on it and then save it without the macros. in the entire process, I saw dozens of date time mistakes in VBA code, but no tests that would catch them...
- catlifeonmars 9mo ago
  You touched on Kolmogorov complexity there :)
- chrisjj 9mo ago
  Different inputs? With no understanding of the program, how is Claude going to determine what input set is sufficient. Tell me if I am wrong, but surely Claude cannot even access execution coverage.
Spivak 9mo ago
Doesn't it help you sleep at night that your 401k might be managed by analysts #yoloing their financial modeling tools with an LLM?
- DaedalusII 9mo ago
  having worked in large financial institutions, this would be a step improvement the largest independent derivatives broker in australia collapsed after it was discovered the board were using astrology and magicians to gamble with all the clients money https://www.abc.net.au/news/2016-09-16/stockbroker-used-psychic-to-make-budget-forecasts/7853646 https://www.abc.net.au/news/2016-09-16/stockbroker-used-psyc...
  - Nevermark 9mo ago
    Well that would do it. Astrology and magic stop working once they are scrutinized. That is their only weakness.
  - leptons 9mo ago
    It sounds like a step sideways, not a step up. LLMs are akin to a Ouija board.
- theshrike79 9mo ago
  I have seen Excel used in places that manage millions of euros of other people's money. I have seen Excel used for financial planning I have seen Excel used for managing people's health data. I have BUILT a test suite for a government offical use communication device - inside Excel. The original was a mish-mash of Excel formulas and VBA. I improved the VBA part of it by adding a web cam to the mix. I don't sleep well at night knowing how many very very essential things are running on top of Excel sheets passed down like stories around a campfire.
chrisjj 9mo ago
To be fair, he said almost one-shot. It's like a CPU that's almost 100% reliable... in that it fails only once every 1 million clock cycles.
- Yossarrian22 9mo ago
  For a 4 GHz CPU that’s 4 failures a second..
  - fcantournet 9mo ago
    4 thousand :)
  - chrisjj 9mo ago
    My point entirely.
djeastm 9mo ago
I doubt that 30-sheet Excel model itself was validated by the most rigorous of standards. Unless it had its own test suite before the one-shot (which could be leveraged by the AI), people have probably been taking its outputs for granted for years.

linsomniac 9mo ago

It depends on how easily testable the Excel is. If Claude has the ability to run both the Excel and the Python with different inputs, and check the outputs, it's stunningly likely to be able to one-shot it.

martinald 9mo ago
That's exactly what it did (author here).
- majormajor 9mo ago
  I'm having trouble reconciling "30 sheet mind numbingly complicated Excel financial model" and "Two or three prompts got it there, using plan mode to figure out the structure of the Excel sheet, then prompting to implement it. It even added unit tests to the Python model itself, which I was impressed with!" "1 or 2 plan mode prompts" to fully describe a 30-sheet complicated doc suggests a massively higher level of granularity than Opus initial plans on existing codebases give me or a less-than-expected level of Excel craziness. And the tooling harnesses have been telling the models to add testing to things they make for months now, so why's that impressive or suprising?
  - martinald 9mo ago
    No it didn't make a giant plan of every detail. It made a plan of the core concepts and then when it was in implementation mode it kept checking the excel file to get more info. It took around ~30 mins in implementation mode to build it. I was impressed because the prompt didn't ask it to do that. It doesn't normally add tests for me without asking, YMMV.
    - majormajor 9mo ago
      Ah, I see. Did it build a test suite for the Excel side? A fuzzer or such? It's the cross-concern interactions that still get me. 80% of what I think about these days when writing software is how to test more exhaustively without build times being absolute shit (and not necessarily actually being exhaustive anyway).
AlotOfReading 9mo ago
Something being simultaneously described as a "30 sheet, mind-numbingly complex Excel model" and "testable" seems somewhat unlikely, even before we get into whether Claude will be able to test such a thing before it runs into context length issues. I've seen Claude hallucinate running test suites before.
- martinald 9mo ago
  It compacted at least twice but continued with no real issues. Anyway, please try it if you find it unbelievable. I didn't expect it to work FWIW like it did. Opus 4.5 is pretty amazing at long running tasks like this.
  - moregrist 9mo ago
    I think the skepticism here is that without tests or a _lot_ of manual QA how would you know that it did it correctly? Maybe you did one or the other , but “nearly one-shotted” doesn’t tend to mean that. Claude Code more than occasionally likes to make weird assumptions, and it’s well known that it hallucinates quite a bit more near the context length, and that compaction only partially helps this issue.
    - skybrian 9mo ago
      If you’re porting some formulas from one language to another, “correct” can be defined as “gets the same answers as before.” Assuming you can run both easily, this is easy to write a property test for. Sure, maybe that’s just building something that’s bug-for-bug compatible, but it’s something Claude can work with.
      - gregoryl 9mo ago
        For starters, Python uses IEEE 754, and Excel uses IEEE 754 (with caveats). I wonder if that's being emulated.
  - stavros 9mo ago
    I generally agree with you, but I tried to get it to modernize a fairly old SaaS codebase, and it couldn't. It had all the code right there, all it had to do was change a few lines, upgrade a few libraries, etc, but it kept getting lots of things wrong. The HTML was wrong, the CSS was completely missing, basic views wouldn't work, things like that. I have no idea why it had so much trouble with this generally easy task. Bizarre.
datsci_est_2015 9mo ago
And also - who understands the system now? Does anyone know Python at this shop? Is it someone’s implicit duty to now learn Python, or is the LLM now the de facto interface for modifying the system? When shit hits the fan and execs need answers yesterday, will they jump to using the LLM to probabilistically make modifications to the system, or will they admit it was a mistake and pull Excel back up to deterministically make modifications the way they know how?
- theshrike79 9mo ago
  At least you can literally buy books that tell you how to test Python code and confirm it works as expected. When shit hits the fan, execs need answers yesterday and the 30 sheet Excel monstrosity is producing the wrong numbers - who fixes it? It was done by Sue, who left the company 4 years ago, people have been using it since and nobody really understands it.
  - datsci_est_2015 9mo ago
    Alright so it is implicit that it is someone’s duty to learn Python. Don’t get me wrong, Python is better than Excel in a million ways, but not many companies are competent at hiring people who are competent at Python - including software companies from my personal experience hah. I’ve also heard plenty of horror stories of bus factor employees leaving (or threatening to leave) behind an excel monstrosity and companies losing 6 months of sales, so maybe there’s a win for AI somewhere in there.
rk06 9mo ago
where exactly have you seen excel forumalas to have tests? I have, in my early careers, gone knee deep into Excel macros and worked on c# automation that will create excel sheet run excel macros on it and then save it without the macros. in the entire process, I saw dozens of date time mistakes in VBA code, but no tests that would catch them...
catlifeonmars 9mo ago
You touched on Kolmogorov complexity there :)
chrisjj 9mo ago
Different inputs? With no understanding of the program, how is Claude going to determine what input set is sufficient. Tell me if I am wrong, but surely Claude cannot even access execution coverage.

Spivak 9mo ago

Doesn't it help you sleep at night that your 401k might be managed by analysts #yoloing their financial modeling tools with an LLM?

DaedalusII 9mo ago
having worked in large financial institutions, this would be a step improvement the largest independent derivatives broker in australia collapsed after it was discovered the board were using astrology and magicians to gamble with all the clients money https://www.abc.net.au/news/2016-09-16/stockbroker-used-psychic-to-make-budget-forecasts/7853646 https://www.abc.net.au/news/2016-09-16/stockbroker-used-psyc...
- Nevermark 9mo ago
  Well that would do it. Astrology and magic stop working once they are scrutinized. That is their only weakness.
- leptons 9mo ago
  It sounds like a step sideways, not a step up. LLMs are akin to a Ouija board.
theshrike79 9mo ago
I have seen Excel used in places that manage millions of euros of other people's money. I have seen Excel used for financial planning I have seen Excel used for managing people's health data. I have BUILT a test suite for a government offical use communication device - inside Excel. The original was a mish-mash of Excel formulas and VBA. I improved the VBA part of it by adding a web cam to the mix. I don't sleep well at night knowing how many very very essential things are running on top of Excel sheets passed down like stories around a campfire.

chrisjj 9mo ago

To be fair, he said almost one-shot. It's like a CPU that's almost 100% reliable... in that it fails only once every 1 million clock cycles.

Yossarrian22 9mo ago
For a 4 GHz CPU that’s 4 failures a second..
- fcantournet 9mo ago
  4 thousand :)
- chrisjj 9mo ago
  My point entirely.

djeastm 9mo ago

I doubt that 30-sheet Excel model itself was validated by the most rigorous of standards. Unless it had its own test suite before the one-shot (which could be leveraged by the AI), people have probably been taking its outputs for granted for years.