3 ms·
On Friday I was converting a constrained solver from python to another language, and ran into some difficulty with subsituting an optimzer that's a few lines of
by kregasaurusrex 1y ago
On Friday I was converting a constrained solver from python to another language, and ran into some difficulty with subsituting an optimzer that's a few lines of easily written Scipy; but barely being supported in another language. One AI tool found this out and fully re-implemented the solver using a custom linear algebra library it wrote from scratch. But another AI tool was really struggling with getting the right syntax to be compatible with the common existing optimization libaries, and I felt like I was repeatedly putting queries (read: $) into the software equivalent of a slot machine that was constantly apologizing for not giving a testable answer while eating tens of dollars in direct costs waiting for the "jackpot" of working code.
The feedback loop of "maybe the next time it'll be right" turned into a few hundred queries resulting in finding the LLM's attempts were a ~20 node cycle of things it tried and didn't work, and now you're out a couple dollars and hours of engineering time.
- brookst 1y agoA very relatable experience. But not all that different from how humans work when in unfamiliar domains.
- moregrist 1y ago> One AI tool found this out and fully re-implemented the solver using a custom linear algebra library it wrote from scratch. So slow, untested, and likely buggy, especially as the inputs become less well-conditioned? If this was a jr dev writing code I’d ask why they didn’t use <insert language-relevant LAPACK equivalent>. Neither llm outcome seems very ideal to me, tbh.
- theshrike79 1y agoWith mathematical things you can always write comprehensive and complete unit tests to check the AIs work. TDD (and exhaustive unit tests in general) are a good idea with LLMs anyway. Just either tell it not to touch test, or in Claude's case you can use Hooks to _actually_ prevent it from editing any test file. Then shove it at the problem and it'll iterate a solution until the tests pass. It's like the Excel formula solver, but for code :D
- th0ma5 1y agoI think we all understand this we just don't think it works.
- quatonion 1y agoI'm curious why you think it doesn't work, when there are plenty of people saying it does. There are limitations at the moment, and I don't see many people disputing that, but it must be doing something right, and its abilities are improving every day. It's learning. Sometimes I get the feeling a lot of antis painted themselves into a corner early on, and will die on this hill despite constant improvements in the technology. I have seen similar things many times in my career. There was a time when everyone were very skeptical of high level languages, writing everything in assembler come hell or high water, for example. At some point it is going to single shot an entire OS or refactor a multi-million line codebase. Will that be enough to convince you? From my perspective I like to be prepared, so I'm doing what I have always done.. understand and gain experience with these new tools. I much prefer that than missing the boat. And, it's quite fun and better than you might imagine as long as you put a bit of effort in.
- quantumHazer 1y ago> From my perspective I like to be prepared The same you that thinks has proved P = NP with ChatGPT?
- moregrist 1y ago