12 ms·
> There are certain tasks, like improving a given program for speed, for instance, where in theory the model can continue to make progress with a very clear rew
by piker 10mo ago
> There are certain tasks, like improving a given program for speed, for instance, where in theory the model can continue to make progress with a very clear reward signal for a very long time.
Super skeptical of this claim. Yes, if I have some toy poorly optimized python example or maybe a sorting algorithm in ASM, but this won’t work in any non-trivial case. My intuition is that the LLM will spin its wheels at a local minimum the performance of which is overdetermined by millions of black-box optimizations in the interpreter or compiler signal from which is not fed back to the LLM.
- dist-epoch 10mo agohttps://github.com/algorithmicsuperintelligence/openevolve https://github.com/algorithmicsuperintelligence/openevolve
- piker 10mo agohttps://chatgpt.com/backend-api/estuary/public_content/enc/eyJpZCI6Im1fNjk0Njg0MjkzNTEwODE5MWE2NzY5MmE4YWRjNTZiMTA6ZmlsZV8wMDAwMDAwMDljNzg3MWZkYTExODc2MDgxZDllYjAyOSIsInRzIjoiMjA0NDIiLCJwIjoicHlpIiwiY2lkIjoiMSIsInNpZyI6IjIxMDJlMDkzMGExNjNkYWY3OWI4ZTI4YmNhZDE5OThlNGFjYmQxNjQzNzQ2ODRiYmM3NDFlZmE1OGViMjQ5NzgiLCJ2IjoiMCIsImdpem1vX2lkIjpudWxsLCJjcyI6bnVsbCwiY3AiOm51bGwsIm1hIjpudWxsfQ== https://chatgpt.com/backend-api/estuary/public_content/enc/e...
- andy99 10mo agoThere was a discussion the other day where someone asked Claude to improve a code base 200x https://news.ycombinator.com/item?id=46197930 https://news.ycombinator.com/item?id=46197930
- exitb 10mo agoThat’s most definitely not the same thing, as „improving a codebase” is an open ended task with no reliable metrics the agent could work against.
- NitpickLawyer 10mo ago> but this won’t work in any non-trivial case Earlier this year google shared that one of their projects (I think it was alphaevolve) found an optimisation in their stack that sped up their real world training runs by 1%. As we're talking about google here, we can be pretty sure it wasn't some trivial python trick that they missed. Anyhow, at ~100M$ / training run, that's a 1M$ save right there. Each and every time they run a training run! And in the past month google also shared another "agentic" workflow where they had gemini2.5-fhash! (their previous gen "small" model) work autonomously on migrating codebases to support aarch64 architecture. There they found ~30% of the projects worked flawlessly end-to-end. Whatever costs they save from switching to ARM will translate in real-world $ saved (at google scale, those can add up quickly).
- piker 10mo agoThe second example has nothing to do with the first. I am optimistic that LLMs are great for translations with good testing frameworks. “Optimize” in a vacuum is a tarpit for an LLM agent today, in my view. The Google case is interesting but 1% while significant at Google scale doesn’t move the needle much in terms of statistical significance. It would be more interesting to see the exact operation and the speed up achieved relative to the prior version. But it’s data contrary to my view for sure. The cynic also notes that Google is in the LLM hype game now, too.
- NitpickLawyer 10mo agoWhy do you think it's not relevant to the "optimise in a loop" thing? The way I think of it, it's using LLMs "in a loop" to move something from arch A (that costs x$) to arch B (that costs y$), where y is cheaper than x. It's still an autonomous optimisation done by LLMs, no?
- piker 10mo agoDid the LLM suggest moving to the new architecture? If not that’s not what’s under discussion. That’s just following an order to translate.