3 ms·
Maybe I'm a control freak, but asking agents to one-shot random apps is nothing like how I actually use AI in software engineering.
by platinumrad 3mo ago
Maybe I'm a control freak, but asking agents to one-shot random apps is nothing like how I actually use AI in software engineering.
- bhu8 3mo agoAbsolutely yes, but that's how you become twitter/X famous
- fragmede 3mo agoIt's not, but it's trying to bring any level of objective measure in this realm, vs just going off of vibes.
- thsbrown 3mo agoMan I do ponder this all the time.
- mikeocool 3mo agoYeah, the models have all been really good at generating greenfield apps for a really long time (in the scope of LLM time). I suppose it’s interesting to see how they make better greenfield apps. But I am much more interested in how they solve hard problems in existing gnarly codebases.
- ValentineC 3mo agoOne-shot benchmarks are great for me as a solo creator, since they slightly correlate to whether the better frontier models (Opus and Fable for me) make better decisions about things I didn't spec, or whether they'll give me better suggestions right off the bat.
- anuramat 3mo agoI imagine one could one-shot a basic app and then feed feature requests one by one, sounds like an obvious way to benchmark architecture/maintainability
- anuramat 3mo agojust found a decent looking benchmark for iterative development: https://swe-milestone.com/ https://swe-milestone.com/ surprised it isn't a bigger thing, eg artificial analysis doesn't report anything like that still doesn't measure the human-agent interaction part, but that's pure vibes atp