4 ms·
I find it interesting that the folks at Qodo are not clear on why more groups try to push the needle on SWE-Bench rather than on Codeforces. I think most softw
by timabdulla 2y ago
I find it interesting that the folks at Qodo are not clear on why more groups try to push the needle on SWE-Bench rather than on Codeforces.
I think most software developers would find that performance on SWE-Bench (wherein an AI system has to resolve a real-world Github issue) is much more relevant to their day-to-day than the raw algorithmic problem-solving capabilities of an AI system.
Competitive programming questions are challenging, obviously, but they are tightly defined with prescribed inputs and outputs and the solutions are usually compact, though hard to arrive at.
In contrast, real-world programming is much fuzzier and more ambiguous. The solutions that one must implement are usually much larger in scope and require much greater project-specific context to build. And of course real-world tasks are generally not things that demand lots of formal data structures and algorithms knowledge, like one would need to do well at Codeforces.
What's interesting is that you can have systems like o1 or AlphaCodium that are much, much better than the median software developer at solving tricky algorithmic puzzles, but that also can't do very well at SWE-Bench, which largely comprises Github issues that have been estimated to take <1h for a human developer to do. Even though o1 would absolutely dominate Claude 3.5 Sonnet at competitive programming questions, it seems to basically be a wash on real-world tasks.
So I suppose my question to Qodo would be: Why the emphasis on Codeforces as a benchmark? It's quite clear -- and has been for some time, since e.g. AlphaCode in 2022 -- that AI systems can be really powerful when it comes to solving Codeforces-style questions. What seems much harder (for now) is making significant progress on real-world development tasks.
- akrlt 2y agoI think LLMs aren't better than the median software developer at LeetCode. They simply have a compressed database of the (stolen) answers. If any software developer had access to Google in an interview, he could "solve" all of the answers instantly.
- henry2023 2y agoExactly what I’ve found. Try giving an LLM any novel easy problem (IMO 2024 Problem 5 is a good example). And absolutely every llm out there fails miserably.
- Cyph0n 2y agohttps://deepmind.google/discover/blog/ai-solves-imo-problems-at-silver-medal-level/ https://deepmind.google/discover/blog/ai-solves-imo-problems...
- fallingsquirrel 2y agoIf I'm reading this right, these are narrowly-applicable purpose-built systems, not general-purpose LLMs. > AlphaProof, a new reinforcement-learning based system for formal math reasoning > AlphaGeometry 2, an improved version of our geometry-solving system
- TacticalCoder 2y agoThe problem is if a problem is formulated like this: "There are 7 pirates on a ship and 61 cannon balls, ..." (doesn't matter what the problem is: say the solution involves some dynamic programming algorithm) and that the LLM then finds a problem starting with: "There are 7 cats in a playground and 61 plushies, ..." (insert basically the same problem, requiring the same solution) Well... Then the LLM shall be able to solve it. And many people will consider it's a novel problem and hence a resounding success. I mean: it is a success, but not anywhere near as impressive as most think.
- magicalhippo 2y agoI had the same with the strawberries test. For the ones that supposedly gave the right answer, I tried again with "strawerberry" and they promptly failed miserably. Now, given the token encoding I think naive letter counting is not something we should expect from LLMs, but still serves as a nice reminder to actually ensure the test/validation data is not part of the training data.
- rmbyrro 2y ago> he could "solve" all of the answers instantly Set an experiment selecting 100 random software developers around the world and test this hypothesis. You're up to be surprised. Nevertheless, most developers who wouldn't be able to "solve" a LeetCode challenge in a couple of hours even with access to Google, I bet would perform much better than o1 on real-world Github issues in their technical domain. This highlights the essence behind OP's question about why focusing on Codeforces. And it shows me that "intelligence" involves a dimension that isn't logical and we don't understand yet.
- gus_massa 2y agoIn competitive programing, it's clear what is a good and a bad splution, and they usualy have a set of test so people can't claim they cheated. With real world questions, everyone can have a different opinion. Let's say you must clasify photos as dogs, cats or other. What about an apple tree? A hyena? An AI generated version of a dog with cat fur? A lion? Hello Kitty?
- ianbutler 2y agoSWE bench has gold code patches and the passing test suite patch for after the github issue was completed. While you may argue over the style of the code produced by the model there is a known good passing state for the model to achieve. For now that's the closest representation of a real world problem solve in a controlled repeatable benchmark we have.