4 ms·

SWE-Bench Verified

46 points by meetpateltech 2y ago

viraptor 2y ago
I'm really glad this got created, but the examples are... terrible? How is the "kernS: 'kern' referenced before assignment" problem an example of a good/verified sample? The comment (if it is a comment?) is using // in python and I can't tell if it's an example of something failing, or a line from the error, or something else. I would hope we can concentrate on benchmarking on full, well structured sentences before we go into "my first hasty SO question" territory. Which don't get me wrong, is useful if you want to write very short prompts, but... we're not great with basics yet and it can be always refined to a quality prompt by the LLM in a separate step. It feels really weird to me to include that in the benchmark.
keyle 2y ago
[flagged]
- Daunk 2y ago
  Amen
- viraptor 2y ago
  I would welcome less typing, more decision making. I don't think that world is anywhere close (and I'm already using Aider and Plandex for development). But yeah, I believe in a couple of decades we're likely going to do this whole thing in a very different way.
  - ori_b 2y ago
    What makes you think that the decision making will be hard to automate? In a couple of decades, keeping this pace of development, doesn't it seem plausible that AI will be able to extract requirements from observed behavior, and decide how to fill them better than a human can?
    - viraptor 2y ago
      Because good decision making requires lots of external context and interpersonal assumptions. These things are not even documented. It's like every outsourcing project for code that I've seen: yes, what was delivered matched the written requirements, but it's useless and needs lots of changes. There's still going to be space for people who understand how things work both in code and in practice. (Cue Office Space "I deal with customers so engineers don't have to. I've got people skills!") There are also right now cases of: I can write a paragraph of what change I need, or change those two lines myself. I don't expect they will go away entirely and having an agent available will be closer to an extra skill than a full replacement.
      - ori_b 2y ago
        It seems like observing usage and setting up a feedback loop, then incrementally a/b testing, is certainly not something that can be done today, but doesn't seem beyond the realm of possibility a couple decades from now. Edit: In fact, now that I think about it, I'd probably be more ok dealing with a requirements document and testing plan generated by chatgpt today from a summary than I would be dealing with code -- hallucinations tend to have less impact at that stage. Requirements tend to be wrong anyways until they're implemented; code is where the rubber hits the road. The missing piece is having LLMs get better at integrating and reasoning about real world data, so that it can iterate on user testing. Heck, at that point, models can decide what to build, and not just how to build it; there's a treasure trove of data that they can access much faster than humans to find gaps in the market and generate solutions to fill those gaps. Assuming we don't hit a wall with model capabilities, we can focus purely on consuming the outputs of AI, as it optimizes itself to best keep us happy.
- shrubble 2y ago
  Imagine a SWE-AI being managed by story points and sprints which are also put in place by a Scrum-AI …
- jstummbillig 2y ago
  > Let's be honest, 24 years of this shit and I still can't write bug free software. Considering how important SWE is today, our particular mix of recklessness and incompetence with almost no regulatory oversight feels increasingly insane, compared to the hoops a physician has to jump through to get a license and the fairly small amount of damage they can then do (given the 1-on-1 nature of their work). I guess the remedy is that software is still better than no software, but yeah, we would have to be fairly delusional to not see that something could be a lot better here and we need all the help we can get while the stakes are getting much higher.
janice1999 2y ago
> 68.3% of SWE-bench samples being filtered out due to underspecification, unfair unit tests, or other issues. I have a feeling that the filtered samples better represent actual issues in the real world. How would "AI" even start with issues like "product sometimes runs slow" or "it crashes 1/20 times with this vague error message". I think my job is secure for the moment. I would love AI tooling to better sort and categorise issues though. In a rush to "replace" people I think AI companies are focusing on the wrong things.
- viraptor 2y ago
  > How would "AI" even start with issues like "product sometimes runs slow" or "it crashes 1/20 times with this vague error message". By asking more questions, just like a developer would. This is already available in the chat form, but that's a separate set of skills and would require the other party to give responses. It's a useful test, but I believe not in the context of what swe-bench wants to measure.
deleted 2y ago
[deleted]

46 points by meetpateltech 2y ago

I'm really glad this got created, but the examples are... terrible? How is the "kernS: 'kern' referenced before assignment" problem an example of a good/verified sample? The comment (if it is a comment?) is using // in python and I can't tell if it's an example of something failing, or a line from the error, or something else. I would hope we can concentrate on benchmarking on full, well structured sentences before we go into "my first hasty SO question" territory. Which don't get me wrong, is useful if you want to write very short prompts, but... we're not great with basics yet and it can be always refined to a quality prompt by the LLM in a separate step. It feels really weird to me to include that in the benchmark.

keyle 2y ago

[flagged]

Daunk 2y ago
Amen
viraptor 2y ago
I would welcome less typing, more decision making. I don't think that world is anywhere close (and I'm already using Aider and Plandex for development). But yeah, I believe in a couple of decades we're likely going to do this whole thing in a very different way.
- ori_b 2y ago
  What makes you think that the decision making will be hard to automate? In a couple of decades, keeping this pace of development, doesn't it seem plausible that AI will be able to extract requirements from observed behavior, and decide how to fill them better than a human can?
  - viraptor 2y ago
    Because good decision making requires lots of external context and interpersonal assumptions. These things are not even documented. It's like every outsourcing project for code that I've seen: yes, what was delivered matched the written requirements, but it's useless and needs lots of changes. There's still going to be space for people who understand how things work both in code and in practice. (Cue Office Space "I deal with customers so engineers don't have to. I've got people skills!") There are also right now cases of: I can write a paragraph of what change I need, or change those two lines myself. I don't expect they will go away entirely and having an agent available will be closer to an extra skill than a full replacement.
    - ori_b 2y ago
      It seems like observing usage and setting up a feedback loop, then incrementally a/b testing, is certainly not something that can be done today, but doesn't seem beyond the realm of possibility a couple decades from now. Edit: In fact, now that I think about it, I'd probably be more ok dealing with a requirements document and testing plan generated by chatgpt today from a summary than I would be dealing with code -- hallucinations tend to have less impact at that stage. Requirements tend to be wrong anyways until they're implemented; code is where the rubber hits the road. The missing piece is having LLMs get better at integrating and reasoning about real world data, so that it can iterate on user testing. Heck, at that point, models can decide what to build, and not just how to build it; there's a treasure trove of data that they can access much faster than humans to find gaps in the market and generate solutions to fill those gaps. Assuming we don't hit a wall with model capabilities, we can focus purely on consuming the outputs of AI, as it optimizes itself to best keep us happy.
shrubble 2y ago
Imagine a SWE-AI being managed by story points and sprints which are also put in place by a Scrum-AI …
jstummbillig 2y ago
> Let's be honest, 24 years of this shit and I still can't write bug free software. Considering how important SWE is today, our particular mix of recklessness and incompetence with almost no regulatory oversight feels increasingly insane, compared to the hoops a physician has to jump through to get a license and the fairly small amount of damage they can then do (given the 1-on-1 nature of their work). I guess the remedy is that software is still better than no software, but yeah, we would have to be fairly delusional to not see that something could be a lot better here and we need all the help we can get while the stakes are getting much higher.

janice1999 2y ago

> 68.3% of SWE-bench samples being filtered out due to underspecification, unfair unit tests, or other issues. I have a feeling that the filtered samples better represent actual issues in the real world. How would "AI" even start with issues like "product sometimes runs slow" or "it crashes 1/20 times with this vague error message". I think my job is secure for the moment. I would love AI tooling to better sort and categorise issues though. In a rush to "replace" people I think AI companies are focusing on the wrong things.

viraptor 2y ago
> How would "AI" even start with issues like "product sometimes runs slow" or "it crashes 1/20 times with this vague error message". By asking more questions, just like a developer would. This is already available in the chat form, but that's a separate set of skills and would require the other party to give responses. It's a useful test, but I believe not in the context of what swe-bench wants to measure.

deleted 2y ago

[deleted]