7 ms·

What makes you say it's not representative?

• arp242 2y ago
  Look at the data. Does that seem like the average bug report to you?
  • falcor84 2y ago
    It would help if you were to provide a specific example or two
    • arp242 2y ago
      You can't demonstrate whether a dataset is representative or not by "an example or two". You need to look at all the data. And all of this is fine. It's just a benchmark suit and doesn't need to be fully representative. The dataset itself doesn't even claim to be that as far as I can find. All I'm saying that the title wasn't really accurate.
• skywhopper 2y ago
  SWE-bench Lite is a subset of extremely simple issues from a cherry-picked subset (SWE-bench) of a handful of large (presumably well-run) Python-only projects. Here are some rules they used to trim down the SWE-bench Lite problems: * We remove instances with images, external hyperlinks, references to specific commit shas and references to other pull requests or issues. * We remove instances that have fewer than 40 words in the problem statement. * We remove instances that edit more than 1 file. * We remove instances where the gold patch has more than 3 edit hunks (see patch). See https://www.swebench.com/lite.html https://www.swebench.com/lite.html
  • kevindamm 2y ago
    That's... rather limiting.