3 ms·

What would be an example of cheating since it says "no cheating".

• NitpickLawyer 2y ago
  The only reasonable way to cheat on this would be to find real bugs in many repos, train your models on the solutions, wait till the cut-off period, report those bugs, propose PRs and hope your bugs get selected. Pretty small chances, tbh and probably not worth the rewards (the 90% solve rate is pretty much impossible given the constraints - 4x L4s and ~4-6min / problem. There's no way any models that can be ran on those machines under those time limits are that much better than the SotA frontier models)
• zamadatix 2y ago
  The "when they can’t cheat" comment relates to the "Why make a contamination free version?" section.
• andyk 2y ago
  That has a double meaning - half tongue in cheek. 1) since we are creating a contamination-free version of SWE-bench (i.e. scraping a new test set after submissions are frozen) it is guaranteed that agents in this contest can't "cheat", i.e., models can't have trained on the benchmark / agents cant memorize answers. 2) as a general rule in life, don't cheat on things (not that there aren't exceptions)
• tm-infringement 2y ago
  Spearfishing the organizers to get access to the shiny fresh dataset, or using some variation on wrench-based social engineering [0]. 0. https://xkcd.com/538/ https://xkcd.com/538/
  • alwa 2y ago
    Does this work if you have to finalize your submission before said organizers harvest shiny fresh dataset into existence?