2 ms·
In competitive programing, it's clear what is a good and a bad splution, and they usualy have a set of test so people can't claim they cheated. With real world
by gus_massa 2y ago
In competitive programing, it's clear what is a good and a bad splution, and they usualy have a set of test so people can't claim they cheated.
With real world questions, everyone can have a different opinion. Let's say you must clasify photos as dogs, cats or other. What about an apple tree? A hyena? An AI generated version of a dog with cat fur? A lion? Hello Kitty?
- ianbutler 2y agoSWE bench has gold code patches and the passing test suite patch for after the github issue was completed. While you may argue over the style of the code produced by the model there is a known good passing state for the model to achieve. For now that's the closest representation of a real world problem solve in a controlled repeatable benchmark we have.