3 ms·
> 68.3% of SWE-bench samples being filtered out due to underspecification, unfair unit tests, or other issues. I have a feeling that the filtered samples bett
by janice1999 2y ago
> 68.3% of SWE-bench samples being filtered out due to underspecification, unfair unit tests, or other issues.
I have a feeling that the filtered samples better represent actual issues in the real world. How would "AI" even start with issues like "product sometimes runs slow" or "it crashes 1/20 times with this vague error message". I think my job is secure for the moment. I would love AI tooling to better sort and categorise issues though. In a rush to "replace" people I think AI companies are focusing on the wrong things.
- viraptor 2y ago> How would "AI" even start with issues like "product sometimes runs slow" or "it crashes 1/20 times with this vague error message". By asking more questions, just like a developer would. This is already available in the chat form, but that's a separate set of skills and would require the other party to give responses. It's a useful test, but I believe not in the context of what swe-bench wants to measure.