3 ms·

This is a terrible benchmark. It literally tests the models on their ability to track shifting line numbers. If they cannot keep up, no amount of abstract reaso

by mordae 5mo ago

This is a terrible benchmark. It literally tests the models on their ability to track shifting line numbers. If they cannot keep up, no amount of abstract reasoning can redeem them.

lordmauve 4mo ago
Where did you get that idea? It uses mini-swe-agent, same as SWE-Bench. https://github.com/datacurve-ai/deep-swe https://github.com/datacurve-ai/deep-swe
- mordae 4mo ago
  [flagged]