> Also... telling that they choose to benchmark against CodeForces rather than SWE-bench. They also worked with Devin to benchmark it on Devin's internal bench

> Also... telling that they choose to benchmark against CodeForces rather than SWE-bench. They also worked with Devin to benchmark it on Devin's internal benchmarks, where it's twice as good as GPT-4o: https://x.com/cognition_labs/status/1834292718174077014 https://x.com/cognition_labs/status/1834292718174077014 https://www.cognition.ai/blog/evaluating-coding-agents https://www.cognition.ai/blog/evaluating-coding-agents