There's a recent "real coding" benchmark that all the top LLMs perform abysmally on: https://www.swebench.com/ https://www.swebench.com/ However, it seems only

There's a recent "real coding" benchmark that all the top LLMs perform abysmally on: https://www.swebench.com/ https://www.swebench.com/ However, it seems only a matter of time before even this challenge is overcome, and when that happens the question will remain whether it's a real capability or just a data leak.