> It beats one benchmark which has only been used to compare one kind of system: neural program synthesis systems (Codex, AlphaCode, GPT-j and the one in the a

> It beats one benchmark which has only been used to compare one kind of system: neural program synthesis systems (Codex, AlphaCode, GPT-j and the one in the article, CodeRL). yes, that's what they call "state of the art", winning one benchmark is enough to call result SOTA. > The benchmark is ad hoc and arbitrary and it tells us nothing about the general capability of the compared systems, except that they are still very bad at that particular benchmark. so, what benchmark is better in your opinion, why and what systems demonstrate strong results there? > Gulwani et al is a technical report but it's a staple reference in the field, and an easy introduction to outsider sorry, I don't understand why you keep referencing on that report. It looks 5 years old, thus outdated. What value it has in your opinion exactly?