The detail how they measure this in the readme. This is directed at all the sibling comments as well! TLDR they recompile and then re-execute (including test s

The detail how they measure this in the readme. This is directed at all the sibling comments as well! TLDR they recompile and then re-execute (including test suites). From the results table it looks like GPT4 still "outperforms" their model in recompilation, but their recompiled code has a much better re-execution success rate (less hallucinations). But, that re-execution rate is still pretty lacking (around 14%), even if better than GPT4.