Overall a very interesting result. I’m not sure why they compare to a list of results for the MBPP benchmark but don’t seem to include these results which are

Overall a very interesting result. I’m not sure why they compare to a list of results for the MBPP benchmark but don’t seem to include these results which are much better: https://paperswithcode.com/sota/code-generation-on-mbpp https://paperswithcode.com/sota/code-generation-on-mbpp