The problem with that approach is that you end up with a system that is very good at solving programming problems of a particular kind, but does not generalise

The problem with that approach is that you end up with a system that is very good at solving programming problems of a particular kind, but does not generalise very well to arbitrary programming tasks. For example, their system will not have very good performance on, I don't know, leetcode (if it had, we'd have heard all about it). So I don't doubt that we'll see better performance on Codecontests/ Codeforces in the future, perhaps some time next year (just to leave some time after the Nature publication). But not anything of substance. Or they'll just come up with a new dataset that their system performs well on. To be honest, I'm scandalised that such a poorly thought-out approach got a publication in Nature. If I had tried to publish something like that in my field ("we made a firehose that spams code willy-nilly and by heuristic this and that got it to hit the right target 30% of the time! And we beat some humans at it!") I would have been shot down in flames, probably also ruined my reputation for good. But, I guess, it's DeepMind publishing in Nature so it's effectively a racket. One just shrugs and moves on.