Hmmm, I mean there is a more general question: in the space of algorithms, how continuous are the paths between different solutions? For instance, if you start

Hmmm, I mean there is a more general question: in the space of algorithms, how continuous are the paths between different solutions? For instance, if you start with bubble sort, I don't think it's at all clear this algorithm can take small finite steps to turn into improved to merge sort. This is the low valley you are talking about. This also aligns nicely with the results in the paper. GPT-4 improves, so it's perhaps good enough to make larger steps or transformations of an algorithm, whereas weaker models don't. The real interesting result would be if they improved an algorithm beyond what people have been able to do. Otherwise it's just interesting and a peek into the future..