The key is that step 1 isn't random: it's guided by past experience, and intuition. At the local level it might be random, but from a higher level, the search i

The key is that step 1 isn't random: it's guided by past experience, and intuition. At the local level it might be random, but from a higher level, the search is guided. That said, there are incremental papers that tweak some parameters, and then there are others like this one that take a big, risky step back from the rest of the community (I.e using convolutions for computer vision), and make a discovery on the common benchmark. Speaking my own experience, I think what keeps researchers addicted is the random variable reward we get from seeing our new algorithm's performance on a common benchmark. Even better if the performance near state of the art, but not necessary.