>exactly the type of research into neural networks we should be expanding. While it certainly makes for some nice visualizations, the technical insight of this

>exactly the type of research into neural networks we should be expanding. While it certainly makes for some nice visualizations, the technical insight of this is pretty limited. First of all, this fractal structure emerges at learning rates that are far higher than those used in training actual neural networks nowadays. It's interesting that the training still converges for some combinations and that the (expected) hit-and-miss procedure yields a fractal structure. But if you look closely at the images, you'll see the best hyperparameters are, while close, not at the border. So even if you want to follow the meta-learning approach outlined in the post, your gradient descent has already screwed up before if it ever ends up in this fractal boundary region.