3 ms·

Might be a stupid question, but can we skip the random walk and just pick the middle?

• pigscantfly 6y ago
  Not a stupid question at all, but one problem is that the boundaries of a zero-train-loss region are not well characterized and evaluating the validation loss even at a single point is computationally expensive. The centroid of one of these regions might not even be inside it (eg. donut shape but in higher dimensions) Interesting discussion though -- probably worth a few papers if someone were to investigate further.
  • chillee 6y ago
    This just sounds like Stochastic Weight Averaging, which works quite well: https://arxiv.org/abs/1803.05407 https://arxiv.org/abs/1803.05407