4 ms·
I am not trying to downplay the contribution of the paper, but isn't it obvious that this is the case?
by xcodevn 3y ago
I am not trying to downplay the contribution of the paper, but isn't it obvious that this is the case?
- teaearlgraycold 3y agoObvious to whom?
- bloaf 3y agoI think the "obvious" comment was a bit snarky, but out of curiosity, I posed the question to the Groq website which currently happens to be on the front page right now. (It claims to run Mixtral 8x7B-32k at 500 T/s) And indeed, the AI response indicated that the boundary between convergence and divergence is not well defined, has many local maxima and minima, and could be quote: "fractal or chaotic, with small changes in hyperparameters leading to drastically different outcomes."
- Buttons840 3y agoI'll defend the idea that it was obvious. (Although, it wasn't obvious to me until someone pointed it out, so maybe that's not obvious.) If you watch this video[0], you'll see in the first frame that there is a clear boundary between learning rates that converge or not. Ignoring this paper for a moment, what if we zoom in really really close to that boundary? There are two possibilities, either (1) the boundary is perfectly sharp no matter how closely we inspect it, or (2) it is a little bit fuzzy. Of those two possibilities, the perfectly sharp boundary would be more surprising. [0]: https://x.com/jaschasd/status/1756930242965606582 https://x.com/jaschasd/status/1756930242965606582
- eapriv 3y agoNot only it is not obvious; it is not known to be true.
- barbarr 3y agoI don't think it's obvious per se, but people who have studied numerical methods at the graduate level have likely seen fractal boundaries like this before - even Newton's method produces them [0]. The phenomenon says more about iterative methods than it says about neural networks. [0] https://en.wikipedia.org/wiki/Newton_fractal https://en.wikipedia.org/wiki/Newton_fractal