3 ms·
> they won't necessarily be much wiser about why those methods work, other than having a good intuition confirmed by numerical experiments. This is the state o
by modeless 3y ago
> they won't necessarily be much wiser about why those methods work, other than having a good intuition confirmed by numerical experiments.
This is the state of the field as a whole, isn't it?
> Wouldn't it be more useful to dedicate the space to convergence proofs for ADAM (which do exist)
Convergence proofs don't really explain why Adam tends to work better than other methods.
It's hard to blame them for not being able to explain things that, currently, nobody understands. But I guess it kind of undermines the idea of a theory-heavy approach to teaching if the theory we have can't predict the things that are actually important.
- go_elmo 3y ago*Cant predict the important things _yet_
- c7b 3y agoADAM is known to have better convergence bounds than other methods. Theoretical bounds may not explain the full story of why a method works well, but it is how mathematicians reason about it. I'm only blaming them for not sharing those relevant parts of what we already know. My bigger pain point even is how they choose to allocate their space: the theorem statements for the most relevant results are missing, the proofs for the more interesting theorems are just citations, while the proofs for basic and arguably tangentially relevant lemmata from eg probability theory take up pages and pages.