3 ms·
TBF backpropagation was introduced only in the 1970's, although in hindsight it's a quite trivial application of the chain rule. There were also plenty of "hac
by jampekka 2y ago
TBF backpropagation was introduced only in the 1970's, although in hindsight it's a quite trivial application of the chain rule.
There were also plenty of "hacks" involved to make the networks scale such as dropout regularization, batch normalization, semi-linear activation functions (e.g. ReLU) and adaptive stochastic gradient descent methods.
The maths for basic NNs is really simple but the practice of them is really messy.
- miven 2y agoResidual connections are also worth mentioning as an extremely ubiquitous adaptation, one will be hard-pressed to find a modern architecture that doesn't use those at least to some extent, to the point where the original Resnet paper sits at over 200k citations according to google scholar[1]. [1] https://scholar.google.com/citations?view_op=view_citation&hl=en&user=DhtAFkwAAAAJ&citation_for_view=DhtAFkwAAAAJ:ALROH1vI_8AC https://scholar.google.com/citations?view_op=view_citation&h...
- cma 2y agoHighway nets introduced them in the 90s