4 ms·
Any Deep ReLU Network Is Shallow
- cheekyfibonacci 3y agodon't confuse this with universal approximation - yes shallow ReLU networks are dense in functional space, so at the limit you should be able to get any function you want - but they are talking about exact representation with finitely many neurons here.
- PaulHoule 3y agoIt doesn’t surprise me. It’s been known a long time that you can model arbitrary functions with a 3-layer network with logistic activation.
- RobbieGM 3y agoDo you have any further information or a source for this? As someone unfamiliar with ML, this sounds crazy to me.
- hazrmard 3y agoIt was first proposed here[1]: "Approximation by superpositions of a sigmoidal function" [1]: https://link.springer.com/content/pdf/10.1007/BF02551274.pdf https://link.springer.com/content/pdf/10.1007/BF02551274.pdf
- alexfromapex 3y agoIt seems intuitive since ReLU is just a type of implicit regularization. Why would subsequent gradient descent help once you've achieved the benefit of throwing away the "outliers" or data beyond the threshold you want?