5 ms·

I'm sorry, but I have to disagree with you here. The "Universal Approximation Theorem" is not the point of neural networks. People should stop mentioning it, o

I'm sorry, but I have to disagree with you here. The "Universal Approximation Theorem" is not the point of neural networks. People should stop mentioning it, or if they do, they should state at the same time that there's nothing special about NNs, that numerous classes of functions possess the same property. Here's my own pitch for neural networks: NN's suck. Big time. They suck in low dimensions and they suck in high dimensions. But the curse of dimensionality is so formidable, that everything sucks in high dimension. Neural networks just happen to suck less than all other known methods. And because they suck a bit less, there are applications where they are useful, and they have no substitute.

abeppu 5y ago
> Neural networks just happen to suck less than all other known methods. Or, perhaps, the best demonstrated performance of NNs exceeds the best demonstrated performance of other known methods for many tasks. But ... the amount of compute, investment in tooling, and attention that have been thrown at deep learning in the past decade is at a scale where ... do we actually know that other methods would perform worse with the same resources? Is there some alternate timeline where in 2012 someone figured out how to run MCMC for bayesian non-parametrics over much larger datasets or something, and the whole field of ML just tilted in a different direction?
- credit_guy 5y ago
  That's a very good observation. However, deep learning didn't just get the share of the first mover. Before DL was popular, Support Vector Machines used to be where all the ML fun research was happening. And just out of nowhere, Random Forests and XGBoost came and took the crown if only for a fleeting moment. Gaussian Processes always showed promise, but I'm not sure they delivered. Deep Learning just delivered. I guess it's because of the composability. But you are absolutely right that there's no proof and now way of knowing right now if DL is the best there possibly can be.
dahart 5y ago
It doesn’t matter if you disagree (BTW I don’t know what you disagree with specifically, and I did not mention the Universal Approximation Theorem. It seems like you’re making some assumptions.) A polynomial spline is still a subset of a neural network, so if you’re right, all you’re demonstrating is that splines also suck at solving the same problems that neural networks solve. The discrepancy between the two here, again, has nothing to do with networks and everything to do with your contrived example.

abeppu 5y ago

> Neural networks just happen to suck less than all other known methods. Or, perhaps, the best demonstrated performance of NNs exceeds the best demonstrated performance of other known methods for many tasks. But ... the amount of compute, investment in tooling, and attention that have been thrown at deep learning in the past decade is at a scale where ... do we actually know that other methods would perform worse with the same resources? Is there some alternate timeline where in 2012 someone figured out how to run MCMC for bayesian non-parametrics over much larger datasets or something, and the whole field of ML just tilted in a different direction?

credit_guy 5y ago
That's a very good observation. However, deep learning didn't just get the share of the first mover. Before DL was popular, Support Vector Machines used to be where all the ML fun research was happening. And just out of nowhere, Random Forests and XGBoost came and took the crown if only for a fleeting moment. Gaussian Processes always showed promise, but I'm not sure they delivered. Deep Learning just delivered. I guess it's because of the composability. But you are absolutely right that there's no proof and now way of knowing right now if DL is the best there possibly can be.

dahart 5y ago

It doesn’t matter if you disagree (BTW I don’t know what you disagree with specifically, and I did not mention the Universal Approximation Theorem. It seems like you’re making some assumptions.) A polynomial spline is still a subset of a neural network, so if you’re right, all you’re demonstrating is that splines also suck at solving the same problems that neural networks solve. The discrepancy between the two here, again, has nothing to do with networks and everything to do with your contrived example.