4 ms·

This whole example is a red herring. This isn’t a splines versus NNs issue at all, you’re talking about the well known fact that choice of basis affects the abi

by dahart 5y ago

This whole example is a red herring. This isn’t a splines versus NNs issue at all, you’re talking about the well known fact that choice of basis affects the ability to fit, which has nothing to do with whether you use a network. As a concrete proof, since a spline is (usually) a polynomial function, it can be defined as a linear network with as many layers as the spline’s polynomial order, in other words splines are a strict subset of the functions you can build using neural networks. You can also make a neural network out of spline neurons if you want. And you can cherry pick lots of different functions that work better for splines than other choices, and you can also cherry pick functions that perform worse for splines than other bases. Splines perform far worse on a periodic function of arbitrary domain than a Fourier fit. Your example is contrived because you artificially constrained the range to [0, 2pi].

credit_guy 5y ago
I'm sorry, but I have to disagree with you here. The "Universal Approximation Theorem" is not the point of neural networks. People should stop mentioning it, or if they do, they should state at the same time that there's nothing special about NNs, that numerous classes of functions possess the same property. Here's my own pitch for neural networks: NN's suck. Big time. They suck in low dimensions and they suck in high dimensions. But the curse of dimensionality is so formidable, that everything sucks in high dimension. Neural networks just happen to suck less than all other known methods. And because they suck a bit less, there are applications where they are useful, and they have no substitute.
- abeppu 5y ago
  > Neural networks just happen to suck less than all other known methods. Or, perhaps, the best demonstrated performance of NNs exceeds the best demonstrated performance of other known methods for many tasks. But ... the amount of compute, investment in tooling, and attention that have been thrown at deep learning in the past decade is at a scale where ... do we actually know that other methods would perform worse with the same resources? Is there some alternate timeline where in 2012 someone figured out how to run MCMC for bayesian non-parametrics over much larger datasets or something, and the whole field of ML just tilted in a different direction?
  - credit_guy 5y ago
    That's a very good observation. However, deep learning didn't just get the share of the first mover. Before DL was popular, Support Vector Machines used to be where all the ML fun research was happening. And just out of nowhere, Random Forests and XGBoost came and took the crown if only for a fleeting moment. Gaussian Processes always showed promise, but I'm not sure they delivered. Deep Learning just delivered. I guess it's because of the composability. But you are absolutely right that there's no proof and now way of knowing right now if DL is the best there possibly can be.
- dahart 5y ago
  It doesn’t matter if you disagree (BTW I don’t know what you disagree with specifically, and I did not mention the Universal Approximation Theorem. It seems like you’re making some assumptions.) A polynomial spline is still a subset of a neural network, so if you’re right, all you’re demonstrating is that splines also suck at solving the same problems that neural networks solve. The discrepancy between the two here, again, has nothing to do with networks and everything to do with your contrived example.

by dahart 5y ago

credit_guy 5y ago

I'm sorry, but I have to disagree with you here. The "Universal Approximation Theorem" is not the point of neural networks. People should stop mentioning it, or if they do, they should state at the same time that there's nothing special about NNs, that numerous classes of functions possess the same property. Here's my own pitch for neural networks: NN's suck. Big time. They suck in low dimensions and they suck in high dimensions. But the curse of dimensionality is so formidable, that everything sucks in high dimension. Neural networks just happen to suck less than all other known methods. And because they suck a bit less, there are applications where they are useful, and they have no substitute.

abeppu 5y ago
> Neural networks just happen to suck less than all other known methods. Or, perhaps, the best demonstrated performance of NNs exceeds the best demonstrated performance of other known methods for many tasks. But ... the amount of compute, investment in tooling, and attention that have been thrown at deep learning in the past decade is at a scale where ... do we actually know that other methods would perform worse with the same resources? Is there some alternate timeline where in 2012 someone figured out how to run MCMC for bayesian non-parametrics over much larger datasets or something, and the whole field of ML just tilted in a different direction?
- credit_guy 5y ago
  That's a very good observation. However, deep learning didn't just get the share of the first mover. Before DL was popular, Support Vector Machines used to be where all the ML fun research was happening. And just out of nowhere, Random Forests and XGBoost came and took the crown if only for a fleeting moment. Gaussian Processes always showed promise, but I'm not sure they delivered. Deep Learning just delivered. I guess it's because of the composability. But you are absolutely right that there's no proof and now way of knowing right now if DL is the best there possibly can be.
dahart 5y ago
It doesn’t matter if you disagree (BTW I don’t know what you disagree with specifically, and I did not mention the Universal Approximation Theorem. It seems like you’re making some assumptions.) A polynomial spline is still a subset of a neural network, so if you’re right, all you’re demonstrating is that splines also suck at solving the same problems that neural networks solve. The discrepancy between the two here, again, has nothing to do with networks and everything to do with your contrived example.