4 ms·

Apparently function approximation is an important property to make something useful. It seems reasonable that it should be part of an intelligent system. Is the

by return0 10y ago

Apparently function approximation is an important property to make something useful. It seems reasonable that it should be part of an intelligent system. Is there a similar "deep learning" based on k-nearest neighbor classifiers?

yzmtf2008 10y ago
With kNN, if you have enough data and a good definition of "distance", then yes, it is in a way similar to "deep learning", just deep in the sense that's data and human-labor intensive. I'm oversimplifying here, but what linear regression and logistic regression did to kNN is that you can automate the "distance" function, but you still have to manually construct the features. What DL did is one step further -- don't even bother feature-engineering, the network can construct the features themselves. You see, there isn't a function that kNN can't approximate. If you have an impressive feature list and a training datum that has every feature exactly the same as your input, there is no reason not to directly use the output of said datum. It's the feasibility that matters. Of course, DL is a huge step forward. It has made "impossible things possible, and hard things easy". The author also acknowledged DL's importance. However, that doesn't mean we should stop at DL.
srean 10y ago
> Apparently function approximation is an important property 'important' sure but not always a good idea. When you are learning from finite (but potentially very large) number of noisy examples, the universal function approximator will try to approximate the noise also. Learning is a delicate dance between the capacity of your learner and the complexity of the thing that you are trying to learn. The key notion is that it takes two to tango. If you have few examples, you are better of not using deep nets. Stated another way, a universal approximator has infinite potential for distraction. "Hey shiny" and it could veer off in a direction you don't want it to go. You want to pick out army tanks from pictures and it might learn to distinguish pictures taken on a cloudy day from those taken on sunny days.
argonaut 10y ago
You missed my point entirely. My point is that showing something is a universal function approximator indicates nothing about whether that method is a good method in general. Necessary but not sufficient. kNN is one of the dumbest/simplest methods out there.
- return0 10y ago
  a good method for what?
  - argonaut 10y ago
    For machine learning. For solving problems. What else? A Fourier series can model pretty much any function. Doesn't mean it's a good model for your problem.

by return0 10y ago

yzmtf2008 10y ago

With kNN, if you have enough data and a good definition of "distance", then yes, it is in a way similar to "deep learning", just deep in the sense that's data and human-labor intensive. I'm oversimplifying here, but what linear regression and logistic regression did to kNN is that you can automate the "distance" function, but you still have to manually construct the features. What DL did is one step further -- don't even bother feature-engineering, the network can construct the features themselves. You see, there isn't a function that kNN can't approximate. If you have an impressive feature list and a training datum that has every feature exactly the same as your input, there is no reason not to directly use the output of said datum. It's the feasibility that matters. Of course, DL is a huge step forward. It has made "impossible things possible, and hard things easy". The author also acknowledged DL's importance. However, that doesn't mean we should stop at DL.

srean 10y ago

> Apparently function approximation is an important property 'important' sure but not always a good idea. When you are learning from finite (but potentially very large) number of noisy examples, the universal function approximator will try to approximate the noise also. Learning is a delicate dance between the capacity of your learner and the complexity of the thing that you are trying to learn. The key notion is that it takes two to tango. If you have few examples, you are better of not using deep nets. Stated another way, a universal approximator has infinite potential for distraction. "Hey shiny" and it could veer off in a direction you don't want it to go. You want to pick out army tanks from pictures and it might learn to distinguish pictures taken on a cloudy day from those taken on sunny days.

argonaut 10y ago

You missed my point entirely. My point is that showing something is a universal function approximator indicates nothing about whether that method is a good method in general. Necessary but not sufficient. kNN is one of the dumbest/simplest methods out there.

return0 10y ago
a good method for what?
- argonaut 10y ago
  For machine learning. For solving problems. What else? A Fourier series can model pretty much any function. Doesn't mean it's a good model for your problem.