3 ms·

I believe it also makes overtraining impossible which is pretty strange for a learning algorithm.

gatlin 5y ago
I don't want or expect you to do a book report for me :) but would you mind linking a thread or two here that an enthusiastic moron could pull at for more detail?
- mrfusion 5y ago
  I could be wrong. If you give it bad data then it could still overfit. https://datacadamia.com/data_mining/overfitting https://datacadamia.com/data_mining/overfitting But if all your data is good There’s no training step per se. And it just keeps improving with more data.
- pedrosorio 5y ago
  Essentially, most ML algorithms, "learn" from data by condensing the data into a set of parameters. For example, in linear regression, you might have N observations (x, y) and condense them into 2 parameters for a line that "best fits" the data (y = mx + b). Instead of linear regression, you can perform "polynomial regression" where you fit a polynomial of some order Q to the data. Since the data you have is often noisy, if Q is large, you will fit a polynomial full of "ups and downs" that goes through the points in the training data exactly, but is not a good representation of what you're trying to learn and will fail miserably on new inputs. In deep learning, you may also specify a network with "too many parameters". A network is trained in steps, by slowly converging the parameters to the "optimal value" to fit the training data you give it. But if we allow it to converge, it might find parameters that "closely fit" the training data, but are not good for generalization, we would prefer a less optimized model. A classic trick (among others) in neural networks is to do "early stopping" to avoid this overfitting phenomenon.
autokad 5y ago
I don't think this is true. you over train in KNN by making the distance small, the best example is when you set k = 1. some would argue that's just a parameter for regularization, but I tend to disagree.
seanhunter 4y ago
I don't think that's the case. Depending on your dataset and objective function, for some (too small) values of k, kNN will settle into local minima and not locate the globally optimal solution (I believe).

I don't want or expect you to do a book report for me :) but would you mind linking a thread or two here that an enthusiastic moron could pull at for more detail?

mrfusion 5y ago
I could be wrong. If you give it bad data then it could still overfit. https://datacadamia.com/data_mining/overfitting https://datacadamia.com/data_mining/overfitting But if all your data is good There’s no training step per se. And it just keeps improving with more data.
pedrosorio 5y ago
Essentially, most ML algorithms, "learn" from data by condensing the data into a set of parameters. For example, in linear regression, you might have N observations (x, y) and condense them into 2 parameters for a line that "best fits" the data (y = mx + b). Instead of linear regression, you can perform "polynomial regression" where you fit a polynomial of some order Q to the data. Since the data you have is often noisy, if Q is large, you will fit a polynomial full of "ups and downs" that goes through the points in the training data exactly, but is not a good representation of what you're trying to learn and will fail miserably on new inputs. In deep learning, you may also specify a network with "too many parameters". A network is trained in steps, by slowly converging the parameters to the "optimal value" to fit the training data you give it. But if we allow it to converge, it might find parameters that "closely fit" the training data, but are not good for generalization, we would prefer a less optimized model. A classic trick (among others) in neural networks is to do "early stopping" to avoid this overfitting phenomenon.

autokad 5y ago

I don't think this is true. you over train in KNN by making the distance small, the best example is when you set k = 1. some would argue that's just a parameter for regularization, but I tend to disagree.

seanhunter 4y ago

I don't think that's the case. Depending on your dataset and objective function, for some (too small) values of k, kNN will settle into local minima and not locate the globally optimal solution (I believe).