4 ms·

This point is very well made: 'local generalization vs. extreme generalization.' Advanced NN's today can locally generalize quite well and there's a lot of rese

This point is very well made: 'local generalization vs. extreme generalization.' Advanced NN's today can locally generalize quite well and there's a lot of research spent to inch their generalization further out. This will probably be done by increasing NN size or increasing the NN building-blocks complexity.

xamuel 9y ago
Or maybe increasing NN size/complexity is the 21st century version of adding epicycles to make geocentrism work. http://wiki.c2.com/?AddingEpicycles http://wiki.c2.com/?AddingEpicycles
- rhaps0dy 9y ago
  Heh, but it makes geocentrism works better! And we don't yet know how 21st century heliocentrism will look like, while adding epicycles is less daunting.
  - mojomark 9y ago
    Yay, I found the rabbit hole - technically, no cellestial body rotates purely around the other (thanks mass!). So, perhaps adding epicycles wasn't erronious after all - just a measurement from a different reference point.
- mehh 9y ago
  Or "adding a few more rules" to a symbolic system ...
Cacti 9y ago
I would really like to hear a definition of what generalization means, because I don't think we have one. Unless we're talking generalization to arbitrary distributions, which is of course unsolvable.
- eanzenberg 9y ago
  Usually in ML context generalization means anytime a model makes a prediction on unseen (not in the training set) inputs. Usually you do CV to see how well your model generalized because you limit the training data seen, predict unseen inputs that you know the answer to, and see how far off the prediction is.
- jd007 9y ago
  I tend to think of "generalization", in the general sense (pun not intended), to be information compression. We are simply trying to answer the question: what is the shortest description (i.e. most informationally compressed/dense version) that fits what we see in this infinite (at least to us mortals) universe of ours? Mathematically the length of such a description can be thought of as the Kolmogorov complexity. Edit: I should add, the information compression performed when generalizing can (and often is) lossy.

xamuel 9y ago

Or maybe increasing NN size/complexity is the 21st century version of adding epicycles to make geocentrism work. http://wiki.c2.com/?AddingEpicycles http://wiki.c2.com/?AddingEpicycles

rhaps0dy 9y ago
Heh, but it makes geocentrism works better! And we don't yet know how 21st century heliocentrism will look like, while adding epicycles is less daunting.
- mojomark 9y ago
  Yay, I found the rabbit hole - technically, no cellestial body rotates purely around the other (thanks mass!). So, perhaps adding epicycles wasn't erronious after all - just a measurement from a different reference point.
mehh 9y ago
Or "adding a few more rules" to a symbolic system ...

Cacti 9y ago

I would really like to hear a definition of what generalization means, because I don't think we have one. Unless we're talking generalization to arbitrary distributions, which is of course unsolvable.

eanzenberg 9y ago
Usually in ML context generalization means anytime a model makes a prediction on unseen (not in the training set) inputs. Usually you do CV to see how well your model generalized because you limit the training data seen, predict unseen inputs that you know the answer to, and see how far off the prediction is.
jd007 9y ago
I tend to think of "generalization", in the general sense (pun not intended), to be information compression. We are simply trying to answer the question: what is the shortest description (i.e. most informationally compressed/dense version) that fits what we see in this infinite (at least to us mortals) universe of ours? Mathematically the length of such a description can be thought of as the Kolmogorov complexity. Edit: I should add, the information compression performed when generalizing can (and often is) lossy.