4 ms·

Intelligence is in the properties of the data supplied to the kNN. Not in its distance function. Yes, kNN is the bound if you formulate learning as: there exis

Intelligence is in the properties of the data supplied to the kNN. Not in its distance function. Yes, kNN is the bound if you formulate learning as: there exists an f_true: Pixels->Animals; we have an infinite sampling of (Pixels, Animals); kNN performs at 100%. Therefore kNN is the bound. Whereas I dont think this has much to do with learning. Learning for me is: the function probably doesnt exist, we dont know what its domain would even be, and we have no idea how to sample that space. You know: the history of human civilization, but also: the history of a day of any animal's life. We encounter problems daily whose functions can only be modelled by partial input and output domains, and our job is to find ways of resolving the inherent ambiguities in those situations. Ie., to find what parts of the world are relevant, to find ways of measuring it, and hence to find more data until the ambiguity resolves itself. kNN being a perfect algorithm is a catastrophe for this whole formulation of AI. All it means is that the whole field of machine learning has as its limit what you can do if you already know the answer. You can see from this that it isnt learning at all. The situation is exactly backwards: ML assumes relevant well-carved unambiguous data. The whole problem of learning is precisely to produce it!

vladf 5y ago
Looks like someone (GP) beat me to the punch with my sibling comment (https://news.ycombinator.com/item?id=31053031 https://news.ycombinator.com/item?id=31053031)! > we have infinite sampling We don't. You're taking the learning out of the process you're discussing by using an infinite sample. Of course, then, the machine learning part becomes trivial, and it's just an optimization problem for minimal training error! If we have all the data, like infinite data, then we have the classification for every one of the (finite) number of possible images already. At that point, why stop at kNN? Use a hashmap!
dotnet00 5y ago
>kNN being a perfect algorithm is a catastrophe for this whole formulation of AI. All it means is that the whole field of machine learning has as its limit what you can do if you already know the answer. You can see from this that it isnt learning at all. Doesn't the existence of unsupervised training and models that solve previously unsolved problems somewhat invalidate the NN=kNN analogy then? The protein folding problem didn't have an already known overall answer for instance. The model clearly 'learned' representational information that we are not aware of in order to be so much more effective at solving the problem than we are. Or similarly with the Starcraft or Dota AI, while some strategies learned were similar to humans, others were entirely novel. I think that's getting pretty close to 'learning' as you've defined it.
- mjburgess 5y ago
  Animal learning is unsupervised, sure, but with no domain goal and no relevant dataset.
  - dotnet00 5y ago
    I'd argue that 'survive and reproduce' is not all too different as a goal from 'maximize this score' when it comes to problems as open ended as playing MOBAs. There isn't much of a dataset and there are endless numbers of ways to go about solving the problem.
    - mjburgess 5y ago
      Sure, animal goals are self-regulative and defined in terms of their capacities alone. If you can build a system like this, with no presumed domain distribution internally, then you're a step towards intelligence.
      - mannykannot 5y ago
        I think you are probably right about needing to be "in the world" (if by that, you mean being able to interact with it and see how its state, and therefore the data you have about it, changes), but I feel you are being too hasty in ruling out any situation where humans have had any role in preparing the data. In the evolution of natural intelligence, physics provides the domain distribution and Darwin has explained what the reward function is. It does not follow that no learning can occur in an environment where either or both of the domain and reward function are defined by humans. As far as I can tell, this was the explicit goal of AlphaGo Zero. Even if we were to accept your position that the data it was started with "contained the solution", then the fact it does rather better than humans at finding that solution is, by itself, significant. (In this view, one might characterize the evolution of natural intelligence as finding, within the biosphere domain, a rather successful solution to the survival problem.) It would seem to me to be begging the question to say that the success of AlphaGo Zero is not learning because the environment was human-specified (At the same time, as Go is a highly-constrained environment, there is no reason to suppose that its successes indicate we are anywhere near AGI.)
randcraw 5y ago
> Intelligence is in the properties of the data supplied to the kNN. Not in its distance function. If this were true, then all people who observe the same training samples would learn equally well and predict equally well. But we don't. Some of us invent a better loss function than others or reject irrelevant data better. The representation formed by the data is surely derived, not fixed in its raw form. Forming a better representation of the data is the art of learning better. I'm convinced learning and remembering what you learn is more than kNN; in humans it involves data compression and recontextualization as you improve your loss function or tie your old representation to newly learned data — old raw data and representations that heretofore you had not believed were relevant to that objective (or lesson).
- just-ok 5y ago
  > all people who observe the same training samples would learn equally well and predict equally well. But we don’t. How do you know that we don’t? It’s an impossible experiment to conduct, right? Since same training samples == every stimuli experienced since birth, and even then you have the problem of possibly-different “initial conditions” i.e. development of the brain in the womb. Maybe if we did replicate that, we would learn & predict equally well.

vladf 5y ago

Looks like someone (GP) beat me to the punch with my sibling comment (https://news.ycombinator.com/item?id=31053031 https://news.ycombinator.com/item?id=31053031)! > we have infinite sampling We don't. You're taking the learning out of the process you're discussing by using an infinite sample. Of course, then, the machine learning part becomes trivial, and it's just an optimization problem for minimal training error! If we have all the data, like infinite data, then we have the classification for every one of the (finite) number of possible images already. At that point, why stop at kNN? Use a hashmap!

dotnet00 5y ago

>kNN being a perfect algorithm is a catastrophe for this whole formulation of AI. All it means is that the whole field of machine learning has as its limit what you can do if you already know the answer. You can see from this that it isnt learning at all. Doesn't the existence of unsupervised training and models that solve previously unsolved problems somewhat invalidate the NN=kNN analogy then? The protein folding problem didn't have an already known overall answer for instance. The model clearly 'learned' representational information that we are not aware of in order to be so much more effective at solving the problem than we are. Or similarly with the Starcraft or Dota AI, while some strategies learned were similar to humans, others were entirely novel. I think that's getting pretty close to 'learning' as you've defined it.

mjburgess 5y ago
Animal learning is unsupervised, sure, but with no domain goal and no relevant dataset.
- dotnet00 5y ago
  I'd argue that 'survive and reproduce' is not all too different as a goal from 'maximize this score' when it comes to problems as open ended as playing MOBAs. There isn't much of a dataset and there are endless numbers of ways to go about solving the problem.
  - mjburgess 5y ago
    Sure, animal goals are self-regulative and defined in terms of their capacities alone. If you can build a system like this, with no presumed domain distribution internally, then you're a step towards intelligence.
    - mannykannot 5y ago
      I think you are probably right about needing to be "in the world" (if by that, you mean being able to interact with it and see how its state, and therefore the data you have about it, changes), but I feel you are being too hasty in ruling out any situation where humans have had any role in preparing the data. In the evolution of natural intelligence, physics provides the domain distribution and Darwin has explained what the reward function is. It does not follow that no learning can occur in an environment where either or both of the domain and reward function are defined by humans. As far as I can tell, this was the explicit goal of AlphaGo Zero. Even if we were to accept your position that the data it was started with "contained the solution", then the fact it does rather better than humans at finding that solution is, by itself, significant. (In this view, one might characterize the evolution of natural intelligence as finding, within the biosphere domain, a rather successful solution to the survival problem.) It would seem to me to be begging the question to say that the success of AlphaGo Zero is not learning because the environment was human-specified (At the same time, as Go is a highly-constrained environment, there is no reason to suppose that its successes indicate we are anywhere near AGI.)

randcraw 5y ago

> Intelligence is in the properties of the data supplied to the kNN. Not in its distance function. If this were true, then all people who observe the same training samples would learn equally well and predict equally well. But we don't. Some of us invent a better loss function than others or reject irrelevant data better. The representation formed by the data is surely derived, not fixed in its raw form. Forming a better representation of the data is the art of learning better. I'm convinced learning and remembering what you learn is more than kNN; in humans it involves data compression and recontextualization as you improve your loss function or tie your old representation to newly learned data — old raw data and representations that heretofore you had not believed were relevant to that objective (or lesson).

just-ok 5y ago
> all people who observe the same training samples would learn equally well and predict equally well. But we don’t. How do you know that we don’t? It’s an impossible experiment to conduct, right? Since same training samples == every stimuli experienced since birth, and even then you have the problem of possibly-different “initial conditions” i.e. development of the brain in the womb. Maybe if we did replicate that, we would learn & predict equally well.