5 ms·

I don't really understand why this theory is attractive. I hope someone here can help explain: Suppose we build a 1000 NNs for object detection, and let them '

I don't really understand why this theory is attractive. I hope someone here can help explain: Suppose we build a 1000 NNs for object detection, and let them 'vote' for which model to believe. What would be the best voting mechanism? Probably a final voting layer, right? This architecture can be described as a single NN where we disallow connections between 1000 separate parts of the network in the first n-1 layers. What would be the advantage of this? I appreciate that the brain - might - do it, but unless you can show any indication of this with experiments, or argue for why it would result in a performance increase, what's the point? Anyone can come up with an inferior architecture and propose that it's what the brain does.

MRD85 8y ago
Is that the best way to describe the architecture if the n-1 layers are fed with completely different input data?
- mollerhoj 8y ago
  No I guess not. My mistake: If the input is completely separate, then it makes sense to call it 1000 different NNs. If the input is not completely separate, then the architure I described with a high percentage of dropout in the first layer seems like a reasonable description? This doesn't address my quarrel with the theory though; Why would that be either superior performance wise or more biologically realistic?
  - jmmcd 8y ago
    I agree that a "voting" mechanism sounds all wrong, not only for the reason you said. In the scenario described, where we have some neurons coming ultimately from the hand and sense of touch, and others coming ultimately from the eye and sense of vision, and they disagree about what they're perceiving, they're not voting! First of all, it might be better described as a "negotiation", and in fairness I think their theory does envisage this (but they do use the term "vote", albeit in scare quotes). Second, what this theory misses (as does a lot of recent work which takes neural networks as a jumping-off point) is the transition from sensory/sub-cognitive processing up to conscious/cognitive processing. If my hand feels a cat and my eye sees a coffee cup, then I'll consciously notice the contradiction and gather more evidence.
    - your-nanny 8y ago
      what may be envisioned is a process like that of information integration in two-choice tasks about which we have several neural models.
  - deleted 8y ago
    [deleted]
  - mannykannot 8y ago
    To address your last question, I think they are proposing (but have not yet demonstrated) that the human neocortex works this way, and I imagine the motivation for being more biologically realistic (beyond its usefulness in learning more about the biology) is that biology is still outperforming AI in these tasks.
nnq 8y ago
...one can probably try to use smth like genetic algorithms to select/refine the N-1 layers ...current neural networks take the architecture to be fixed or at least "given" (by some outside process, which might itself be an NN, but still). We'll probably need a way to break out of this fixed/given architecture mode for next level breakthroughs. Smth like "differentiating through architecture-space too"... (guess the marketing dpts. will call it "profound deep learning" or something even cheesier :P) (But I'm totally clueless as to whether Numenta is doing anything useful on this or getting anywhere useful with what they're doing ...no time to research this and kind of way above my current knowledge and understanding :))
evrydayhustling 8y ago
The takeaway I got from OP is that Numenta is promoting more redundancy in those first [n-1] layers than might be typical in (e.g.) existing vision systems. The paper the original post is about doesn't discuss value-of-redundancy in those terms, but suggests that having different modules with similar structure is useful for learning compositional models where objects are composed out of other objects. I'm not really in a position to say how well this aligns with SOTA techniques; intuitively it seems like you could reinterpret the algebra of those modules a few different ways and have the same function, so I'd want to see more proof-of-novelty. Separately, there's a somewhat dusty area of machine learning called "Boosting" which treats this problem of combining a bunch of different classifiers explicitly. You're exactly right that traditional DNNs can implement a pretty decent approach to boosting, but there are some interesting techniques from that community that don't fit as easily into the standard DNN data graph perspective. For example, check out BrownBoost, which lets you set parameters about believed noise in training data: https://en.wikipedia.org/wiki/BrownBoost https://en.wikipedia.org/wiki/BrownBoost
gambler 8y ago
This is how random forests work. Each takes a different subset of features and builds a model. Then models are averaged for much better accuracy. Unfortunately, averaging seems to be the crudest mechanism for combining models and far from the best. There are more sophisticated approaches like boosting and synergy (of monotonic rules).
bonoboTP 8y ago
This is a quite old idea, called ensembling, and it works really well with all sorts of machine learning algorithms.

MRD85 8y ago

Is that the best way to describe the architecture if the n-1 layers are fed with completely different input data?

mollerhoj 8y ago
No I guess not. My mistake: If the input is completely separate, then it makes sense to call it 1000 different NNs. If the input is not completely separate, then the architure I described with a high percentage of dropout in the first layer seems like a reasonable description? This doesn't address my quarrel with the theory though; Why would that be either superior performance wise or more biologically realistic?
- jmmcd 8y ago
  I agree that a "voting" mechanism sounds all wrong, not only for the reason you said. In the scenario described, where we have some neurons coming ultimately from the hand and sense of touch, and others coming ultimately from the eye and sense of vision, and they disagree about what they're perceiving, they're not voting! First of all, it might be better described as a "negotiation", and in fairness I think their theory does envisage this (but they do use the term "vote", albeit in scare quotes). Second, what this theory misses (as does a lot of recent work which takes neural networks as a jumping-off point) is the transition from sensory/sub-cognitive processing up to conscious/cognitive processing. If my hand feels a cat and my eye sees a coffee cup, then I'll consciously notice the contradiction and gather more evidence.
  - your-nanny 8y ago
    what may be envisioned is a process like that of information integration in two-choice tasks about which we have several neural models.
- deleted 8y ago
  [deleted]
- mannykannot 8y ago
  To address your last question, I think they are proposing (but have not yet demonstrated) that the human neocortex works this way, and I imagine the motivation for being more biologically realistic (beyond its usefulness in learning more about the biology) is that biology is still outperforming AI in these tasks.

nnq 8y ago

...one can probably try to use smth like genetic algorithms to select/refine the N-1 layers ...current neural networks take the architecture to be fixed or at least "given" (by some outside process, which might itself be an NN, but still). We'll probably need a way to break out of this fixed/given architecture mode for next level breakthroughs. Smth like "differentiating through architecture-space too"... (guess the marketing dpts. will call it "profound deep learning" or something even cheesier :P) (But I'm totally clueless as to whether Numenta is doing anything useful on this or getting anywhere useful with what they're doing ...no time to research this and kind of way above my current knowledge and understanding :))

evrydayhustling 8y ago

The takeaway I got from OP is that Numenta is promoting more redundancy in those first [n-1] layers than might be typical in (e.g.) existing vision systems. The paper the original post is about doesn't discuss value-of-redundancy in those terms, but suggests that having different modules with similar structure is useful for learning compositional models where objects are composed out of other objects. I'm not really in a position to say how well this aligns with SOTA techniques; intuitively it seems like you could reinterpret the algebra of those modules a few different ways and have the same function, so I'd want to see more proof-of-novelty. Separately, there's a somewhat dusty area of machine learning called "Boosting" which treats this problem of combining a bunch of different classifiers explicitly. You're exactly right that traditional DNNs can implement a pretty decent approach to boosting, but there are some interesting techniques from that community that don't fit as easily into the standard DNN data graph perspective. For example, check out BrownBoost, which lets you set parameters about believed noise in training data: https://en.wikipedia.org/wiki/BrownBoost https://en.wikipedia.org/wiki/BrownBoost

gambler 8y ago

This is how random forests work. Each takes a different subset of features and builds a model. Then models are averaged for much better accuracy. Unfortunately, averaging seems to be the crudest mechanism for combining models and far from the best. There are more sophisticated approaches like boosting and synergy (of monotonic rules).

bonoboTP 8y ago

This is a quite old idea, called ensembling, and it works really well with all sorts of machine learning algorithms.