4 ms·

Jeff and Subutai have been on this hobby horse for a very long time. They don't have much to show for it, and many of their ideas are controversial even within

Jeff and Subutai have been on this hobby horse for a very long time. They don't have much to show for it, and many of their ideas are controversial even within neuroscience (for example, some think that their venerated "cortical columns" are structures without meaningful functions). It is very easy to bet against them given the fact that nothing much has happened, and "theories" about the brain are a dime-a-dozen. I'm glad that they have the gumption to go all-in on these ideas, but I'm not expecting anything earth shattering anytime soon.

nabla9 7y ago
That's the blessing/curse with completely self funded research group. No strict timelines, yearly review, benchmarks or external pressure. There is no hurry to deliver, you can dream on and keep developing your pet idea indefinitely. Nobody can tell you that you were wrong and you are now done.
- tux1968 7y ago
  That explains why they can continue, but it doesn't make it more likely that the idea will ever bare fruit.
- p1esk 7y ago
  No strict timelines, yearly review, benchmarks or external pressure. Maybe not imposed on Jeff himself, at least not in this narrow sense, but the people who work for him most likely have all that.
espeed 7y ago
Understanding the underlying structure is key. And the significance of structure -- the ability to see it, understand it, and re-create it -- has been recognized and pointed out by Feynman [1] and some of the world's best programmers [2]. Using data-structure representations that best match reality's underlying structure may be key too. With this in mind I had a thought -- a "dopey idea" in the Steve Jobs to Jony Ive sense of the word [3] -- the brain doesn't do backprop [4] -- at least not directly (as far as we know) -- but neural nets using backprop/gradient-descent work pretty well. Why so? What if the underlying structure these nets are modeling isn't the brain but something else...maybe something more fundamental...like the curvature of spacetime, a path integral representation of information, or some other crude representation of an underlying quantum structure. And if so, what if learning is just the brain optimizing itself in accordance with this structure, and has this idea been explored before? Evidently it has. The idea of the brain as a quantum information processor has been explored to some degree in holonomic brain theory [5]. [1] "What I cannot create, I do not understand" https://en.wikiquote.org/wiki/Richard_Feynman https://en.wikiquote.org/wiki/Richard_Feynman [2] Programmers on Data Structures https://news.ycombinator.com/item?id=20047607 https://news.ycombinator.com/item?id=20047607 [3] Jony Ive talking about Steve Jobs' Dopey Ideas [video] https://www.youtube.com/watch?v=4WNYYCX3Tg0 https://www.youtube.com/watch?v=4WNYYCX3Tg0 [4] Artificial intelligence pioneer says we need to start over https://www.axios.com/artificial-intelligence-pioneer-says-we-need-to-start-over-1513305524-f619efbd-9db0-4947-a9b2-7a4c310a28fe.html https://www.axios.com/artificial-intelligence-pioneer-says-w... [5] Holonomic Brain Theory https://en.wikipedia.org/wiki/Holonomic_brain_theory https://en.wikipedia.org/wiki/Holonomic_brain_theory
- bawana 7y ago
  the principle of backprop can be accomplished in structural changes to the synapse. Although that would be analogous to changing the activation function in each 'neuron' of a layer in a convnet-we have just 'factored it out' into a sequence of weights used in the conv layers. Up regulating and down regulating the number of receptors can change how each synapse contributes to a biological net.
ThomPete 7y ago
We need more of this. We’ve sadly lost our general sense of how long fundamental research can take without any promise of success. My biggest respect for people who do this.

nabla9 7y ago

That's the blessing/curse with completely self funded research group. No strict timelines, yearly review, benchmarks or external pressure. There is no hurry to deliver, you can dream on and keep developing your pet idea indefinitely. Nobody can tell you that you were wrong and you are now done.

tux1968 7y ago
That explains why they can continue, but it doesn't make it more likely that the idea will ever bare fruit.
p1esk 7y ago
No strict timelines, yearly review, benchmarks or external pressure. Maybe not imposed on Jeff himself, at least not in this narrow sense, but the people who work for him most likely have all that.

espeed 7y ago

Understanding the underlying structure is key. And the significance of structure -- the ability to see it, understand it, and re-create it -- has been recognized and pointed out by Feynman [1] and some of the world's best programmers [2]. Using data-structure representations that best match reality's underlying structure may be key too. With this in mind I had a thought -- a "dopey idea" in the Steve Jobs to Jony Ive sense of the word [3] -- the brain doesn't do backprop [4] -- at least not directly (as far as we know) -- but neural nets using backprop/gradient-descent work pretty well. Why so? What if the underlying structure these nets are modeling isn't the brain but something else...maybe something more fundamental...like the curvature of spacetime, a path integral representation of information, or some other crude representation of an underlying quantum structure. And if so, what if learning is just the brain optimizing itself in accordance with this structure, and has this idea been explored before? Evidently it has. The idea of the brain as a quantum information processor has been explored to some degree in holonomic brain theory [5]. [1] "What I cannot create, I do not understand" https://en.wikiquote.org/wiki/Richard_Feynman https://en.wikiquote.org/wiki/Richard_Feynman [2] Programmers on Data Structures https://news.ycombinator.com/item?id=20047607 https://news.ycombinator.com/item?id=20047607 [3] Jony Ive talking about Steve Jobs' Dopey Ideas [video] https://www.youtube.com/watch?v=4WNYYCX3Tg0 https://www.youtube.com/watch?v=4WNYYCX3Tg0 [4] Artificial intelligence pioneer says we need to start over https://www.axios.com/artificial-intelligence-pioneer-says-we-need-to-start-over-1513305524-f619efbd-9db0-4947-a9b2-7a4c310a28fe.html https://www.axios.com/artificial-intelligence-pioneer-says-w... [5] Holonomic Brain Theory https://en.wikipedia.org/wiki/Holonomic_brain_theory https://en.wikipedia.org/wiki/Holonomic_brain_theory

bawana 7y ago
the principle of backprop can be accomplished in structural changes to the synapse. Although that would be analogous to changing the activation function in each 'neuron' of a layer in a convnet-we have just 'factored it out' into a sequence of weights used in the conv layers. Up regulating and down regulating the number of receptors can change how each synapse contributes to a biological net.

ThomPete 7y ago

We need more of this. We’ve sadly lost our general sense of how long fundamental research can take without any promise of success. My biggest respect for people who do this.