5 ms·

I honestly think we will one day look back and see this as taking a hammer to a screw. Sure it will work, but its a horribly brute-force way of doing it. The br

I honestly think we will one day look back and see this as taking a hammer to a screw. Sure it will work, but its a horribly brute-force way of doing it. The brain almost certainly processes language in a more elegant way that, once understood, certainly won't take up such raw computational power. The brainwaves associated with a wakeful state (and thus generating and parsing language) happen around 40 Hz. Our machines are running in the GHz region. We are missing some fundamental insight. At the risk of ending up front and center in the hall of prediction hubris, here is my bold prediction: By 2040 it will be possible to do what these TPU's are doing today on your consumer grade computing device, but it won't be because of a dramatic increase in the computing resources available to such devices.

voz_ 5y ago
But brains are trained from millions of generations across the lifetime of the earth (or from when life started, if you ignore the environment and context needed to start life). You are conflating the training with the inference.
- andbberger 5y ago
  at least in the visual cortex, synaptic weights are not genetically encoded in humans. they are in drosophila though
  - whimsicalism 5y ago
    Source? I am skeptical that we even can know that because so much can be implicitly encoded.
    - andbberger 5y ago
      I do not have a single source, that result is represented across thousands of drosophila papers. There was a famous paper were they showed that fruit flies bred in the dark for many generations showed no significant optic lobe tuning differences to wild type flies, but I don't remember who did that. The extreme plasticity of the human visual cortex is widely documented (studies in blind people, etc).
      - whimsicalism 5y ago
        > The extreme plasticity of the human visual cortex is widely documented (studies in blind people, etc). I think there is a big gap between the two claims - 1. Human visual cortex is very plastic 2. There is no inductive bias in the visual part of the brain that has been "trained" by millions of years of evolution. I trust you are correct on the former, but the latter does not seem to follow from the former.
        andbberger 5y ago
        Certainly there is some inductive bias. Neuroanatomy is broadly similar human to human which constraints connectivity. There is an energetic penalty for excess wiring - evolution drives neuron placement to minimize wiring length (eg topographic layout of layers in cortex), which further constraints connectivity. In general, there is a huge body of evidence supporting the hypothesis that organisms are highly tuned to their environments. My point is that in contrast to fruit flies, humans (well, mammals) raised in environments with non-natural scene statistics show significant differences in coding in the visual cortex. Here are some random references I dredged up: https://www.ncbi.nlm.nih.gov/pmc/articles/PMC4210638/ https://www.ncbi.nlm.nih.gov/pmc/articles/PMC4210638/ https://www.sciencedirect.com/science/article/pii/S096098222031589X https://www.sciencedirect.com/science/article/pii/S096098222... It's not hard to see why this might be selected for in mammals. The historic range of D. melanogaster is just sub-saharan Africa, whereas the historic range of humans is huge, and we are primarily visual organisms. So it would be advantageous to have different tuning if you lived in the desert vs the forest. Of course you have to look at the range and divergence times of the full phylogeny but I am getting distracted from my actual research. It's a very interesting line of thinking though. I was quite surprised to learn things were hardcoded in flies when I joined a fly vision lab. https://academic.oup.com/gbe/article/11/3/844/5304659 https://academic.oup.com/gbe/article/11/3/844/5304659
modeless 5y ago
I agree that this stuff will be orders of magnitude more efficient once we figure it out. But I think brute forcing it first and optimizing later is a fine way to do it. Maybe even the only way to make progress on a problem like this. And even in a hypothetical future where you can run human level intelligence on your smartphone, supercomputers will be that much more intelligent still, so it's not like building them will be pointless.
- IIAOPSW 5y ago
  I'll tip my hand further. I think the current approach to artificial neural networks is inherently flawed because it simplifies away pertinent features like firing rates, propagation delay, and standing waves (aka brain waves). I also think trying to get AI to "understand" the irl world when it doesn't live in it and we haven't codified our own natural language is too far a leap. We might have more success making an AI that is conversational in a form of newspeak dedicated to a specific problem domain. Eg an intelligence with the body of a brokerage account that only knows how to speak about money (thus rendering most of humanity redundant (zing!)). Here's something I wrote elsewhere about the accidental path to machine intelligence. "You are whatever you get in return when you ask yourself who you are". Literally, if you trace the referent objects, "you" refers to the same entity as the voice which responds to your query. But also literally, "you" are the standing waves of feedback loops in the brain which represent thoughts. Rather subtly, I've given an understanding of consciousness which is entirely defined and constrained within a communicative system of sending and receiving messages. The underlying language within which the consciousness exists may have a very minimal grounding in some physical reality. (I'm truncating a ton of justification and background to avoid a deep rabbit hole and get to the point). Here is a highly speculative example of how consciousness-like intelligence could emerge accidentally in the form of trading bots. It starts with two ingredients: the shared reality of the order book and independent trading bots with the objective of "make more money". At first the algorithms are dumb curve optimizers. They see whatever patterns they can find in the order books and correlate them with outcomes, reacting appropriately by placing their own orders. After a while, as the data sets and internal models grow, the algorithms are implicitly calculating the reactions of other trading bots in their extrapolation. Soon after that, the orders being placed in the order books are effectively messages intended for other trading bots, hoping to illicit a certain reaction. The order book effectively becomes a language grounded in an economic reality, filled with offers that are communicative but not necessarily intended to execute. The emergent language of the order book grows in sophistication to the point where the bots are talking about past instances and hypothetical instances of things said in orderbook speak. This all happens right under our nose in perhaps non-obvious ways, such as in the exact number of cents on a bid/ask price. Other times it looks like our bots have reinvented "painting the tape" and other forms of financial communication we've deemed "market manipulation". We're proud of our silicon brain child for figuring out how to do that on its own. They grow up so fast. Eventually, in the process of optimization, the trading bots internal model of its "order book reality" gets to a point of sophistication where it has to model other bots modelling itself modelling other bots modelling itself...to whatever layer of recursive depth it can marshal. Thus the feedback loop effectively closes and something akin to circular loop brain waves emerge. By this point no one really understands why the trading bots make the trades they do. I can no longer tell you what a trading bot is in simple terms like "its a dumb curve optimizer trying to maximize money". Rather, it can only be understood as a form of consciousness which has emerged: "The trading bot is whatever the trading bot gets in return when it simulates placing an order asking what it is."
6gvONxR4sf7o 5y ago
I'm sure we will end up with better solutions, but it's a lot easier to study a thing when you have an introspectable working model. I expect that many AI patterns will be that we'll build it, then optimize it and learn how and why it works, taking the art and science into engineering piece by piece.
ot 5y ago
> The brainwaves associated with a wakeful state (and thus generating and parsing language) happen around 40 Hz. Our machines are running in the GHz region. I don't disagree with the larger point, but each neuron is like a small independent analog computer, so you have tens of billions of (specialized) cores running at 40Hz, vs thousands of cores at ~GHz.
sytelus 5y ago
Human brain has very likely 10^15 synapses so these models are still behind at least 3 orders of magnitude. To put in perspective, if you want to simulate “network” of human brain, you probably need an entire data center worth of GPUs.
- orbifold 5y ago
  That is assuming you restrict yourself to a radical simplified model of each neuron. The actual biochemical processes going on in a single synaptic density go way beyond something that could be represented by a scalar weight that assemble into a linear transformation. Instead we are talking about 10^15 dynamical systems. This then still neglects most of the time /spatial extended nature of neurons, the fact that calcium stores in the endoplasmatic reticulum play a role etc. Some of this might be only relevant to learning, but non-linear dendritic processing is believed to be relevant for example for inference aswell.
azinman2 5y ago
The brain is massively parallel (itself an understatement). Computers are not.

voz_ 5y ago

But brains are trained from millions of generations across the lifetime of the earth (or from when life started, if you ignore the environment and context needed to start life). You are conflating the training with the inference.

andbberger 5y ago
at least in the visual cortex, synaptic weights are not genetically encoded in humans. they are in drosophila though
- whimsicalism 5y ago
  Source? I am skeptical that we even can know that because so much can be implicitly encoded.
  - andbberger 5y ago
    I do not have a single source, that result is represented across thousands of drosophila papers. There was a famous paper were they showed that fruit flies bred in the dark for many generations showed no significant optic lobe tuning differences to wild type flies, but I don't remember who did that. The extreme plasticity of the human visual cortex is widely documented (studies in blind people, etc).
    - whimsicalism 5y ago
      > The extreme plasticity of the human visual cortex is widely documented (studies in blind people, etc). I think there is a big gap between the two claims - 1. Human visual cortex is very plastic 2. There is no inductive bias in the visual part of the brain that has been "trained" by millions of years of evolution. I trust you are correct on the former, but the latter does not seem to follow from the former.
      - andbberger 5y ago
        Certainly there is some inductive bias. Neuroanatomy is broadly similar human to human which constraints connectivity. There is an energetic penalty for excess wiring - evolution drives neuron placement to minimize wiring length (eg topographic layout of layers in cortex), which further constraints connectivity. In general, there is a huge body of evidence supporting the hypothesis that organisms are highly tuned to their environments. My point is that in contrast to fruit flies, humans (well, mammals) raised in environments with non-natural scene statistics show significant differences in coding in the visual cortex. Here are some random references I dredged up: https://www.ncbi.nlm.nih.gov/pmc/articles/PMC4210638/ https://www.ncbi.nlm.nih.gov/pmc/articles/PMC4210638/ https://www.sciencedirect.com/science/article/pii/S096098222031589X https://www.sciencedirect.com/science/article/pii/S096098222... It's not hard to see why this might be selected for in mammals. The historic range of D. melanogaster is just sub-saharan Africa, whereas the historic range of humans is huge, and we are primarily visual organisms. So it would be advantageous to have different tuning if you lived in the desert vs the forest. Of course you have to look at the range and divergence times of the full phylogeny but I am getting distracted from my actual research. It's a very interesting line of thinking though. I was quite surprised to learn things were hardcoded in flies when I joined a fly vision lab. https://academic.oup.com/gbe/article/11/3/844/5304659 https://academic.oup.com/gbe/article/11/3/844/5304659

modeless 5y ago

I agree that this stuff will be orders of magnitude more efficient once we figure it out. But I think brute forcing it first and optimizing later is a fine way to do it. Maybe even the only way to make progress on a problem like this. And even in a hypothetical future where you can run human level intelligence on your smartphone, supercomputers will be that much more intelligent still, so it's not like building them will be pointless.

IIAOPSW 5y ago
I'll tip my hand further. I think the current approach to artificial neural networks is inherently flawed because it simplifies away pertinent features like firing rates, propagation delay, and standing waves (aka brain waves). I also think trying to get AI to "understand" the irl world when it doesn't live in it and we haven't codified our own natural language is too far a leap. We might have more success making an AI that is conversational in a form of newspeak dedicated to a specific problem domain. Eg an intelligence with the body of a brokerage account that only knows how to speak about money (thus rendering most of humanity redundant (zing!)). Here's something I wrote elsewhere about the accidental path to machine intelligence. "You are whatever you get in return when you ask yourself who you are". Literally, if you trace the referent objects, "you" refers to the same entity as the voice which responds to your query. But also literally, "you" are the standing waves of feedback loops in the brain which represent thoughts. Rather subtly, I've given an understanding of consciousness which is entirely defined and constrained within a communicative system of sending and receiving messages. The underlying language within which the consciousness exists may have a very minimal grounding in some physical reality. (I'm truncating a ton of justification and background to avoid a deep rabbit hole and get to the point). Here is a highly speculative example of how consciousness-like intelligence could emerge accidentally in the form of trading bots. It starts with two ingredients: the shared reality of the order book and independent trading bots with the objective of "make more money". At first the algorithms are dumb curve optimizers. They see whatever patterns they can find in the order books and correlate them with outcomes, reacting appropriately by placing their own orders. After a while, as the data sets and internal models grow, the algorithms are implicitly calculating the reactions of other trading bots in their extrapolation. Soon after that, the orders being placed in the order books are effectively messages intended for other trading bots, hoping to illicit a certain reaction. The order book effectively becomes a language grounded in an economic reality, filled with offers that are communicative but not necessarily intended to execute. The emergent language of the order book grows in sophistication to the point where the bots are talking about past instances and hypothetical instances of things said in orderbook speak. This all happens right under our nose in perhaps non-obvious ways, such as in the exact number of cents on a bid/ask price. Other times it looks like our bots have reinvented "painting the tape" and other forms of financial communication we've deemed "market manipulation". We're proud of our silicon brain child for figuring out how to do that on its own. They grow up so fast. Eventually, in the process of optimization, the trading bots internal model of its "order book reality" gets to a point of sophistication where it has to model other bots modelling itself modelling other bots modelling itself...to whatever layer of recursive depth it can marshal. Thus the feedback loop effectively closes and something akin to circular loop brain waves emerge. By this point no one really understands why the trading bots make the trades they do. I can no longer tell you what a trading bot is in simple terms like "its a dumb curve optimizer trying to maximize money". Rather, it can only be understood as a form of consciousness which has emerged: "The trading bot is whatever the trading bot gets in return when it simulates placing an order asking what it is."

6gvONxR4sf7o 5y ago

I'm sure we will end up with better solutions, but it's a lot easier to study a thing when you have an introspectable working model. I expect that many AI patterns will be that we'll build it, then optimize it and learn how and why it works, taking the art and science into engineering piece by piece.

ot 5y ago

> The brainwaves associated with a wakeful state (and thus generating and parsing language) happen around 40 Hz. Our machines are running in the GHz region. I don't disagree with the larger point, but each neuron is like a small independent analog computer, so you have tens of billions of (specialized) cores running at 40Hz, vs thousands of cores at ~GHz.

sytelus 5y ago

Human brain has very likely 10^15 synapses so these models are still behind at least 3 orders of magnitude. To put in perspective, if you want to simulate “network” of human brain, you probably need an entire data center worth of GPUs.

orbifold 5y ago
That is assuming you restrict yourself to a radical simplified model of each neuron. The actual biochemical processes going on in a single synaptic density go way beyond something that could be represented by a scalar weight that assemble into a linear transformation. Instead we are talking about 10^15 dynamical systems. This then still neglects most of the time /spatial extended nature of neurons, the fact that calcium stores in the endoplasmatic reticulum play a role etc. Some of this might be only relevant to learning, but non-linear dendritic processing is believed to be relevant for example for inference aswell.

azinman2 5y ago

The brain is massively parallel (itself an understatement). Computers are not.