7 ms·

Wild that the human brain can squeeze in 100 trillion synapses (very roughly analogous to model parameters / transistors) in a 3lb piece of meat that draws 20 W

• londons_explore 3y ago
  > is it just that we haven't cracked 3-d stacking of integrated circuits yet? Yes. If we could stack transistors in the Z dimension as closely as we do in X and Y, we'd easily exceed the brains density.
• user90131313 3y ago
  I think biggest difference is millions of years evolutionary development. That's a lot of time difference.
• npalli 3y ago
  No neurons are the equivalent of transistors. Synapses are the equivalent of connections between transistors. The total neurons are about 100 Billion but the connections are 100 Trillion. A neuron can have up to 10,000 synapses, while a transistor can have on average only about 3. Impressive nonetheless on power efficiency.
  • highfrequency 3y ago
    # synapses should be analogous to # model parameters no? And # model parameters should be linear in # transistors.
    • phkahler 3y ago
      >> # synapses should be analogous to # model parameters no? I think they're equivalent to a parameter AND the multiplier. Or in analog terms they'd just be a resistor whose value can be changed. Digital stuff is not a good fit for this.
      • 0xcde4c3db 3y ago
        > Or in analog terms they'd just be a resistor whose value can be changed For what it's worth, that's actually a thing (ReRAM/memristors), but I think it got put on the back burner because it requires novel materials and nobody figured out how to cost-effectively scale up the fabrication versus scaling up flash memory. I saw some mention recently that advances in perovskite materials (a big deal lately due to potential solar applications) might revive the concept.
    • retrofrost 3y ago
      We can not even get close to saying our current networks can be even close to synapses in performance or functional because architecturally we still use feedforward networks no recursion, no timing elements, very static connections. Transitors will definitely have some advantages in terms of being able to synchronize information and steps to an infinitely better degree than biological neurons, but as long as we stick with transformers it's the equivalent of trying to get to space by stacking sand, could you get there eventually? Yes, but there's better ways.
  • ksec 3y ago
    So the actual equation should be 100 billion times 100 Trillion times clock speed? ( I have zero ideas about how our brains works ) Even we have 10 times higher clock speed, a trillion transistor GPU is still very slow comparatively speaking ? Surely memory has to come in somewhere?
• layer8 3y ago
  Ignoring for the moment that transistors and synapses are very different in their function, the current in a CPU transistor is in the milliampere range, whereas in the ion channels of a synapse it is in the picoampere range. The voltage differs by roughly a factor of ten. So the wattage differs by a factor of 10^10. One important reason for the difference in current is that transistors need to reliably switch between two reliably distinguishable states, which requires a comparatively high current, whereas synapses are very analog in nature. It may not be possible to reach the brain’s efficiency with deterministic binary logic.
  • TRDRVR 3y ago
    Thank you for this - the name neural networks has made a whole generation of people forget that they have an endocrine system. We know things like sleep, hunger, fear, and stress all impact how we think, yet people want to still build this mental model that synapses are just dot products that either reach an activation threshold or don't.
    • deleted 3y ago
      [deleted]
    • __loam 3y ago
      People will spout off about how machine learning is based on the brain while having no idea how the brain works.
      • gary_0 3y ago
        It is based on the brain, but only in the loosest possible terms; ML is a cargo cult of biology. It's kind of surprising that it works at all.
        • Borg3 3y ago
          It works because well, its actually pretty primitive at its core. Whole learning process is actually pretty brutal. Doing millions of interations w/ random (and semi-random) adjustments.
• marcosdumay 3y ago
  On the power usage, the difference is that those synapses are almost always in stand-by. The equivalent would be a CMOS circuit with a clock of minutes. On the complexity, AFAIK a synapse is way more complex than a transistor. Larger too, if you include its share of the neuron's volume. And yes, the count difference is due to the 3D packing.
  • sroussey 3y ago
    The brain’s 3d packing is via folding. Maybe something similar would be better than just stacking.
• mrtksn 3y ago
  IMHO the breakthrough will come with an analog computer. The current systems simulate stuff through computation using switches. It’s like real sand vs sand simulator in the browser. One spins your fans and drains your battery when showing you 1000 particles acting like sand, the other just obeys laws of physics locally per particle and can do millions of particles much more accurately and with very slight increase in temperature. Of course, analog computations are much less controllable but in this case that’s not a deal breaker.
  • deleted 3y ago
    [deleted]
• zer00eyz 3y ago
  >> 100 trillion synapses (very roughly analogous to transistors?) Not even remotely comparable * its unlikely synapses are binary. Candidly they probably serve more than one purpose. * transistor count is a bad proxy for other reasons. A pipeline to do floats are not going to be useful for fetch from memory. "Where" the density lies is important. * Power: On this front transistors are a joke. * The brain is clockless, and analog... frequency is an interesting metric Binary systems are going to be bad at simulating complex processes. LLM's are a simulation of intelligence, like forecasting is a simulation of weather. Lorenz shows us why simulation of weather has limits, there isnt some magical math that will change those rules for ML to make the leap to "AGI"
  • luyu_wu 3y ago
    Transistor power is really not a joke. Synapses would take far far FAR more power at close to similar frequencies. Biological neurons are incredibly inefficient.
    • goatlover 3y ago
      Biology is incredibly efficient at what it does well though. Thus only 20 watts of brain energy to coordinate everything we do. We didn't evolve to be mentats.
• vasco 3y ago
  We also lose a lot when building computers due to the fact we have to convert the analog world into digital representations. A neural analog computer would be more efficient I think, and due to the non-deterministic nature of AI would probably suit the task as well.
  • gricardo99 3y ago
    interesting point. i know of at least one startup working with that concept[1]. Im sure there are others. 1 - https://www.extropic.ai/ https://www.extropic.ai/
  • sroussey 3y ago
    There are nueromorphic companies like https://rain.ai https://rain.ai
  • barelyauser 3y ago
    Non-deterministic means random. AI or natural I is not random. Analog suffers immensely from noise and it is the reason the brain has such a large number of neurons, part to deal with noise and part to deal with losing some neurons along the way.
    • doug_durham 3y ago
      Non-deterministic doesn't mean random. Random means random. Non-deterministic means that specific inputs don't generate the same outputs. It says nothing about the distribution of the output values. A chaotic system isn't random, its just non-deterministic.
      • dragonwriter 3y ago
        “Nondeternimism” gets used jn a variety of ways both from differing context and conflicting uses in the same context, but chaotic systems are fully deterministic in the most common relevant sense, but highly sensitive to inputs, resulting in even very small uncertainty in inputs to render them largely unpredictable.
    • 3y ago
• fiftyfifty 3y ago
  By some accounts it takes a pretty sizable neural network to simulate a single neuron: https://www.sciencedirect.com/science/article/pii/S0896627321005018 https://www.sciencedirect.com/science/article/pii/S089662732... So we are going to need a lot of computational power to approximate what’s going on in an entire human brain.
• pnjunction 3y ago
  I wonder if this trajectory will only lead to reinventing the biological brain. It is hard to imagine the emergence of consciousness, as we know it, on a fundamentally deterministic system.
• haltIncomplete 3y ago
  Why are static 3D cells going to get us there when other ideas have not? Is it needed to replicate “arbitrary” academic ideas of consciousness (despite our best efforts our models are always approximation) to make a useful machine? “Living things” are not static designs off the drafters table. They’ll never intelligent from their own curiosity, but from ours and the rules we embed. No matter how hard we push puerile hallucinations embedded by Star Trek. It’s still a computer and human agency does not have to bend to it.
• HarHarVeryFunny 3y ago
  > The power efficiency difference may be explainable by the much slower frequency of brain computation (200 Hz vs. 2GHz). Partly, but also because the brain has an asynchronous data-flow design, while the GPU is synchronous, and as you say clocked at a very high frequency. In a clocked design the clock signal needs to be routed to every element on the chip which requires a lot of power, the more so the higher the frequency is. It's a bit like the amount of energy used doing "battle ropes" at the gym. The heavier the ropes (cf more gates the clock is connected to), the more power it takes to move them, and the faster you want to move them (cf faster clock frequency) the more power it takes. In a data-flow design, like the brain, there is no clock. Each neuron fires, or not, independent of what other neurons are doing, based on their own individual inputs. If the inputs are changing (i.e. receiving signal spikes from attached neurons), then at some threshold of spike accumulation the neuron will fire (expending energy). If the inputs are not changing, or at a level below threshold, then the neuron will not fire. To consider the difference, imagine our visual cortex if we're looking at a seagull flying across a blue sky. The seagull represents a tiny part of the visual field, and is the only part that is moving/changing, so there are only a few neurons who's inputs are changing and which themselves will therefore fire and expend energy. The blue sky comprising the rest of the visual field is not changing and we therefore don't expend any energy reprocessing it over and over. In contrast, if you fed a video (frame by frame) of that same visual scene into a CNN being processed on a GPU, then it does not distinguish between what is changing or not, so 95% of the energy processing each frame will be wasted, and this will be repeated frame by frame as long as we're looking at that scene!
  • taktoa 3y ago
    > In a clocked design the clock signal needs to be routed to every element on the chip which requires a lot of power, the more so the higher the frequency is. Clock only needs to be distributed to sequential components like flip flops or SRAMs. The number of clock distribution wire-millimeters in typical chip is dwarfed by the number of data wire-millimeters, and if a neural network is well trained and quantized activations should be random, so number of transitions per clock should be 0.5 (as opposed to 1 for clock wires), meaning that power can't be dominated by clock. The flops that prevent clock skew are a small % of area, so I don't think those can tip the scales either. On the other hand, in asynchronous digital logic you need to have valid bit calculation on every single piece of logic, which seems like a pretty huge overhead to me.
• ordu 3y ago
  > The power efficiency difference may be explainable by the much slower frequency of brain computation (200 Hz vs. 2GHz). Or by a more static design. A GPU can't do a thing without all the weights and shaders. There are benefits of this, you can easily swap one model for another. Human mind from the other hand is not reprogrammable. It can learn new tricks, but you cannot extract a firmware from one person and to upload it to another person. Just imagine if every logical neuron of AI was a real thing, with physical connections to other neurons as inputs. No more need to have a high throughput memory, no more need to have compute units with gigaherz frequency.
• wslh 3y ago
  Neurons are not just linear algebra pieces by the way. This is why there is bibliography that talks about the complexity of a single neuron. So, comparing at the unit level at this point is apples vs. oranges. But, yes, the brain continues to be a surprising machine and ML accomplishements are amazing for that machine.
• orbital-decay 3y ago
  Apples to oranges. Gate count indicates nothing when the architectures are nothing alike. Brain is a spiking network with mutable connectivity, mostly asynchronous. Only the active path is spending energy at a single moment in time, and "compute" is tightly coupled with memory to the point of being indistinguishable. No need to move data anywhere. In contrast, GPUs/TPUs are clocked and run fully connected networks, they have to iterate over humongous data arrays every time. Memory is decoupled from compute due to the semiconductor process differences between the two. As a result, they waste a huge amount of energy just moving data back and forth. Fundamental advancements in SNNs are also required, it's not just about the transistors.
• Traubenfuchs 3y ago
  Neurons and synapses have more than 100 different neurotransmitters and their many receptors, there is reuptake and destructive encyme activity and they are connected up to many thousands of their peer. Every single neuron is a dizzyingly complex machine, employing countless sub machines and processes. You can not reasonably compare this to model parameters or transistors at all.
• ozim 3y ago
  Nature was “building” this stuff for eons. I feel pretty good about our progress in still less than a 100 years.
• 0x000xca0xfe 3y ago
  It has some interesting implications: If we manage to replicate a human-like brain in silicon, it could think multiple million times faster than the biological equivalent. So even if it needs a lot of energy to run, it could "out-think" us by orders of magnitude and would have crazy reaction times. Just imagine a sentient being with the human mental equivalent of millions of years in one year. Crazy. But we know very little about how biological brains actually work and very few connectomes have been fully mapped as of yet. We still cannot fully explain how C. elegans with 302 neurons "thinks". Also every neuron contains the entire genome, which is hundreds of millions to billions of base pairs depending on the animal and we also don't really know how important it is to the function of the brain. So the "state" of biological brains is potentially utterly gigantic even for simple animals.
• agumonkey 3y ago
  Hmm maybe the frequency metric is off. If neurons can derive output from potentially N overlapping / multiplexed inputs .. it's as if the clock was N times higher too.