5 ms·
Ignoring for the moment that transistors and synapses are very different in their function, the current in a CPU transistor is in the milliampere range, whereas
by layer8 3y ago
Ignoring for the moment that transistors and synapses are very different in their function, the current in a CPU transistor is in the milliampere range, whereas in the ion channels of a synapse it is in the picoampere range. The voltage differs by roughly a factor of ten. So the wattage differs by a factor of 10^10.
One important reason for the difference in current is that transistors need to reliably switch between two reliably distinguishable states, which requires a comparatively high current, whereas synapses are very analog in nature. It may not be possible to reach the brain’s efficiency with deterministic binary logic.
- TRDRVR 3y agoThank you for this - the name neural networks has made a whole generation of people forget that they have an endocrine system. We know things like sleep, hunger, fear, and stress all impact how we think, yet people want to still build this mental model that synapses are just dot products that either reach an activation threshold or don't.
- deleted 3y ago[deleted]
- __loam 3y agoPeople will spout off about how machine learning is based on the brain while having no idea how the brain works.
- gary_0 3y agoIt is based on the brain, but only in the loosest possible terms; ML is a cargo cult of biology. It's kind of surprising that it works at all.
- Borg3 3y agoIt works because well, its actually pretty primitive at its core. Whole learning process is actually pretty brutal. Doing millions of interations w/ random (and semi-random) adjustments.
- Terr_ 3y agoI think I've fallen into the "it's just a very fancy kind of lossy compression" camp.
- eli_gottlieb 3y agoHonestly once you understand maximum-likelihood estimation, empirical risk minimization, automatic differentiation, and stochastic gradient descent, it's not that much of a surprise it works.
- cosmojg 3y agoFortunately for academics looking for a new start in industry, this widespread misunderstanding has made it only far too easy to transition from a slow-paced career in computational neuroscience to an overwhelmingly lucrative one in machine learning!
- goatlover 3y agoThere have been people on HN arguing that the human brain is a biological LLM, because they can't think of any other way it could work, as if we evolved to generate the next token, instead of fitness as organisms in the real world. Where things like eating, sleeping, shelter, avoiding danger, social bonds, reproduction and child rearing are important. Things that require a body.
- kungito 3y agoI'm one of those people. To me those things only sounded like a different prompt. Priorities set for the llm
- TRDRVR 3y agoHere's the hubris of thinking that way: I would imagine the baseline assumption of your thinking is that things like sleep and emotions are a 'bug' in terms of cognition (or at the very least, 'prompts' that are optional). Said differently, the assumption is that with the right engineer, you could reach human-parity cognition with a model that doesn't sleep or feel emotions (after all what's the point of an LLM if it gets tired and doesn't want to answer your questions sometimes? Or even worse knowingly deceives you because it is mad at you or prejudiced against you). The problem with that assumption is that as far as we can tell, every being with even the slightest amount of cognition sleeps in some form and has something akin to emotional states. As far as we can prove, sleep and emotions are necessary preconditions to cognition. A worldview where the 'good' parts of the brain (reasoning and logic) are replicated in LLM but the 'bad' parts (sleep, hunger, emotions, etc.) are not is likely an incomplete model.
- kelseyfrog 3y agoDo airplanes need sleep because they fly like birds who also require sleep?
- goatlover 3y agoBirds flap their wings and maneuver differently. They don't fly the same way.
- philipkglass 3y agoIgnoring for the moment that transistors and synapses are very different in their function, the current in a CPU transistor is in the milliampere range... That seems implausible. Apple's M2 has 20 billion transistors and draws 15 watts at full power [1]. Even assuming that 90% of those transistors are for cache and not logic, that would still be 2 billion logic transistors * 1 milliampere = 2 million amperes at full power. That would imply a voltage of 7.5 microvolts, which is far too low for silicon transistors. [1] https://www.anandtech.com/show/17431/apple-announces-m2-soc-apple-silicon-updated-for-2022 https://www.anandtech.com/show/17431/apple-announces-m2-soc-...
- fsckboy 3y ago>the current in a CPU transistor is in the milliampere range ? you sure about that? in a single transistor? over what time period, more than nanoseconds? milliamps is huge, and there are millions of transistors on a single chip these days, and with voltage drops of ... 3V? .7V? you're talking major power. FETs should be operating on field more than flow, though there is some capacitive charge/discharge.
- Ductapemaster 3y agoSingle transistors in modern processes switch currents orders of magnitude lower than milliamps. More like micro- to picoamps. There's leakage to account for too as features get smaller and smaller due to tunneling and other effects, but still in aggregate the current per transistor is tiny. Also the transistors are working at 1V or lower, but as you say they are FETs and don't have the same Vbe drop as a BJT.
- layer8 3y agoYou are right, I mixed this up. If you take a CPU running at 100 W with 10 billion transistors (not quite realistically assumed to all be wired in parallel) at 1 V, you would get an average of 0.01 microamps. So the factor would reduce to roughly 10^5.
- nuancebydefault 3y agoWait a minute, a lot of those transistors are switching the same currents since they are in series. Also, FETs only draw most current while switching, so in between switches there's almost no flow of electrons. So in fact you cannot calculate things that way.
- layer8 3y agoYes, as I said the parallel assumption is not quite realistic, and the number is an average, covering all states a transistor may be in. So it amounts to a rough lower bound for when a transistor is switching.
- jvanderbot 3y agoA simple discretization of the various levels of signal at each input/output, a discretization to handle time-of-propagation (which is almost surely part of the computation just because it _can be_ and nature probably hijacks all mechanisms), and a further discretization to handle the various serum levels in the brain, which are either inputs, outputs, or probably both. Just add a factor 2^D transistors for each original "brain transistor" and re-run your hardware. Hope field effects don't count, and cross your fingers that neurons are idempotent! Easy! /s Modelling an analog system in digital will always have a combinatorial curse of dimensionality. Modelling a biological system is so insanely complex I can't even begin to think about it.
- omikun 3y agoA single precision flop is in the order of pJ. [1] A transistor would be much less. [1] https://arxiv.org/pdf/1809.09206.pdf https://arxiv.org/pdf/1809.09206.pdf
- whiplash451 3y agoHow do you get to 10^10? I might be missing a fundamental of physics here (asking genuinely).
- CuriouslyC 3y agoThe more of our logic we can implement with addition, the more can be offloaded to noisy analog systems with approximate computing. It would be funny if model temperature stopped being metaphorical.
- eli_gottlieb 3y agoSynapses are very analog, but then the neuronal soma (cell body) and axon are pretty pulsatile again. A spike is fired or it isn't!