9 ms·
There are far more transistors in this chip than neurons in the human brain. In 100 of these chips, there are more transistors than there are synapses in the hu
by modeless 7y ago
There are far more transistors in this chip than neurons in the human brain. In 100 of these chips, there are more transistors than there are synapses in the human brain.
I don't mean to suggest that transistors are equivalent to neurons or synapses; clearly they are very different things (though it is not clear that neurons are much more computationally powerful than transistors, despite assertions from many people that this is the case). But I think it is still useful to compare the complexity of structure of chips vs. the human brain. We are finally approaching the same order of magnitude of complexity in structure.
Also note that this is not manufactured on TSMC's highest density process. Assuming TSMC's 3nm process development is successful, that will probably be 6x denser than the 16nm process used here.
- jdsully 7y agoI haven’t seen a perceptron circuit done in less than 4 transistors plus a few diodes. However these chips are almost certainly digital and will use a lot more to perform the floating point math. Single transistors can’t even compute our simplified model of a neuron let alone the complexities of the real thing.
- mlyle 7y agoIf we're going to talk about net computational power per unit (neuron/transistor), we also need to note that single transistors do stuff a whole lot faster than biological neurons, too. If we had neurons that were 1,000x faster, could we use fewer of them for the same task?
- jacquesm 7y agoWell, we can turn that argument around: if an artificial brain could be faster than a real one, does that mean that right now we have slow artificial brains that are just as good as the real ones? The answer being an obvious 'no' may help answer your question.
- mlyle 7y agoJust because we don't know a program that creates a decent artificial brain doesn't mean that such a program doesn't exist for the hardware we have. We're almost certainly going to be able to make speed/size trade-offs in AGI, because we're used to making them in all other computing. Even biological systems make these types of trade-offs. We're beings optimized for the environment we evolved in, not optimized for "smarts". There's many "choices" in energy balance, reaction time, etc, that have been made. Transistors are really powerful and fast computing devices; really our use of them in digital logic in some ways greatly under-utilizes them for ease in design and predictability. Saying that it takes 4 transistors to make a really simple neuron analog in a circuit undersells that those 4 transistors can operate many, many orders of magnitude faster.
- jcims 7y agoFaster than what? There are a hundred trillion atoms in a neuron operating in a coordinated fashion to create its global behavior in the context of the brain. How many interactions between those components occur in a billionth of a second? Many.
- tlb 7y agoIt would be very hard to tell if an AGI running 1000x slower than a brain were intelligent. Humans babies don’t talk for about a year, but nobody has tried firing up a putative slow AI and waiting for 1000 years.
- jacquesm 7y agoThen let's run 1000 of them in parallel, right? You're on to the exact point I'm trying to make. The whole idea that 'just' complexity or 'just' speed is enough to make something intelligent that is not yet intelligent is off-base to a degree that resorting to metaphors to try to explain why also breaks down. I liken intelligence to life: once it exists it is obvious and self-perpetrating but before it exists it is non-obvious and no amount of fantasizing about what it would take to make it will get you incrementally closer until one day you've got it. But that day will start out just like the thousands of days before it. It's a quantum leap, a difference in quality, not an incremental one. Yes, enough order of magnitude change can make a quantitative change into a qualitative one. But so far the proof that this holds for intelligence is eluding us. Maybe we're just not intelligent enough, in the same way that you can't 100% describe a system from within the system.
- trott 7y ago> If we had neurons that were 1,000x faster, could we use fewer of them for the same task? Absolutely. In fact, we do: in batched processing (pervasive in deep learning today), CNNs and other weight sharing schemes (e.g. transformers) the same "neurons" get re-used many times.
- mkl 7y agoThose aren't neurons, though. Biology is different, synaptic connections have to physically fit, etc. Individual neurons serve multiple purposes, but I don't think we know if their speed is a limiting factor in that.
- jacquesm 7y agoA few buckets of sand also contain more elements than there are synapses in the human brain. They clearly lack any structure, and so are computationally stuck at a big fat '0' no matter what you do with the sand (unless you want to turn it into a giant abacus or turn it into integrated circuits). Where on the scale between a few buckets of sand and an actual working human brain this chip is is not so much a function of it's structure but of what it does, and the brain's finer structures are so complex that even modelling a few neurons is going to crush that complexity downwards by so many orders of magnitude that we simply hope that we are not accidentally throwing out the useful bits in the simulation. That's a roundabout way of saying that I think that: > We are finally approaching the same order of magnitude of complexity in structure. Is not necessarily true in a way that is relevant. The parts count or basic interconnection may have nothing to do with how the brain is connected internally, nor with how it functions at the lowest levels.
- Veedrac 7y ago> A few buckets of sand also contain more elements than there are synapses in the human brain. Actually you'd need about a thousand buckets. https://www.quora.com/How-many-grains-of-sand-could-fit-in-a-cylindrical-bucket-1-5-feet-tall-and-1-foot-in-diameter https://www.quora.com/How-many-grains-of-sand-could-fit-in-a... --- Ultimately I think these numbers do matter. As chips approach the size of human brains, only 1,000,000,000x faster, you remove the question of engineering and scale, and are left with just research.
- jacquesm 7y agoUsing fine sand I make that out to be about 6 buckets, give or take, besides, this isn't about whether it is 6, 10 or even 100 (or 1000) buckets, it is about how the number itself does not make a huge difference. Before if you wanted 1.2T transistors you needed 50 or so regular large dies, and presumably there have already been dies with more than the 21 billion that I'm referring to here (the NVidia 5K cores chip). If that was all it took then we'd have had 'human brains' in a box long ago, after all, 50 GPUs is something that isn't all that rare in plenty of institutions.
- neural_thing 7y agoI have written a book about the true computational power of pyramidal neurons. Spoiler: they are A LOT more powerful than transistors. The book is free @ http://www.corticalcircuitry.com/ http://www.corticalcircuitry.com/
- thelittleone 7y agoI'm a few pages in and with found it fascinating, humorous, compelling and understandable even with zero background in the field. Thanks for sharing.
- neural_thing 7y agoThank you for the kind words!
- plainOldText 7y agoI have started reading the first two chapters of your book and I really like it. It's well written and easy to understand. If it continues to be this wonderful for the remaining chapters I will buy it as well. Thanks for writing it.
- christoph 7y agoI made the mistake of starting to read this in bed last night and got totally engrossed. Looking forward to finishing it off over the next few days. Many thanks for writing this and sharing it.
- berdon 7y agoYou're mixing terminology. A neuron is not a synapse. - There are ~100 billion neurons in the human brain - There are 100-1000 trillion synapses in the human brain There's strong evidence to suggest that each synapse is more akin to a digital perceptron than a neuron is. Synaptic cleft distance, transmitter types, dendritic structure, reuptake, etc are all factors that can allow for some level of long-term storage and mediation of subsequent neural activation. [Edit] Maybe not "mixing" but seemingly comparing apples and oranges.
- Simon321 7y agoIt takes around 15-20 transistors to make a simple neuron simulation: https://www.quora.com/How-many-transistors-can-be-used-to-replace-one-neuro-cell https://www.quora.com/How-many-transistors-can-be-used-to-re... But: "Synapses are usually separately modeled in transistors (they are not part of the neuron circuits described above) and dramatically add to the transistor count." "[A neuron] has on average 7000 synaptic connections to other neurons." So we're not there yet, but it's definitely an improvement!