4 ms·

No neurons are the equivalent of transistors. Synapses are the equivalent of connections between transistors. The total neurons are about 100 Billion but the co

by npalli 3y ago

No neurons are the equivalent of transistors. Synapses are the equivalent of connections between transistors. The total neurons are about 100 Billion but the connections are 100 Trillion. A neuron can have up to 10,000 synapses, while a transistor can have on average only about 3. Impressive nonetheless on power efficiency.

highfrequency 3y ago
# synapses should be analogous to # model parameters no? And # model parameters should be linear in # transistors.
- phkahler 3y ago
  >> # synapses should be analogous to # model parameters no? I think they're equivalent to a parameter AND the multiplier. Or in analog terms they'd just be a resistor whose value can be changed. Digital stuff is not a good fit for this.
  - 0xcde4c3db 3y ago
    > Or in analog terms they'd just be a resistor whose value can be changed For what it's worth, that's actually a thing (ReRAM/memristors), but I think it got put on the back burner because it requires novel materials and nobody figured out how to cost-effectively scale up the fabrication versus scaling up flash memory. I saw some mention recently that advances in perovskite materials (a big deal lately due to potential solar applications) might revive the concept.
- retrofrost 3y ago
  We can not even get close to saying our current networks can be even close to synapses in performance or functional because architecturally we still use feedforward networks no recursion, no timing elements, very static connections. Transitors will definitely have some advantages in terms of being able to synchronize information and steps to an infinitely better degree than biological neurons, but as long as we stick with transformers it's the equivalent of trying to get to space by stacking sand, could you get there eventually? Yes, but there's better ways.
ksec 3y ago
So the actual equation should be 100 billion times 100 Trillion times clock speed? ( I have zero ideas about how our brains works ) Even we have 10 times higher clock speed, a trillion transistor GPU is still very slow comparatively speaking ? Surely memory has to come in somewhere?

by npalli 3y ago

highfrequency 3y ago

# synapses should be analogous to # model parameters no? And # model parameters should be linear in # transistors.

phkahler 3y ago
>> # synapses should be analogous to # model parameters no? I think they're equivalent to a parameter AND the multiplier. Or in analog terms they'd just be a resistor whose value can be changed. Digital stuff is not a good fit for this.
- 0xcde4c3db 3y ago
  > Or in analog terms they'd just be a resistor whose value can be changed For what it's worth, that's actually a thing (ReRAM/memristors), but I think it got put on the back burner because it requires novel materials and nobody figured out how to cost-effectively scale up the fabrication versus scaling up flash memory. I saw some mention recently that advances in perovskite materials (a big deal lately due to potential solar applications) might revive the concept.
retrofrost 3y ago
We can not even get close to saying our current networks can be even close to synapses in performance or functional because architecturally we still use feedforward networks no recursion, no timing elements, very static connections. Transitors will definitely have some advantages in terms of being able to synchronize information and steps to an infinitely better degree than biological neurons, but as long as we stick with transformers it's the equivalent of trying to get to space by stacking sand, could you get there eventually? Yes, but there's better ways.

ksec 3y ago

So the actual equation should be 100 billion times 100 Trillion times clock speed? ( I have zero ideas about how our brains works ) Even we have 10 times higher clock speed, a trillion transistor GPU is still very slow comparatively speaking ? Surely memory has to come in somewhere?