A synapse is responsible both for storing its weight, and for operating on its weight. Therefore, model size and throughput are interrelated numbers. If a synap

A synapse is responsible both for storing its weight, and for operating on its weight. Therefore, model size and throughput are interrelated numbers. If a synapse in a bee brain fires at 10 Hz on average (my guess), total events per second is 10 times the brain ‘capacity’. Max firing rate will be a lot larger, since most neurons are silent most of the time. In silicon we separate out the storage from the compute. Generally we have much more storage than we have compute, and we route the data to the compute units as we need computation. Therefore ALUs have aggregate throughput of only a small fraction of the model capacity per cycle; at 400,000 cores with, say, 8 16-bit operations per cycle each, that is ‘only’ ~10^6 operations per cycle. A better way to get a measure of computational throughput is to compare those values. Numbers are hard to find, but 10^10 synaptic events/s for a bee and 10^15 edge calculations/s for Cerebras seems the right sort of ballpark. That's a difference of 10^5, a little less than your estimate of 10^7.