4 ms·

And yet here we are scaling transformer models.

by ip26 3y ago

kristopolous 3y ago
by that logic, the SAGE computer from 1957, being the biggest computer ever built, should be the one to run it on.
- inglor_cz 3y ago
  The difference is that in mammals, neurons are approximately the same size, so bigger brains tend to have more connections and exhibit more complexity. (To be sure, that is a correlation, not an iron law.) In computers, the basic building block, an individual transistor, has shrunk by orders of magnitude since 1957. If we, theoretically, built a computer of the size of the SAGE computer from contemporary M2 chips, it would be very computationally powerful indeed.
  - kristopolous 3y ago
    Correct. You can't use a single number to accurately measure something that takes more than a single number to express. It's reductionist and impossible. All kinds of animals are far better then us at specific core brain functions with way smaller brains. Mantis shrimps are way better at vision than we are and lyre birds are way better at vocalization. (https://m.youtube.com/watch?v=VjE0Kdfos4Y https://m.youtube.com/watch?v=VjE0Kdfos4Y) You can find all kinds of vastly superior aspects of systems in the animal world. Ants, termites and bees for instance, are probably better at socializing. Some supercolonies span thousands of miles with hundreds of millions of ants - rivaling the size of human countries. Ants, 0.25mg brains. So brain size doesn't seem that important in organizing or building complex networks. We're certainly not 5.2 million times better (which is our difference in brain mass) Ants even "farm" with things like aphids and keep those colonies alive. Look it up. It's fascinating what they can do with 0.25mg. We're the only animals that have built rocketships and computers - so there's certainly something there but it's fleeting and complicated. Reducing it to a single measurement is inherently inaccurate in the same way you cannot describe the material, color, shape and volume of 3 dimensional objects with 1 number. Brain size alone cannot be a complete or useful metric.

And yet here we are scaling transformer models.

by ip26 3y ago

And yet here we are scaling transformer models.

kristopolous 3y ago

by that logic, the SAGE computer from 1957, being the biggest computer ever built, should be the one to run it on.

inglor_cz 3y ago
The difference is that in mammals, neurons are approximately the same size, so bigger brains tend to have more connections and exhibit more complexity. (To be sure, that is a correlation, not an iron law.) In computers, the basic building block, an individual transistor, has shrunk by orders of magnitude since 1957. If we, theoretically, built a computer of the size of the SAGE computer from contemporary M2 chips, it would be very computationally powerful indeed.
- kristopolous 3y ago
  Correct. You can't use a single number to accurately measure something that takes more than a single number to express. It's reductionist and impossible. All kinds of animals are far better then us at specific core brain functions with way smaller brains. Mantis shrimps are way better at vision than we are and lyre birds are way better at vocalization. (https://m.youtube.com/watch?v=VjE0Kdfos4Y https://m.youtube.com/watch?v=VjE0Kdfos4Y) You can find all kinds of vastly superior aspects of systems in the animal world. Ants, termites and bees for instance, are probably better at socializing. Some supercolonies span thousands of miles with hundreds of millions of ants - rivaling the size of human countries. Ants, 0.25mg brains. So brain size doesn't seem that important in organizing or building complex networks. We're certainly not 5.2 million times better (which is our difference in brain mass) Ants even "farm" with things like aphids and keep those colonies alive. Look it up. It's fascinating what they can do with 0.25mg. We're the only animals that have built rocketships and computers - so there's certainly something there but it's fleeting and complicated. Reducing it to a single measurement is inherently inaccurate in the same way you cannot describe the material, color, shape and volume of 3 dimensional objects with 1 number. Brain size alone cannot be a complete or useful metric.