3 ms·
We are massively far away from modeling the human brain. First of all, no one can agree what level is necessary to model the brain, and that varies tremendously
by tbenst 3y ago
We are massively far away from modeling the human brain. First of all, no one can agree what level is necessary to model the brain, and that varies tremendously by scientific question. Personally, my lower limit would be something like the computational package Neuron which models voltages across axon compartments and distribution of ion channels, My upper limit confidence bound is we don’t care about anything subatomic.
At the upper bound: In molecular dynamics, which is used extensively in modern day neuroscience to understand the function of ion channels and GPCRs, a single H100 can model 70ns/day of compute for 1M atoms. There are 8.64e+13 nanoseconds per day. There are ~10^26 atoms in a human brain. Therefore, an upper limit back of envelope is you need fewer than 10e+26 atoms / 10e+9 atoms * 8.64e+13 ns / 70 ns = 1.23e+29 H100 GPUs.
Calculating the lower bound is more difficult, but let’s start by saying you can get away with a fp16 for each synapse. Storing the weights of that model for 100 trillion synapses is 200 Terabytes, and if you figure weight size * 4 or so to do anything useful then this is in spitting distance. Note that this example lower bound is massively less complex than the Neuron model I suggested, as the entire field of neuromodulators, homeostatic mechanisms, glia, and more are thrown out, which are all important for modeling how the brain works under certain computational regimes.
- Jedd 3y agoThat is a spectacular response! My bio knowledge is very basic, so forgive naiviety in these two questions. First, I'm not asking you to go through the math on the spot, but I'm guessing that lower-bound capability is well understood in 'the field', but is it documented against various species? Perhaps mapping against current / projected GPU/compute systems capabilities? (I know there's a project to model a worm's brain, IIRC down to molecular level. But I'm picturing a 'we are 3 years away from being able to emulate a basset hound, 4 years for a border collie' - that kind of roadmap.) Second, you said upper bound is to ignore sub-atomic. I thought we had proton and electron gradients, at least in metabolism. I believe proton there is a synonym for Hydrogen (atom), but electron would imply some potential need to emulate at sub-atomic? Have I misunderstood the bounding / chemistry involved?
- DoctorOetker 3y agoThe electron and proton are considered parts of the realm in atomic physics still. When physicists speak of sub-atomic they tend to mean any physics below the Aufbau model or electrons and nuclei.
- ftxbro 3y agoFor the lower bound there is a dark horse factor that has spooked Geoffrey Hinton. He thinks that biological brains aren't able to do backpropagation effectively through multiple layers, and so differentiable programming frameworks are much more powerful than what the brain has, at an algorithmic level. In other words, he thinks that computers are able to learn more effectively than any neuron-based biological brain. Of course right now there are caveats. The brain appears to have more 'statistical efficiency' meaning it appears to learn more from less data, and the brain is obviously more energy-efficient. There is also the possibility that Geoffrey Hinton is just wrong.
- DoctorOetker 3y agoCan you cite a reference for Geoffrey Hinton (and or others) on this line of thought on this "dark horse factor"? I think this resonates with a lot of people, having a focal point Schelling point to refer to in discussions would be handy. I believe the situation is a lot more extreme than the absurd efficiency of differentiable programming. I have been meaning to write up (but been too busy to do so) an insight where I believe training can be made ridiculously cheap computationally speaking (in a way that combines with differentiable programming, not replaces it). I am agnostic if this is what the brain does, but wouldn't be surprised at all if the brain does in fact do back-propagation (or uses the insight that I've been meaning to write up).
- moffkalast 3y agoBiological brains also don't really operate layer by layer and can have connections between random neurons, so it's probably a lot more space efficient. Impossible to say if any of that actually matters though.
- ly3xqhl8g9 3y agoWe will never simulate the entire brain atom-by-atom, we won't need to, the same way we never simulate atom-by-atom and we don't even place structural atoms by hand when we build a bridge, a rocket, or a tree house, we can be way more intelligent than that [1]. In the limit, the entire thing could be even more simple than we currently can imagine [2]. But yes, before we start leveraging equations, we must find the principle of gravitation for collective intelligence first [3]. [1] https://en.wikipedia.org/wiki/Hodgkin%E2%80%93Huxley_model https://en.wikipedia.org/wiki/Hodgkin%E2%80%93Huxley_model [2] https://en.wikipedia.org/wiki/Reaction%E2%80%93diffusion_system https://en.wikipedia.org/wiki/Reaction%E2%80%93diffusion_sys... [3] Michael Levin | Cell Intelligence in Physiological and Morphological Spaces, https://www.youtube.com/watch?v=jLiHLDrOTW8 https://www.youtube.com/watch?v=jLiHLDrOTW8