3 ms·
I think the trick is, our computational steps are massively parallel. Take the visual system. By the time you notice - consciously - the pothole you're about to
by TeMPOraL 4y ago
I think the trick is, our computational steps are massively parallel. Take the visual system. By the time you notice - consciously - the pothole you're about to drive into, it's already processed into high-level concepts.
Our visual senses aren't camera feeds - the image we are aware of isn't raw input from the eyes, but rather it's generated out of some combination of raw inputs, low-level hacks to hide limited FOV and discontinuities caused by saccades, and higher-level interpretations based on how we feel, what seems to fit the context based on our prior experiences, etc.
I think it makes sense to assume that this process has many extra off-ramps that can trigger high-level reactions while bypassing conscious awareness. I.e. imagine that half-way through the post-processing stack, the relevant parts of the brain are somewhat convinced you're looking at a fast-moving object about to hit you in the face. There's a point at which it makes sense for the brain to make your body start a dodging move, even if it may be a misprediction, rather than waste a few dozen milliseconds to make sure, at the risk of getting a ball (or a rock, or a fist) up your nose.
- marcosdumay 4y ago> I think the trick is, our computational steps are massively parallel. Yes, one trick is that. Our computers have to serialize everything. We don't know how to build or program massively parallel computers with distributed memory. But the GP already touched on that point.
- MichaelZuo 4y agoWell it seems like Nvidia is taking some baby steps in that direction with the 'Grace Hopper Superchip'.
- daveguy 4y agoThis is an extremely important point. "10 computational steps" is obviously underestimated. Each neuron can communicate with 10s to 100s of additional neurons, so if you're talking about 10 ms per step 100ms being 10 steps you are missing the fact that each step can be an exponential increase in the amount of computation involved. This is one of the main reasons I believe we are nowhere near human or even mouse level AGI. Many of the simple neuron count metrics do not include the synapse counts or the incredible amount of parallel information that is processed. We have network interfaces with 100-400 Gb throughput whereas the communications between synapses represents exabits of throughput between computational units. We are very far from AGI just on a fundamental processing scale. Not to mention we know very little about the algorithms involved.
- TeMPOraL 4y agoI would almost agree with you on the AGI angle, but my beliefs are still recovering from the shock of how we kind of randomly stumbled into solving multiple classes of seemingly intractable cognitive problems, just by feeding a program half the Internet and letting it position pieces of words in a hundred thousand dimensional vector space. Human brain has to do a lot of work processing and integrating all the senses we use, which could cover at least for part of the neural capacity you mention, but I feel that where thinking is concerned, we may have accidentally solved the core problems. I somewhat expect that we'll look back and see that current LLMs were to human minds what the ol' Model T is to Tesla Model S - vastly different in levels of advancement, but also fundamentally the same thing.
- daveguy 4y ago> the shock of how we kind of randomly stumbled into solving multiple classes of seemingly intractable cognitive problems... Another excellent point. The idea that we may stumble onto massive computational leaps based on imprecise data processing requirements is terrifying. I do think we are nowhere near where we will need to be with respect to computation and, more important, information throughput. But the fact that we "oops"-ed into such a strong statistical model of conversation is deeply unsettling. I do believe GPT based language models will be part of an AGI.
- TeMPOraL 4y agoPart of the reason I think LLMs are a step in the right direction is that it's too big of a coincidence. We don't know how our own brains work, but we have good reasons to believe that, whatever the "magic" trick is, it can't be too complex, because evolution had to "oops" into it too. Now, it would be one hell of a coincidence if the model we "oops-ed" into with LLMs, and the model evolution "oops-ed" into, were completely unrelated approaches to the same problem.
- EliRivers 4y agoIf there are a billion different ways to solve this problem, it would seem much more of a coincidence if with LLMs we've "oops-ed" into an approach just like the one inside our heads. It would seem more likely that they're unrelated approaches.