3 ms·
I studied neuroscience and I agree, this is unlikely to have anything to do with anything. In my view it's much more likely that the speed of processing is due
by frereubu 2y ago
I studied neuroscience and I agree, this is unlikely to have anything to do with anything. In my view it's much more likely that the speed of processing is due to the brain constantly making predictions - it's an interesting thought experiment to consider the brain as simply a huge prediction machine in various modalities - most of which are pretty accurate, and therefore fast, once you've been alive for a while.
Once you get unexpected outcomes then the speed of thought slows down a lot. I was studying the neurobiology of reading and a good example is the difference in brain activity caused by the two sentences "The jam on the motorway was very slow" and "The jam on the motorway was very sticky" - the first elicits almost no activity, but the second causes a lot more activity because of the unexpected word "sticky" needs to be parsed consciously.
I think it's an issue of scale in some ways. The brain is so complex it's almost impossible to think about it clearly except in very focused studies. There's a lot of deeply methodologically flawed research out there making overly broad claims.
- TaylorAlexander 2y agoThis is a great example of issues with LLM design, which is that they apply a constant amount of compute to both sentences (modulo a token or two).
- croemer 2y agoDon't LLMs do some degree of search and backtracking if they end up in a dead end (with limits, a couple of tokens at least). Sometimes you'd see ChatGPT hanger words it had already output.
- TaylorAlexander 2y agoActually I am not very familiar with the internals, I am mostly repeating what Yann Lecun said in an interview a few months ago about autoregressive models. https://www.youtube.com/watch?v=1lHFUR-yD6I https://www.youtube.com/watch?v=1lHFUR-yD6I
- croemer 2y agoSo beam search _does_ do some sort of search: https://towardsdatascience.com/temperature-scaling-and-beam-search-text-generation-in-llms-for-the-ml-adjacent-21212cc5dddb https://towardsdatascience.com/temperature-scaling-and-beam-...