13 ms·
> like we all do Do we though? Sure, we communicate sequentially, but that doesn't mean that our internal effort is piecewise and linear. A modern transformer
by Bjartr 1y ago
> like we all do
Do we though? Sure, we communicate sequentially, but that doesn't mean that our internal effort is piecewise and linear. A modern transformer LLM however is. Each token is sampled from a population exclusively dependent on the tokens that came before it.
Mechanistically speaking, it works similarly to autocomplete, but at a very different scale.
Now how much of an unavoidable handicap this incurs, if any, is absolutely up for debate.
But yes, taking this mechanistic truth and only considering it in a shallow manner underestimates the capability of LLMs by a large degree.
- kenjackson 1y agoOur thinking is also based only on events that occurred previously in time. We don’t use events in the future.
- ElevenLathe 1y agoIs this a certainty? I thought it was an open question whether quantum effects are at play in the brain, and those have a counterintuitive relationship with time (to vastly dumb things down in a way my grug mind can comprehend).
- kenjackson 1y agoWell there’s no evidence of this that I’ve seen. If so, then maybe that is what is the blocker for AGI.
- ElevenLathe 1y agoI think it's more that there isn't yet evidence against it. In other words, we're not sure or not if the brain has some kind of special sauce that doesn't just reduce to linear algebra.
- wasabi991011 1y agoQuantum effects definitely reduce to linear algebra however.
- kenjackson 1y ago"I think it's more that there isn't yet evidence against it." We don't? AFAIK we have no proof of anyone being able to see into the future. Now maybe there are other manifestations of this, but I know of no test today that even hints at it.
- wasabi991011 1y agoI'm aware of a counterintuitive relationship with space, but what's the one with time?
- freejazz 1y agoThis is unhelpfully obtuse
- kenjackson 1y agoWhat's obtuse about it? It's honestly a very straightforward statement. Every thing we think or say is a function of past events. We don't incorporate future events into what we think or say. Even speculation or imagination of future events occurred in the past (that is the act of imagining it occurred in the past). It's really a super simple concept -- maybe it's so simple that it seems obtuse.
- freejazz 1y agoBecause the other poster's point wasn't that it was a 'past event.' The point was that it's just predicting based upon the previous token. It's disingenuous to mix the two concepts up.
- kenjackson 1y ago> The point was that it's just predicting based upon the previous token. Well that's just wrong. None of the LLMs of interest predict based upon the previous token.