3 ms·
The models literally do just repeatedly calculate the most (or randomly not quite the most) often used next word. So those problems can in fact be solved by doi
by wrs 2y ago
The models literally do just repeatedly calculate the most (or randomly not quite the most) often used next word. So those problems can in fact be solved by doing that, because that's how they're being solved.
- Spivak 2y agoIt's really not though, you're confusing one teeny tiny part of the decoder with the entire decoder. Yes you sample from output probabilities but that's literally the least interesting bit. How you generate those probabilities is the hard part. You do this! When you're speaking or writing you're going one word at a time just like the model, and just like the model your attention is moving between different things to form the thoughts. When you're writing you need at least a semi-complete thought in order for you figure out what word you should write next. The fact that it generates one word at a time is a red herring as to what's really going on.
- Brian_K_White 2y agoIf neither an AI nor a person does anything but fit the next word by matching with other prior texts, then where did those other texts come from? A language is a vocabulary and a set of rules which can be described and codified, and is generally learned from others and other prior examples. But all that describes is a tool called a language. The tool itself is mechanistic. The wielder may or may not be. The tool and the wielder are two different things. A piece of paper with text on it is not a writer. A machine that writes text onto paper is still not a writer even though a human writer performs the same physical part of the process as the machine. Similarly, just like it's possible for an mp3 player to make the same sounds we use to express concepts and convey understanding, without itself having any concepts or understanding, it is also possible for an intellect to perform mechanistic acts that don't require an intellect. Sure, actually quite a lot of human activity can be described by simple rules that a machine can replicate. So what? You can move a brick from one place to another. And so can a cart. And so can gravity. Therefor you are no different from a field? It's utterly silly to be mystified by this. It's challenging the other ape in mirror for looking at you. https://www.youtube.com/watch?v=GaMylwohL14&t=23s https://www.youtube.com/watch?v=GaMylwohL14&t=23s
- dmvdoug 2y agoNow you’re being remarkably reductive—-about the human language facility. We don’t ”go one word at a time.” (Simple example: “hey, look at that tig boad! Uh… ha! Tig boad, I meant big toad.” Stupid example but the point is that the cognitive psychology of language is much more complex that you’re making out.) Also, what are these “thoughts” you have when writing and what’s a “complete” vs. “semi-complete” one? Stupid question, yes, but you again vastly over-trivialize the real dynamic interplay between figuring out what word(s) to write and figuring out what precisely you’re actually trying to say at all. Writing really is another form of thinking for us.
- Spivak 2y agoYou went the opposite direction than what I was trying to say. I'm saying that the cool part of these models is that they have this dynamic interplay because of their self-attention mechanism and aren't really going one word at a time in the same way that humans aren't despite the appearance of typing or saying one word at a time.
- wrs 2y agoI don't know about you, but I seem to solve a lot of problems without generating any words at all.