5 ms·
What makes you think that? What we can LLMs are a specific architecture that literally does predict the next token, one at a time. This isn’t the only way to ge
by 4death4 3y ago
What makes you think that? What we can LLMs are a specific architecture that literally does predict the next token, one at a time. This isn’t the only way to generate text. Perhaps humans use a method that synthesizes all the text at once. Or some other, unimaginable way.
- FrustratedMonky 3y agoNot sure about everyone else, but I write one word at a time.
- JKCalhoun 3y agoAnd often unsure about where I'm going until I get there, ha ha.
- simondotau 3y agoTo be fair, I rarely write more than a sentence or two in serial form, and I have often determined the point of a sentence before I write the first few words. Occasionally the act of writing out an idea in words clarifies or changes my mind about that idea, causing me to edit or rewrite what I’ve already written.
- the_other 3y agoYou may write one word at a time but the grammar of most languages forces their users to know what they’re going to write several words ahead of the current word.
- FrustratedMonky 3y agoOk. Why do you think GPT isn't doing that? or is different? It might be calculating the next word, because you can only write one word at a time, but you can't say that the current next word isn't influenced by a few words ahead. Don't think the current understanding of what LLM's do internally can rule that out.
- deleted 3y ago[deleted]
- ofjcihen 3y agoBut are you writing to write or are you writing to convey a thought?
- dartos 3y agoYeah but do you write that one word based on the past 1000 or so words that come before it and how each of those words relate to each other with a perfect recollection of each word at the same time? And if you believe you do that subconsciously, then how can you be sure you don’t subconsciously plan a few words ahead?
- FrustratedMonky 3y agoI don't think I do. To a large degree, GPT does write better.
- dartos 3y agoWrite better as is has more correct and clear syntax, sure. But it doesn’t write better as in has new, challenging ideas. Or ways to move/relate to humans on a personal level. GPT writes clear, concise, authoritative, boring, and generic text.
- deleted 3y ago[deleted]
- Qwertious 3y agoWe ~~all~~ mostly do that partially, but some of us plan our comments ahead and go back and edit previous parts of it after review.
- taeric 3y agoAlmost certainly not true. Consider pronouns, as they reference other words, therefore cannot be written in isolation. Or simple rules about when to use "an" or "a". Shaping your words to sound correct is very common. Both in speech and in writing. Sometimes it is finding how to fit a word you want to use into a sentence. Sometimes it is building a rhyme. You may feel that you go a word at a time, but that really shows how embedded language is.
- FrustratedMonky 3y agoRight now we don't know if LLM's are also doing this, or not. In their calculating the 'next' word, as part of that 'weighting', are the 'simple rules' for future words, that you are saying humans do but LLM's can't.
- taeric 3y agoAh, fair. I was only commenting on the idea that you write one word at a time. Seems fairly unlikely to me. You expand out ideas.
- FrustratedMonky 3y agoLOL. Yes, started as a joke.
- 4death4 3y agoIt doesn't seem LLMs are capable of "understanding" that they don't "know" something, so there are certainly some observable differences. But the intent of my original comment was precisely to highlight that we don't know. It could be that next token predication is somehow mathematically equivalent to what we call consciousness. That would certainly be a revelation.
- ben_w 3y agoI seem to have two parts of my inner monologue, one comes up with complete concepts, the other puts them into words. When I started to notice this, I tried skipping the "make words" part to save time, as clearly I had not only had the thought but also was aware that I had already had the thought. This felt wrong in a way I have no words to describe as there's nothing else like that particular feeling of wrongness, though I can analogise it as being almost but not quite entirely unlike annoyance. Anyway, point is 80% of this comment was already in my head before I started typing; the other 20% was light editing and an H2G2 reference.
- FrustratedMonky 3y agoSounds kind of like some Zen concepts about moving beyond language. And that language is a faulty method for communicating.