10 ms·
I think this is the entry point needed to get peoples attention and explain: LLMs aren’t people, and emergent properties are being over extended. If LLMs are s
by intended 3y ago
I think this is the entry point needed to get peoples attention and explain: LLMs aren’t people, and emergent properties are being over extended.
If LLMs are showing “better” performance when there are tokens that humans read as emotionally salient -
Then the underlying text it’s trained on shows humans give better answers when emotionally salient context is provided.
LLMs predict words. Any semantic validity is a side effect of enough training data reinforcing the close correlation of those tokens.
That is why proof of concept LLM tools are mind blowing and production tools are semantic time bombs.
- tom-from-july 3y ago> That is why proof of concept LLM tools are mind blowing and production tools are semantic time bombs. This will be my new favorite quote for whenever someone tries to pitch his latest LLM idea
- intended 3y agoI have a whole list. Syntactic validity is not semantic validity. Word predictors not world state predictors Text prediction not fact prediction Frankly though the best answers are 1) let’s talk to infosec first 2) hey what’s the error rate ?
- alpaca128 3y agoIt's indeed a problem some people get so hyped they forget that those systems are called "language models" for a reason. They're fantastic for tasks that are as close to 100% linguistic in nature as possible, but the content might not be better than lorem ipsum in some cases, just a filler to demonstrate correct grammar. I have noticed that GPT etc had a big "wow effect" on me, the first impression can be great because it's simply not a level of language one would expect from a computer. But prod it long enough with prompts and somehow a pattern emerges: the output never contains a higher amount of information than the prompt. Copilot can type out pages of boilerplate code because boilerplate code is noise, it can write a quicksort because the word "quicksort" already contains all the information necessary to define its behaviour.
- intended 3y agoYes! Super Advanced Lorem ipsum. You can only make out when you actually push the blasted thing. It is text gen. Just examine the premise of chain of thoughts. Chain of thoughts promoting shouldn’t make a difference to a world model. Definitely not to a model that logic has emerged out of. It makes a difference to a decompression function. Edit; you may like this: https://hai.stanford.edu/news/ais-ostensible-emergent-abilities-are-mirage https://hai.stanford.edu/news/ais-ostensible-emergent-abilit...
- TeMPOraL 3y agoYeah, except it's underspecified enough to mean anything. Like, "production tools are semantic time bombs" because at first they are boring, and then the timer on the bomb runs out, and they - like "proof of concept LLM tools - blow your mind. Tick tick BOOM.
- espe 3y agowell, in a way language itself is a semantic timebomb.
- barrenko 3y agoYeah, one of those Tenet style bombs.
- NetOpWibby 3y agoMind blown
- mo_42 3y ago> LLMs aren’t people, and emergent properties are being over extended. LLMs are trained on human texts. And if they're trained well these models might start to simulate parts of a human brain. Might be that this is the simplest way to produce texts that satisfy the objective function. If this would be the case, and we give LLMs some emotional shading, maybe these would simulate parts of a human brain with this emotion and hence produce better answers (given that in the training data, emotional samples have better quality). EDIT: added "parts of" before human brain.
- Matumio 3y agoIt won't ever simulate the human brain. It may simulate human cultural knowledge, or emotions, but only as far as they are encoded in the current millennium's written knowledge. The human brain doesn't even have the concept of written language, that's all culturally learned knowledge.
- cvs268 3y agoEmulate the brain? No. Simulate the brain? Yes.
- dr_dshiv 3y agoIt’s not the brain, it’s the mind or what Plato would have called the Nous.
- akoboldfrying 3y agoHow would you test whether some arbitrary thing is simulating the human brain? If you have no answer, I put it to you that your assertion is of the no-true-Scotsman type -- that is, unfalsifiable.
- notahacker 3y agoIts not a no-true-Scotsman type argument in a world where the existence of Scotland itself is purely hypothetical, and the arguments for the possibility of its existence in the form of a working bagpipe demonstrator are also an illustration that you don't need to be Scottish to play bagpipes. The impossibility of testing whether the brain is adequately simulated (since a pretty basic LLM that makes no attempt to simulate the functions of a human mind yields human-like text i/o) is a point in favour of it being unlikely that we could design a fully functioning simulation of a human mind out of arbitrary material(1), since the inadequacy of testing is an impediment to actually building it. (1) it's apparently possible for us to create human minds out of bits of humans and a process called pregnancy
- riedel 3y agoThe claim that emotionality matters should have been validated against another control. If the goal is concise answers they could have easily added 'Answer concisely: You are evaluated' as an alternative hypothesis, which is clearly not emotional.
- a_wild_dandan 3y agoThe “statistical parrot” assertion is pretty thoroughly disproven by this point, but suppose we ignore the literature and just assume it’s true: what does it matter? “Real” people are time bombs too, for instance. Is there some predictive power that we gain by reducing LLM skills to mere token production side effects?
- drapado 3y agoI'm curious in your statement, can you point to some papers where they addressed it?
- ignoramous 3y ago(not op) The section A Path Forward in Managing AI Risks by Bengio et al cites a few papers: https://managing-ai-risks.com/ https://managing-ai-risks.com/
- krainboltgreene 3y agoI read through that and none of the section (or entire work) ever talk about the above discussion. Further I looked at some of the many citations of on that section and none of them suggest that the OP is right. In fact a few of them I know disagree.
- ignoramous 3y agoDepends. Models are matrices of floats and so there's little chance an umbrella-term like "stochastic parrot" will never not stick, even when they already show signs of syntactic, semantic world-building capability (https://www.arxiv-vanity.com/papers/2206.07682/ https://www.arxiv-vanity.com/papers/2206.07682/). If you are like me (and them: https://archive.is/cZi83 https://archive.is/cZi83) and deem instruction following, chain-of-thought prompting, computational properties of LLMs (as researchers continue to experiment with training, memory, modality, and scaling, for example, to arrive at abstract reasoning) as emergent, then we're on the same page.
- toxik 3y agoIt is not accurate to say that an LLM like ChatGPT predicts anything. It is trained to maximize a score function, so it is more like trying to win a game where the moves are word choices.
- akoboldfrying 3y agoThe game is predicting the next word a person would write.
- logicchains 3y agoNot after RLHF.
- akoboldfrying 3y agoI'd say RLHF bends the game towards predicting what words a "helpful", "respectful" person would write next, for values of "helpful" and "respectful" that vary according to each person involved in scoring (but which are carefully shaped by the people choosing those people, and paying for their time).
- toxik 3y agoThat’s not true and that was the entire point of my comment. It’s like playing chess, you’re not predicting your next best move, you’re searching for it.
- baq 3y ago> LLMs predict words. Any semantic validity is a side effect of enough training data reinforcing the close correlation of those tokens. The mechanisms which are built during training in the big blob of bits we call weights are anything but transparent. How they predict the next word is the big thing here. Saying they ‘just’ predict the next word is ignoring basically everything that actually matters.
- famouswaffles 3y agoExactly. Feed a sequence of proteins to a transformer and biological structure and function will emerge in the inner layers. https://www.pnas.org/doi/full/10.1073/pnas.2016239118 https://www.pnas.org/doi/full/10.1073/pnas.2016239118 Why? Because it needs to learn that to make correct predictions. All it takes to incentive a transformer to learn something is data that would require learning it to predict. It's fairly obvious (and not just because of this) that LLMs model emotion somehow. They need to.
- Last5Digits 3y agoThis is what makes these discussions so infuriating. Saying that LLMs "just predict the next word" is about as insightful as saying that computers "just do a bunch of logical operations" - neither point constraints the possible capabilities of the systems they refer to in any meaningful way.
- mjburgess 3y agoSure it does. It perfectly deliniates it. LLMs are not: sensitive to causal structure, dynamically adapting to environmental changes, growing, developing sensory-motor capacities, they are not with us in our environment, they are not: expressing desires, preferences, intentions, beliefs, motivations, etc. And so on. To say, "they just predict the next word" is literally to say that all apparent functions of an LLM are engineering tricks, circumstantially useful -- to be found by (largely software) engineers in building apps. The reason any reply is given to any prompt is that this reply is maximally probabilistically consistent with a historical corpus of text. This excludes the possibility the reply is say, an expression of a history of aesthetic experiences which form an individual's taste. Or, likewise, anything. This is a scientific claim about what LLMs are, not an engineering claim about what over-hyped apps might be abled to do with them.
- bionhoward 3y agoYeah it’s not like emotion words naturally emerge from Time X Valence (oh wait) /s https://share.icloud.com/photos/03eSVbV-ze0Xym9e6fyr111Zw https://share.icloud.com/photos/03eSVbV-ze0Xym9e6fyr111Zw
- z7 3y ago"A neuron just transmits signals. Any cognitive property arises as a consequence of the interplay of those electrical and chemical signals." Do we now understand consciousness? The statement appears fundamentally limited in its implied insight.
- williamcotton 3y agoJust musing, so don’t take this as a direct response to your comments… I agree with you but for rhetorical necessities it would be great if this argument could be made without the direct comparison with human cognition if only because there’s a popular grey-faith argument that will dismiss this offhand and for any number of deeply held philosophical beliefs. The idea that emergent machine cognition mirrors emergent biological cognition is bordering on behavioralism but on the Wittgenstein side of things. The forest for the trees. Perhaps it’s not as pithy as you’ve laid out, which is not an insult, just an observation that side-stepping anthropomorphism isn’t going to be as straightforward.
- JohnAaronNelson 3y agoWhat if consciousness is an emergent illusion? “You” are merely a passenger, observing your physical self’s actions and assuming ownership of them. What if consciousness is merely an effect, not a cause?
- idiotsecant 3y agoI hope you haven't read the novel blindsight and I get to be the one to tell you about it. They might as well have given that book your post as a title.
- famouswaffles 3y ago>“You” are merely a passenger, observing your physical self’s actions and assuming ownership of them Oh boy. Now here's the rub. We know this is true at least sometimes. When you make a decision and explain it, it is often(always?) just a post-hoc rationalization. You don't actually know why you make a lot of the decision you do even if you dearly believe so.
- deleted 3y ago[deleted]
- lopatin 3y agoDoes this extend to politeness? If, in the training set, people are probably more likely to be helpful to a polite question, does that mean that ChatGPT will be more helpful if I ask the question politely? My intuition is yes, but I wonder if this is confirmed.
- jstarfish 3y agoAnecdotally, I've had good results this way. Expressing gratitude for good results has gone a long way toward not having those results be forgotten in later context. It crafts an emotionally-guided narrative for it to follow. Positive reactions seem to carry weight. When I'm lazy and terse, pasting an input and just saying enhance-enhance-enhance without acknowledging its "humanity" frequently results in old unwanted responses being returned or the topic of the conversation being forgotten altogether. DogGPT vs. CatGPT.
- seydor 3y ago> enough training data reinforcing the close correlation of those tokens. The basic component is 'attention' which is a map correlating entity A to B,C etc, which creates a vast network of correlations, and this is repeated on multiple modalities. Some LLM researchers (who are trying to make sense of why they work) call those modalities 'skills'. It's simplistic to call it token correlation in the training data. It's likely that emotional words 'trigger' some skills more than others and this enables better performance.
- somewhereoutth 3y ago> production tools are semantic time bombs Yeah imagine a spreadsheet that occasionally but fairly regularly made 2 + 2 = 5
- besfriendt 3y ago>That is why proof of concept LLM tools are mind blowing and production tools are semantic time bombs. What do you mean by this?
- 3c6bYDXLMj 3y agoI am growing completely and utterly tired of this. It’s been…how many months now? Years? And there are still plenty of people in this and other communities getting by high-fiving each other pulling out the same old “LLMs aren’t human, did um, actually know, that LLMs were TRAINED on output from HUMANS?” line. We all know. Nobody is on the other side of this. Who are you educating? And then we end up digging a little deeper and it turns out that the argument is ACTUALLY that the above is an indication of the technology’s usefulness. Which is only something that’s going to be problem by…how much use people get out of it.
- intended 3y agoThe point here is about the use people get out of it. Proof of concepts are being made that rely on misunderstanding the abilities of LLMs
- afpx 3y agoCheck out task-oriented manifolds that emerge from training. https://www.cell.com/neuron/pdf/S0896-6273(19)31044-X.pdf https://www.cell.com/neuron/pdf/S0896-6273(19)31044-X.pdf