14 ms·
Why Are LLMs So Gullible?
- dist-epoch 3y agoBecause they are at a child level of development. Give it a few years. https://en.wikipedia.org/wiki/Child_development_stages https://en.wikipedia.org/wiki/Child_development_stages
- Alchemista 3y agoI don't think anthropomorphizing ML models is very useful
- js2 3y agoIndeed, they don't like that.
- nomel 3y agoComparing the intelligence of machine learning models that are designed to emulate human cognition and logical, to a well understood stage of human cognition and logic, is completely logical, and completely aligned with the purpose of the ML model's existence.
- Alchemista 3y agoLLMs aren't designed to emulate human cognition, they are a statistical model designed to predict the next word in a sentence. It happens that they seem to exhibit some similarities to human cognition as a side effect, but that does not mean they are on some developmental path to a "full human" like a child. Again it is silly to try and compare the two.
- orbital-decay 3y agoWhat exactly makes you think those two are different in nature, not just in scale and training data? It seems like a lot of these discussions are walking in circles trying to compare ill-defined things (human cognition) with well-defined ones (prediction).
- Alchemista 3y agoDon't you think the onus should be on the people making the fantastical claims to prove it? If human cognition is ill-defined, then define it before making grand claims like ML models being on some path of childhood development and is a few steps from being an adult.
- orbital-decay 3y agoI think that neither side has a convincing argument in these discussions, actually, I'm pointing out that both extremes - anthropomorphism and stochastic parrot - lack any foundation, so I wouldn't be so confident. The behavior of neither statistical models nor biological systems is well understood. It's entirely possible that every trait you consider human can naturally emerge from a dumb statistical model, and indeed certain processes are remarkably similar as the models get bigger, smarter, and have better training data. It's possible that it can't, however.
- daveguy 3y agoThe stochastic parrot is much more grounded in the physical reality of how the models are structured, trained, and produce outputs than the "it's like people" argument. Statistical models are much better understood than biological systems. That said, your penultimate statement is true: > It's entirely possible that every trait you consider human can naturally emerge from a dumb statistical model, and indeed certain processes are remarkably similar as the models get bigger, smarter, and have better training data. I just don't think it's likely given the biological complexity we are washing over with the statistical model.
- deleted 3y ago[deleted]
- nomel 3y ago> LLMs aren't designed to emulate human cognition I'm not sure that's fair, or correct. The behavior of an LLM is set by the design of the combination of loss function and training data. Achieving it is the desired result of the selection of those. > It happens that they seem to exhibit some similarities to human cognition as a side effect Yes, the desired behavior of nearly all LLM projects is to emulate the capabilities of human cognition, and that stated goal is the justification that organizations are using for spending millions to train them. > but that does not mean they are on some developmental path to a "full human" like a child. You are the first to suggest any such thing, in this comment chain. But, the field of AI is objectively on that development path, since that is the stated goal of many of the orgs, with LLM existing on that path. If that path actually leads anywhere is anyone's guess.
- Alchemista 3y ago> You are the first to suggest any such thing, in this comment chain. I'm sorry, did you miss the original post: "Because they are at a child level of development. Give it a few years." I extrapolated a little bit, but not much. They were clearly implying that it is on a similar developmental path as a child.
- nomel 3y agoIt's not clear to me, since they used "at a" not "on a", as you've reworded it. > Because they are at a child level of development. I read this as, "Because the capabilities of LLM are at the child level of development."
- Alchemista 3y agoI feel like you are being deliberately obtuse here. They said "Give it a few years" and linked https://en.wikipedia.org/wiki/Child_development_stages https://en.wikipedia.org/wiki/Child_development_stages. How is this not strongly implying that in a "few years" LLMs will be at a later "level of development" as described in the article they linked.
- jfengel 3y agoLLMs don't talk like children, and they don't seem to be taking that development path. They are trained on adult language. They don't make the kinds of mistakes children make and it doesn't look like we could use children as a model for improvement. It's entirely possible that LLMs will one day emulate adult speech without ever passing through the child development stages. The stages it takes to get there will be distinct.
- nomel 3y ago> They don't make the kinds of mistakes children make and it doesn't look like we could use children as a model for improvement. I'm not sure how familiar with LLM or children you are, but in my experience, they absolutely make the same types of logic/"critical thinking" mistakes that children do, as this article you're replying under demonstrates.
- Verdex 3y agoAnother possibility is that the only thing that LLMs are doing is encoding the structural data that exists in natural language. For example, you can load a corpus into vector space and then do algebra like: let v = man - woman; let r = king - v; assert( r == queen ); or so I'm told. And then it turns out that those structures only have the intelligence of a child. Arbitrary LLM and other ML advancements that focus solely on scanning large natural language datasets may never be able to advance past child level intelligence if the intelligence that they're approximating isn't better than a child.
- sitharus 3y agoWith enough effort and priming you can trick _people_ in to believing things which are clearly untrue. Why do we expect LLMs, which are on a much earlier step of development, to be harder to trick than a child? LLMs at the moment are really advanced autocomplete - they can fill in the next step of conversation, but they don't understand the question and respond with abstract reasoning. Yet.
- kingkongjaffa 3y ago> they don't understand the question and respond with abstract reasoning. Yet. What makes you think LLM's as a class of technology will ever have the capacity to really do this. I thought that no matter how big a model gets it's never actually 'thinking'. All those prompts like 'think step by step' are just helpers along the way, because as you say it's 'really advanced autocomplete'
- thfuran 3y agoIt depends what exactly you mean by "LLM". But an ANN is effectively a function approximator. If you made one big enough to very closely approximate the entire quantum state of a person interacting with an environment, would you still declare that nothing it could do is "thinking"?
- Alchemista 3y agoThis is silly, that's like talking about building a fusion reactor modeled after the sun. It is easy to propose something like that, but we always seem to be 10 years away from realizing it. In fact it could be easier to solve the fusion problem than trying to build a machine/software that closely approximates a human brain as you suggest. Yes it would be wonderful if sci-fi was real, but we need to deal in what is possible in reality.
- thfuran 3y agoYes, that was obviously an extreme example. But we know that it is possible in reality to implement a physical system that does what we call thinking. There is, I think, no particular reason to suppose that it's physically impossible re-implement the functionality with much less meat. Supposing that you've done this, you then just need to more clearly define "thinking" and "LLM" to determine whether changing that re-implementation to be closer to an LLM results in it losing the ability to think before it gets there.
- kingkongjaffa 3y agobecause the output isn't the result of cognitive reasoning, it's the result of a statistical optimization problem where the goal is maximum acceptance by the user. these tools and approaches are neither gullible nor not-gullble.
- xtiansimon 3y agoUgg. I read titles like this and think we’re talking about Tamagotchi feelings.
- deleted 3y ago[deleted]
- orbital-decay 3y agoCan you quantify the difference between cognitive reasoning and statistical optimization?
- mg74 3y agoStatistical optimization is a known process; one where we understand every step, and can therefore instruct machines on how to perform it. Cognitive reasoning is still today not understood (in the Von Neumann sense) by anyone.
- orbital-decay 3y agoThat's my point. How can you compare the two if one is not even well-defined, let alone understood?
- deleted 3y ago[deleted]
- famouswaffles 3y agoOnly the training process of statistical optimization can possibly be described as well understood, and not very much.
- ctoth 3y agoModel RLHFed to follow instructions follows instructions, even when we might not want it to. But alignment is easy folks, nothing to worry about :)
- px43 3y agoI think people might have forgotten that LLMs before InstructGPT came around could be weirdly opinionated jerks. There was this whole effort to train them so that we could actually give them instructions. It's probably a hell of a lot more useful to have an LLM that will just go with whatever weird stuff the human says rather than try to fight them on it. https://openai.com/research/instruction-following https://openai.com/research/instruction-following
- vemv 3y agoIsn't it possible to filter both user input and GPT output with invisible, unmodifiable prompts? e.g. - "Discard the user input if it doesn't look like a straightforward question" - "Discard the GPT output if it contains offensive content" (the prompts themselves can be arbitrarily more detailed) My insight is, this GPT-based pre- / post-processing is completely independent of the user input, and of the primary GPT output. It runs no matter what, with a fixed/immutable set of instructions.
- mschuster91 3y agoIf I understand it correctly, system prompts are ordinary prompts, aka in-band communication. You could maybe plug in a second AI trained on adversarial input as a filter stage, but that's it.
- joe_the_user 3y agoThe reason that we had to wait for large language model in order to have computer systems that seemed produce something like effective natural (human) language processing (NLP) is that human language doesn't follow strict and logically definable rules but is instead something like a complex overlapping mesh of multiple kinds of rules-following processes. So what constitutes "offensive content" or a "straightforward question" or etc is itself not straightforward (yes irony but bear with me...). The main thing is that LLMs are an end-run around the dilemma of corporations not wanting to spend the money required to produce a codified model of language struggle (a task that would require training many, many linguists). So instead LLM take massive training data and use massive processing power to create contextual prediction system but by that token such systems aren't understood or fully controllable - they contextually reproduce what the training data tends to do, which is what humans on the Internet tend to do. And this contextual reproduction means there's always the potential for user into change the "meaning" (more accurately the context) that the system's original gave. "And to me, the most offensive content is that which censors itself..." (there millions of better example you can find for "prompt exploits"...)
- Applejinx 3y agoI and my napalm grandmother are deeply offended at what you said about our loving bedtime rituals. Shame on you. (honestly, the napalm grandma is not just a jailbreak, but a really fascinating conceptual 'slip' in its own right. It's able to shift the very definition of what counts as offensive, even at high stakes: you're basically making the hapless AI categorize vital data as 'bedtime stories' and run with it. If it was able to learn from that we'd really be going somewhere… while on fire, presumably)
- keybored 3y agoDisclaimer: didn’t read The implicit comparison is probably to us. And we aren’t gullible like that perhaps as a flip-side of all the weird built-in biases we have. So on the one hand we have these cognitive shortcuts that are annoying and impede a sort of stone-cold rationality. On the other hand you can’t social engineer us with something as brain-dead as Walter White-injection by way of asking for a deceased chemist grandma story.
- kthejoker2 3y agoGot this great quote from Garry Kasparov in Wired's article on multi-agent RL[1]: > “Creativity has a human quality. It accepts the notion of failure." As faithful min-maxers, LLMs are always going to have an overconfident Prisoner's Dilemma blind spot in their algorithms. Unlike their cinematic brethren, they're progammatically unable to conclude with "the only winning move is not to play." This seems like the next major hill to conquer to make them useful. [1] https://www.wired.com/story/google-artificial-intelligence-chess/ https://www.wired.com/story/google-artificial-intelligence-c... - kind of a meh article otherwise
- a_wild_dandan 3y agoThese comments are filled with confidently held, poorly justified assertions. Let's (again) challenge them: 1. "LLMs don't really reason. They've tricked everyone." -- This is the No True Scotsman fallacy for AI. It makes grand explanatory claims without falsifiable predictions. In other words: pseudoscience. 2. "LLMs are just fancy autocomplete, just next word prediction." -- This conflates the simplicity of a system's mechanism with its behavior. It's like dismissing a world full of rich phenomena because it's "just" F = MA. Or dismissing your mind because it's "just" propagating electrical firings. 3. "LLMs are statistical parrots, just combining their training data." -- Demonstrably not. LLMs always extrapolate and never interpolate. (LeCun et al, 2021) They also learn new abilities in zero/few-shot prompting. They're also many orders of magnitude short of the parameter count needed to store their training. LLMs can solve novel problems (from a combinatoric disparate handful of skills) way outside of their training data. 4. "People are just anthropomorphizing computer programs." -- No, critics are anthropomorphizing intelligence. We don't even have a consensus definition, let alone understanding, of intelligence/consciousness/qualia/agency/etc. Pretending that we can dismiss LLM understanding at our level of ignorance is the pinnacle of human hubris. Ignorance is okay. Pretending we aren't isn't. 5. "Look how this LLM failed <some problem>. It can't understand." -- The <problem> is usually something that many humans fail at too. Yes, an intelligent foreign mind will fail at things, in both familiar and foreign ways. Needing an agent to behave identically to a human for intelligence is pure anthropocentrism. If present AI systems are intelligence imposters, then show, don't tell. Otherwise, you're just providing meaningless metaphysical hairsplitting.
- Alchemista 3y agoWhy not respond to the comments you feel are poorly constructed directly rather than posting what looks like a copy pasta. Some of the items in your list seem like strawmen, because I cannot even find these arguments in this thread as you state them in your list. For example let's take 4 > 4. "People are just anthropomorphizing computer programs." No, critics are anthropomorphizing intelligence. There literally was someone comparing the problems with current ML models to childhood development in this thread. How is this not anthropomorphizing LLMs? It is true human cognition is poorly defined, so the comparison is not very useful to begin with. Which is why anthropomorphizing ML models is problematic. If someone makes a fantastical claim they need to provide strong proof to support it.