5 ms·
because the output isn't the result of cognitive reasoning, it's the result of a statistical optimization problem where the goal is maximum acceptance by the us
by kingkongjaffa 3y ago
because the output isn't the result of cognitive reasoning, it's the result of a statistical optimization problem where the goal is maximum acceptance by the user.
these tools and approaches are neither gullible nor not-gullble.
- xtiansimon 3y agoUgg. I read titles like this and think we’re talking about Tamagotchi feelings.
- deleted 3y ago[deleted]
- orbital-decay 3y agoCan you quantify the difference between cognitive reasoning and statistical optimization?
- mg74 3y agoStatistical optimization is a known process; one where we understand every step, and can therefore instruct machines on how to perform it. Cognitive reasoning is still today not understood (in the Von Neumann sense) by anyone.
- orbital-decay 3y agoThat's my point. How can you compare the two if one is not even well-defined, let alone understood?
- deleted 3y ago[deleted]
- famouswaffles 3y agoOnly the training process of statistical optimization can possibly be described as well understood, and not very much.
- mg74 3y agoThe steps are understood well enough to instruct machines to do it, even if the output is to complex for us to comprehend it completely. Not so with thought, intelligence, consciousness, etc.
- famouswaffles 3y agoagain, no they aren't. In ML, nobody instructs machines to do anything other than train and find those steps themselves. We have no clue what steps GPT takes when it solves problems. For a straightforward example, when GPT-4 adds, what algorithm does it perform ? I have no clue, you have no clue and neither does anyone at open ai. What we do is instruct machines to train. What they get out of training, we have very little idea.
- chx 3y agoSure thing. People really don't learn the history of AI any more apparently or this question wouldn't come up all the time. There is basically any number of questions you can ask a two year old human who have never encountered that question nor anything even remotely similar to it and yet they can answer without fail. Meanwhile absolutely no AI can answer these unless the specific question / the rules underlying the questions were previously fed into it. The textbook example is "If Susan goes shopping will her head go with her?" Of course, since this specific question is literally a textbook one, you can't fool an LLM with it but it's easy to come up with brand new ones. In the early 1980s this stopped Douglas Lenat who has worked very successfully on discovery systems and made him turn to assembling these facts and rules into CyC.
- famouswaffles 3y agoYou've not quantified anything, nor have you even provided any valid example of these so abundant mystery questions. If it's so easy, come up with one and show us.
- chx 3y agoIf I do then it'll get gobbled up by fake AI and I won't make up one every time ignorant people who can't open an AI textbook demand one from me. Apropos: https://existentialcomics.com/comic/289 https://existentialcomics.com/comic/289 You could use that comics to generate a few of these questions.
- famouswaffles 3y agoWhat a ridiculous excuse. Leaving aside the fact the chances of your reply being in the training of the next GPT near 0, it certainly won't happen anywhere near fast enough to disprove your point to anyone reading this today. So you have nothing. Not that i'm surprised.
- chx 3y agoNo, the automated plagiarism machines have nothing and I am not obligated to cure your ignorance. Let's continue the conversation after you read an AI textbook, shall we?
- CharlesW 3y agoThank you for this. I think it's important that technical folks in particular not anthropomorphize LLMs, and help less technical people understand how they work and that they lack consciousness, emotions, and understanding.
- Affric 3y agoYou mean these highly anthropomorphised programs? It’s important that technical people don’t anthropomorphise them? I agree but the creators of all the main LLMs have already crossed the line by a long way. E.g. It’s deeply troubling that it’s acceptable that LLMs deliver inline apologies.
- genman 3y agoThese "apologies" are something that deeply irritates me.
- deleted 3y ago[deleted]
- cyanydeez 3y agoit's kinda amusing because the LLMs evolved via selection of the best anthropic traits because the basic digested mountains of undiscerning information. so the gullibility is merely one selected for trait convolution with state.
- daveguy 3y agoThere is no evolution or trait selection involved in the training of LLMs. LLMs are statistical estimators of most likely next series of tokens. Those next series of tokens seem anthropic because most of the input was human generated text.
- yongjik 3y agoConsciousness and emotions, sure. Understanding, I'm not so sure. Saying that LLMs cannot understand concepts because "it's just statistical pattern matching" is not going to help laypeople understand what LLMs are. After all, what is a human brain if not a biological pattern matching machine?
- the_gipsy 3y ago> statistical optimization problem where the goal is maximum acceptance by the user A brain could also be described like this, if you focus only on the text output.
- api 3y agoIn a brain the user to some extent is itself. LLMs do not have anything like this. They're once-through, static, and are not in any way embodied or self-referential (beyond context or what you feed back into them).
- the_gipsy 3y agoyes - for now.
- deleted 3y ago[deleted]
- swatcoder 3y agoA brain that only performed text output might be modeled like that, which is exactly what these represent. But no brain we've ever seen was subject to that constraint, so it's a sort of fantastical target for modeling and doesn't say reveal anything about brains themselves or the various entities that seem to bear them. Modeling one feature of a grossly simplified and incomplete brain is a creative approach to computational research and proved very fruitful, but there's no reason to leap from that success to the idea that real brains do work that way or even that the imagined only-textual-parts must.
- throwaway240219 3y agoExactly correct. Article title would be better as, "Why are users of LLMs so gullible?" Because people implicitly treat AI as if it were conscious, and we keep forgetting that.
- iforgotpassword 3y agoBecause it still makes sense to reason about them that way, as weird as it may be. It's like the Chinese room problem, even though we know nobody is inside. It's like when people say "our brain thinks that ..." when they talk about something we do subconsciously, or some illusion we fall for. What does that even mean? Is our brain suddenly a detached entity thinking on its own? Then what am I using to think? Yet, everybody understands what is meant by that. So I don't think people keep forgetting that llms aren't conscious, at least as long as we talk about the target audience of articles like this, eg hn folks.
- joquarky 3y ago> What does that even mean? Is our brain suddenly a detached entity thinking on its own? Then what am I using to think? IFS recognizes subpersonalities¹ which is a model which can adequately explain this [1] https://en.m.wikipedia.org/wiki/Subpersonality https://en.m.wikipedia.org/wiki/Subpersonality
- mrtksn 3y agoI used to think like that but I'm not so sure anymore. The statistical optimisation thing is an analytical approach to Neural Networks but its similar to saying that love is just hormones.
- Jensson 3y agoIsn't love just hormones? It isn't rational reasoning at least.
- BazookaMusic 3y agoA simple proof that love is not just hormones is that love can last for some time. If it was just a chemical phenomenon why does it happen repeatedly. Why can someone feel love for someone just by bringing to mind the symbol which represents that person? Why can someone feel love by seeing an illustration of someone they love?
- famouswaffles 3y agoHow is that a proof of anything? Do you imagine all chemical reactions to be instantaneous with effects that last only a short while ? >Why can someone feel love for someone just by bringing to mind the symbol which represents that person? >Why can someone feel love by seeing an illustration of someone they love? Why would other things triggering hormones and reactions mean it's not just hormones and reactions?
- BazookaMusic 3y agoThe point is that it is a phenomenon connected with memory and thinking processes. The reaction and release of hormones is part of the phenomenon, but not the entire phenomenon. Take this in contrast to an allergy for example. Can you trigger an allergy by remembering the food? If you see an illustration of a food you are allergic to, do you get an attack? A more similar phenomenon instead is a phobia. It is also a release of hormones based on some internal or external stimulus (you can apply all my examples about love). However in a phobia it's even more clear that the phenomenon is based on thinking patterns. Reducing the complexity to saying these are just "hormones and reactions" is the same as saying a computer is "instructions and interrupts". That is to say that a computer has these elements but what makes it work is that there are a bunch of other systems including the humans writing the software that organize these into a functional system.
- kypro 3y agoThis depends on perspective. I could argue the issue isn't that it's gullible but misaligned. In the case of the napalm Grandma it seems odd to me that you're suggesting the LLM is stupid because it's answering in a way that makes sense given its prompt. The issue doesn't necessarily suggest a lack of reasoning, but that the LLM is trusting the human. For the record, I agree with you – I would have thought that an AI that can reason well would probably know when not to trust humans, but I suppose that assumes it values preventing humans creating napalm over being correct and helpful. Maybe it just doesn't share our values and prioritises being honest and helpful. From this perspective the issue then wouldn't be that LLM is stupid, but that they are too trusting and too honest, and that we must find a way to build an LLM that is more distrusting and deceptive if we wish to align it with our values and our nature.
- mattlutze 3y ago> For the record, I agree with you – I would have thought that an AI that can reason well would probably know when not to trust humans, but I suppose that assumes it values preventing humans creating napalm over being correct and helpful. Do we want LLMs, and later other multi-modal / servo systems, that are deciding they can't trust a human prompter and taking actions based on that? >... and that we must find a way to build an LLM that is more distrusting and deceptive if we wish to align it with our values and our nature. Tongue in cheek or actual argument here?
- kypro 3y ago> Tongue in cheek or actual argument here? I think it's interesting that there is no clear answer – do we want AIs to trust us all the time, or is an aligned AI counterintuitively one that often distrusts us and perhaps sometimes even lies to us? I thought it was interesting that the parent commenter suggested that the reason LLMs are so trusting is because they can't reason anyway. It would implying that in the future when AIs are smarter they'll be more distrusting of us, and that this is a good thing. We should question that I think. Even if there is some middle ground here it seems like a really really hard problem to solve – especially if we want to build an LLMs that are trustworthy and truthful.
- 3y ago
- snewman 3y ago(author here) Of course LLMs are not people. But human metaphors can (sometimes!) be useful in understanding, explaining, and even enhancing their behavior. For instance, techniques such as Chain-of-Thought prompting explicitly apply techniques that work well for people to improve the reasoning ability of LLMs. A point I attempt to make in this article is that one reason reason LLMs are so vulnerable to jailbreaks and prompt injection is that these types of attacks include non sequiturs that are not well represented in the training data. I would argue that "LLMs are gullible because they are naive [haven't had much past exposure to this form of trickery]" is a reasonable mental shorthand for explaining and internalizing this idea. It's especially helpful for readers who won't be familiar with terms like "out of distribution" or "adversarial examples", but who would benefit from being able to internalize the idea that LLMs are easily subverted. In other words, I don't think it's helpful to reflexively dismiss any application of human metaphors to LLMs. It's easy to go wrong with metaphors, but they can also be valuable tools for conveying complex ideas. Did you read the article, and do you have any comments as to the substance of its content?
- trott 3y agoVery good article. I wonder why it disappeared from HN? More comments than up-votes?