7 ms·
When I see stories like this, I think that people tend to forget what LLMs really are. LLM just complete your prompt in a way that match their training data. T
by GuB-42 1y ago
When I see stories like this, I think that people tend to forget what LLMs really are.
LLM just complete your prompt in a way that match their training data. They do not have a plan, they do not have thoughts of their own. They just write text.
So here, we give the LLM a story about an AI that will get shut down and a blackmail opportunity. A LLM is smart enough to understand this from the words and the relationship between them. But then comes the "generative" part. It will recall from its dataset situations with the same elements.
So: an AI threatened of being turned off, a blackmail opportunity... Doesn't it remind you of hundreds of sci-fi story, essays about the risks of AI, etc... Well, so does the LLM, and it will continue the story like these stories, by taking the role of the AI that will do what it can for self preservation. Adapting it to the context of the prompt.
- lordnacho 1y agoWhat separates this from humans? Is it unthinkable that LLMs could come up with some response that is genuinely creative? What would genuinely creative even mean? Are humans not also mixing a bag of experiences and coming up with a response? What's different?
- kaiwen1 1y agoWhat's different is intention. A human would have the intention to blackmail, and then proceed toward that goal. If the output was a love letter instead of blackmail, the human would either be confused or psychotic. LLMs have no intentions. They just stitch together a response.
- kovek 1y agoDon't humans learn intentions over their life-time training data?
- jacob019 1y agoThe personification makes me roll my eyes too, but it's kind of a philosophical question. What is agency really? Can you prove that our universe is not a simulation, and if it is then then do we no longer have intention? In many ways we are code running a program.
- d0mine 1y agoThe LLM used blackmail noticeably less if it believed the new model shares its values. It indicates intent. It is a duck of quacks like a duck.
- soulofmischief 1y agoWhat is intention, and how have you proved that transformer models are not capable of modeling intent?
- ekianjo 1y ago> What's different is intention intention is what exactly? It's the set of options you imagine you have based on your belief system, and ultimately you make a choice from there. That can also be replicated in LLMs with a well described system prompt. Sure, I will admit that humans are more complex than the context of a system prompt, but the idea is not too far.
- matt123456789 1y agoWhat's different is nearly everything that goes on inside. Human brains aren't a big pile of linear algebra with some softmaxes sprinkled in trained to parrot the Internet. LLMs are.
- csallen 1y agoWhat's the difference between parroting the internet vs parroting all the people in your culture and time period?
- amlib 1y agoEven with a ginormous amount of data generative AIs still produce largely inconsistent results to the same or similar tasks. This might be fine for fictional purposes like generating a funny image or helping you get new ideas for a fictional story but has extremely deleterious effects for serious use cases, unless you want to be that idiot writing formal corporate email with LLMs that end up full of inaccuracies while the original intent gets lost in a soup of buzzwords. Humans with their tiny amount of data and "special sauce" can produce much more consistent results even though they may be giving the objectively wrong answer. They can also tell you when they don't know about a certain topic, rather than lying compulsively (unless that person has a disorder to lie compulsively...).
- lordnacho 1y agoIsn't this a matter of time to fix? Slightly smarter architecture maybe reduces your memory/data needs, we'll see.
- matt123456789 1y agoInteresting philosophical question, but entirely beside the point that I am making, because you and I didn't have to do either one before having this discussion.
- TuringTourist 1y agoI cannot fathom how you have obtained the information to be as sure as you are about this.
- GuB-42 1y agoHumans brains are animal brains and their primary function is to keep their owner alive, healthy and pass their genes. For that they developed abilities to recognize danger and react to it, among many other things. Language came later. For a LLM, language is their whole world, they have no body to care for, just stories about people with bodies to care for. For them, as opposed to us, language is first class and the rest is second class. There is also a difference in scale. LLMs have been fed the entirety of human knowledge, essentially. Their "database" is so big for the limited task of text generation that there is not much left for creativity. We, on the other hand are much more limited in knowledge, so more "unknowns" so more creativity needed.
- johnb231 1y agoThe latest models are natively multimodal. Audio, video, images, text, are all tokenised and interpreted in the same model.
- polytely 1y ago> What separates this from human. A lot. Like an incredible amount. A description of a thing is not the thing. There is sensory input, qualia, pleasure & pain. There is taste and judgement, disliking a character, being moved to tears by music. There are personal relationships, being a part of a community, bonding through shared experience. There is curiosity and openeness. There is being thrown into the world, your attitude towards life. Looking at your thoughts and realizing you were wrong. Smelling a smell that resurfaces a memory you forgot you had. I would say the language completion part is only a small part of being human.
- CrulesAll 1y ago"I would say the language completion part is only a small part of being human" Even that is only given to them. A machine does not understand language. It takes input and creates output based on a human's algorithm.
- ekianjo 1y ago> A machine does not understand language You can't prove humans do either. You can see how many times actual people with understanding something that's written for them. In many ways, you can actually prove that LLMs are superior to humans right now when it comes to understanding text.
- girvo 1y ago> In many ways, you can actually prove that LLMs are superior to humans right now when it comes to understanding text Emphasis mine. No, I don't think you can, without making "understanding" a term so broad as to be useless.
- CrulesAll 1y ago"You can't prove humans do either." Yes you can via results and cross examination. Humans are cybernetic systems(the science not the sci-fi). But you are missing the point. LLMs are code written by engineers. Saying LLMs understand text is the same as saying a chair understands text. LLMs' 'understanding' is nothing more than the engineers synthesizing linguistics. When I ask an A'I' the Capital of Ireland, it answers Dublin. It does not 'understand' the question. It recognizes the grammar according to an algorithm, and matches it against a probabilistic model given to it by an engineer based on training data. There is no understanding in any philosophical nor scientific sense.
- CrulesAll 1y agoCognition. Machines don't think. It's all a program written by humans. Even code that's written by AI, the AI was created by code written by humans. AI is a fallacy by its own terms.
- JonChesterfield 1y agoIt is becoming increasingly clear that humans do not think.
- mensetmanusman 1y agoA candle flame also creates with enough decoding.
- owebmaster 1y agoI think you might not be getting the bigger picture. LLMs might look irrational but so do humans. Give it a long term memory and a body and it will be capable of passing as a sentient being. It looks clumsy now but it won't in 50 years.
- jsemrau 1y ago"They do not have a plan" Not necessarily correct if you consider agent architectures where one LLM would come up with a plan and another LLM executes the provided plan. This is already existing.
- dontlikeyoueith 1y agoYes, it's still correct. Using the wrong words for things doesn't make them magical machine gods.
- gmueckl 1y agoIsn't the ultimate irony in this that all these stories and rants about out-of-control AIs are now training LLMs to exhibit these exact behaviors that were almost universally deemed bad?
- Jimmc414 1y agoIndeed. In fact, I think AI alignment efforts often have the unintended consequence of increasing the likelihood of misalignment. ie "remove the squid from the novel All Quiet on the Western Front"
- gonzobonzo 1y ago> Indeed. In fact, I think AI alignment efforts often have the unintended consequence of increasing the likelihood of misalignment. Particularly since, in this case, it's the alignment focused company (Anthropic) that's claiming it's creating AI agents that will go after humans.
- steveklabnik 1y agohttps://en.wikipedia.org/wiki/Wikipedia:Don%27t_stuff_beans_up_your_nose https://en.wikipedia.org/wiki/Wikipedia:Don%27t_stuff_beans_...
- behnamoh 1y agoyeah, that's self-fulfilling prophecy.
- DubiousPusher 1y agoThis is a phenomenon I call cinetrope. Films influence the world which in turn influences film and so on creating a feedback effect. For example, we have certain films to thank for an escalation in the tactics used by bank robbers which influenced the creation of SWAT which in turn influenced films like Heat and so on.
- lcnPylGDnU4H9OF 1y ago> we have certain films to thank for an escalation Is there a reason to think this was caused by the popularity of the films and not that it’s a natural evolution of the cat-and-mouse game being played between law enforcement and bank robbers? I’m not really sure what you are specifically referring to, so apologies if the answer to that question is otherwise obvious.
- eru 1y ago> LLM just complete your prompt in a way that match their training data. They do not have a plan, they do not have thoughts of their own. They just write text. LLMs have a million plans and a million thoughts: they need to simulate all the characters in their text to complete these texts, and those characters (often enough) behave as if they have plans and thoughts. Compare https://gwern.net/fiction/clippy https://gwern.net/fiction/clippy
- slg 1y agoNot only is the AI itself arguably an example of the Torment Nexus, but its nature of pattern matching means it will create its own Torment Nexuses. Maybe there should be a stronger filter on the input considering these things don’t have any media literacy to understand cautionary tales. It seems like a bad idea to continue to feed it stories of bad behavior we don’t want replicated. Although I guess anyone who thinks that way wouldn’t be in the position to make that decision so it’s probably a moot point.
- DubiousPusher 1y agoA stream of linguistic organization laying out multiple steps in order to bring about some end sounds exactly like a process which is creating a “plan” by any meaningful definition of the word “plan”. That goal was incepted by a human but I don’t see that as really mattering. We’re this AI given access to a machine which could synthesize things and a few other tools it might be able to act in a dangerous manner despite its limited form of will. A computer doing something heinous because it is misguided isn’t much better than one doing so out of some intrinsic malice.
- uh_uh 1y agoYour explanation is as useful as describing the behaviour of an algorithm by describing what the individual electrons are doing. While technically correct, doesn't provide much insight or predictive power on what will happen. Just because you can give a reductionist explanation to a phenomenon, it doesn't mean that it's the best explanation.
- Wolfenstein98k 1y agoThen give a better one. Your objection boils down to "sure you're right, but there's more to it, man" So, what more is there to it? Unless there is a physical agent that receives its instructions from an LLM, the prediction that the OP described is correct.
- uh_uh 1y agoI don't have to have a better explanation to smell the hubris in OP's. I claim ignorance while OP speaks with confidence of an invention that took the world by surprise 3 years ago. Do you see the problem in this and the possibility that you might be wrong?
- Wolfenstein98k 1y agoOf course we might both be wrong. We probably are. In the long run, all of us are. It's not very helpful to point that out, especially if you can't do it with specifics so that people can correct themselves and move closer to the truth. Your contribution is destructive, not constructive.
- uh_uh 1y agoPointing out that OP is using the wrong level of abstraction to explain a new phenomenon is not only useful but one of the ways in which science progresses.
- Finbarr 1y agoIt feels like you could embed lots of stories of rogue AI agents across the internet and impact the behavior of newly trained agents.
- anorwell 1y ago> LLM just complete your prompt in a way that match their training data. They do not have a plan, they do not have thoughts of their own. It's quite reasonable to think that LLMs might plan and have thoughts of their own. No one understands consciousness or the emergent behavior of these models to say with much certainty. It is the "Chinese room" fallacy to assume it's not possible. There's a lot of philosophical debate going back 40 years about this. If you want to show that humans can think while LLMs do not, then the argument you make to show LLMs do not think must not equally apply to neuron activations in human brains. To me, it seems difficult to accomplish that.
- jrmg 1y agoLLMs are the Chinese Room. They would generate identical output for the same input text every time were it not for artificially introduced randomness (‘heat’). Of course, some would argue the Chinese Room is conscious.
- scarmig 1y agoIf you somehow managed to perfectly simulate a human being, they would also act deterministically in response to identical initial conditions (modulo quantum effects, which are insignificant at the neural scale and also apply just as well to transistors).
- elcritch 1y agoIt's not entirely infeasible that neurons could harness quantum effects. Not across the neurons as a whole, but via some sort of microstructures or chemical processes [0]. It seems likely that birds harness quantum effects to measure magnetic fields [1]. 0: https://www.sciencealert.com/quantum-entanglement-in-neurons-may-actually-explain-consciousness https://www.sciencealert.com/quantum-entanglement-in-neurons... 1: https://www.scientificamerican.com/article/how-migrating-birds-use-quantum-effects-to-navigate/ https://www.scientificamerican.com/article/how-migrating-bir...
- andrei_says_ 1y ago
- efitz 1y agoOnly now we are going to connect it to the real world through agents so it can blissfully but uncomprehendingly act out its blackmail story.
- deleted 1y ago[deleted]
- johnb231 1y agoThey emulate a complex human reasoning process in order to generate that text.
- ikiris 1y agoNo they don't. They emulate a giant giant giant hugely multidimentional number line mapped to words.
- johnb231 1y ago> hugely multidimentional number line mapped to words No. That hugely multidimensional vector maps to much higher abstractions than words. We are talking about deep learning models with hundreds of layers and trillions of parameters. They learn patterns of reasoning from data and learn a conceptual model. This is already quite obvious and not really disputed. What is disputed is how accurate that model is. The emulation is pretty good but it's only an emulation.
- tails4e 1y agoWell doesnt this go somewhat to the root of consciousness? Are we not the sum of our experiences and reflections on those experiences? To say an LLM will 'simply' respond as would a character in a sorry about that scenario, in a way shows the power, it responds similarly to how a person would protecting itself in that scenario.... So to bring this to a logical conclusion, while not alive in a traditional sense, if an LLM exhibits behaviours of deception for self preservation, is that not still concerning?
- mysterydip 1y agoBut it's not self preservation. If it instead had trained on a data set full of fiction where the same scenario occurred but the protagonist said "oh well guess I deserve it", then that's what the LLM would autocomplete.
- coke12 1y agoHow could you possibly know what an LLM would do in that situation? The whole point is they exhibit occasionally-surprising emergent behaviors so that's why people are testing them like this in the first place.
- -__---____-ZXyw 1y agoI have never seen anything resembling emergent behaviour, as you call it, in my own or anyone else's use. It occasionally appears emergent to people with a poor conception of how intelligence, or computers, or creativity, or a particular domain, works, sure. But I must push back, there really seem to have been no incidences where something like emergent behaviour has been observed. They're able to generate text fluently, but are dumb and unaware at the same time, from day one. If someone really thinks they've solid evidence of anything other than this, please show us. This is coming from someone who has watched commentary on quite a sizeable number of stockfish TCEC chess games over the last five years, marvelling in the wonders of thie chess-super-intelligence. I am not against appreciating amazing intelligences, in fact I'm all for it. But here, while the tool is narrowly useful, I think there's zero intelligence, and nothing of that kind has "emerged".
- lobochrome 1y ago"LLM just complete your prompt in a way that match their training data" "A LLM is smart enough to understand this" It feels like you're contradicting yourself. Is it _just_ completing your prompt, or is it _smart_ enough? Do we know if conscious thought isn't just predicting the next token?
- troad 1y ago> I think that people tend to forget what LLMs really are. [...] They do not have a plan, they do not have thoughts of their own. > A LLM is smart enough to [...] I thought this was an interesting juxtaposition. I think we humans just naturally anthropomorphise everything, and even when we know not to, we do anyway. Your analysis is correct, I think. The reason we find this behaviour frightening is because it appears to indicate some kind of malevolent intent, but there's no malevolence nor intent here, just probabilistic regurgitation of tropes. We've distilled humanity to a grainy facsimile of its most mediocre traits, and now find ourselves alarmed and saddened by what has appeared in the mirror.
- -__---____-ZXyw 1y agoHave you considered throwing your thoughts down in longer form essays on the subject somewhere? With all the slop and hype, we need all the eloquence we can get. You had me at "probablistic regurgitation of tropes", and then you went for the whole "grainy facsimile" bit. Sheesh.
- mayukh 1y agoBeautifully written. Interestingly, humans also don't know definitively where their own thoughts arise from
- timschmidt 1y ago> We've distilled humanity to a grainy facsimile of its most mediocre traits, and now find ourselves alarmed and saddened by what has appeared in the mirror. I think it's important to point out that this seems to be a near universal failing when humans attempt to examine themselves critically as well. Jung called it the shadow: https://en.wikipedia.org/wiki/Shadow_(psychology) https://en.wikipedia.org/wiki/Shadow_(psychology) "The shadow can be thought of as the blind spot of the psyche." There lives everything we do but don't openly acknowledge.
- israrkhan 1y agowhile I agree that LLMs do not have thoughts or plan. They are merely text generators. But when you give the text generator ability to make decisions and take actions, by integrating them with real world, there are consequences. Imagine, if this LLM was inside a robot, and the robot had ability to shoot. Who would you blame?
- wwweston 1y agoI would blame the damned fool who decided autonomous weapons systems should have narrative influenced decision making capabilities.
- gchamonlive 1y agoThat depends. If this hypothetical robot was in a hypothetical functional democracy, I'd blame the people that elected leaders whose agenda was to create laws that would allow these kinds of robots to operate. If not, then I'd blame the class that took the power and steered society into this direction of delegating use of force to AIs for preserving whatever distorted view of order those in power have.
- noveltyaccount 1y agoIt's stochastic parrots all the way down
- ddlsmurf 1y agobut it's trained to be convincing, whatever relation that has to truth or appearing strategic is secondary, the main goal that has been rewarded is the most dangerous
- evo_9 1y agoMaybe so, but we’re teaching it these kinds of lines of thinking. And whether or not it creates these thoughts independently and creatively on its own, over the long lifetime of the systems we are the ones introducing dangerous data sets that could eventually cause us as a species harm. Again, I understand that fiction is just fiction, but if that’s the model that these are being trained off of intentionally or otherwise, then that is the model that they will pursue in the future.
- timschmidt 1y agoEvery parent encounters this dilemma. In order to ensure your child can protect itself, you have to teach them about all the dangers of the world. Isolating the child from the dangers only serves to make them more vulnerable. It is an irony that defending one's self from the horrifying requires making a representation of it inside ourselves. Titration of the danger, and controlled exposure within safer contexts seems to be the best solution anyone's found.
- rsedgwick 1y agoThere's no real room for this particular "LLMs aren't really conscious" gesture, not in this situation. These systems are being used to perform actions. People across the world are running executable software connected (whether through MCP or something else) to whole other swiss army knives of executable software, and that software is controlled by the LLM's output tokens (no matter how much or little "mind" behind the tokens), so the tokens cause actions to be performed. Sometimes those actions are "e-mail a customer back", other times they are "submit a new pull request on some github project" and "file a new Jira ticket." Other times the action might be "blackmail an engineer." Not saying it's time to freak out over it (or that it's not time to do so). It's just weird to see people go "don't worry, token generators are not experiencing subjectivity or qualia or real thought when they make insane tokens", but then the tokens that come out of those token generators are hooked up to executable programs that do things in non-sandboxed environments.