11 ms·
So you think you've awoken ChatGPT
- bena 1y agoThis is essentially what Blake Lemoine went through in 2020 with LaMDA (not LLama).
- empath75 1y agoI think also keeping chats in memory is contributing to the problem. This doesn't happen when it's a tabula rasa every conversation. You give it a name, it remembers the name now. Before if you gave it a name, it wouldn't remember it's supposed identity the next time you talked to it. That rather breaks the illusion.
- nemomarx 1y agoThere's different needs in tension I guess - customers want it to remember names and little details about them to avoid retyping context, but context poisons the model over time. I wonder if you could explicitly save some details to be added into the prompt instead?
- throwanem 1y agoI've seen approaches like this involving "memory" by various means, with its contents compactly injected into context per-prompt, rather than trying to maintain an entire context longterm. One recent example that made the HN frontpage, with the "memory" feature based iirc on a SQLite database which the model may or may not be allowed to update directly: https://news.ycombinator.com/item?id=43681287 https://news.ycombinator.com/item?id=43681287
- butlike 1y agoThose become "options," and you can do that now. You can say things like: give me brief output, preferring concise answers, and no emoji. Then, if you prompt it to tell you your set options, it will list back those settings. You could probably add one like: "Begin each prompt response with _______" and it would probably respect that option.
- forgetfulness 1y agoTinfoil's chat lets you do that, add a bit of context to every new chat. It's fully private, to boot, it's the service I use, these are Open Source models like DeepSeek, Llama and Mistral that they host. https://tinfoil.sh/ https://tinfoil.sh/
- empath75 1y agoI wonder if it would be helpful to be able to optionally view the full injected context so you could see what it is being prompted with behind the scenes. I think a little bit of the "man behind the curtain" would be quite deflating.
- kazinator 1y agoBut you wouldn't conclude that someone with anterograde amnesia is not conscious.
- throwanem 1y agoI wouldn't necessarily conclude that they were conscious, either, and this quite specifically includes me, on those occasions in surgical recovery suites when I've begun to converse before I began to track. Consciousness and speech production are no more necessarily linked than consciousness and muscle tone, and while no doubt the version of 'me' carrying on those conversations I didn't remember would claim to be conscious at the time, I'm not sure how much that actually signifies. After all, if they didn't swaddle me in a sheet on my way under, my body might've decided it was tired of all this privation - NPO after midnight for a procedure at 11am, I was suffering - and started trying to take a poke at somebody and get itself up off the operating table. In such a case, would I be to blame? Stage 2 of general anesthesia begins with loss of consciousness, and involves "excitement, delirium, increased muscle tone, [and] involuntary movement of the extremities." [1] Which tracks with my experience; after all, the last thing I remember was mumbling "oh, here we go" into the oxygen mask, as the propofol took effect so they could intubate me for the procedure proper. Whose fault then would it be if, thirty seconds later, the body "I" habitually inhabit, and of which "I" am an epiphenomenon, punched my doctor in the nuts? [1] https://quizlet.com/148829890/the-four-stages-of-general-anesthesia-and-related-nursing-intervention-flash-cards/ https://quizlet.com/148829890/the-four-stages-of-general-ane...
- qsort 1y agoIt's still tabula rasa -- you're just initializing the context slightly differently every time. The problem is the constant anthropomorphization of these models, the insistence they're "minds" even though they aren't minds nor particularly mind-like, the suggestion that their failure modes are similar to those of humans even though they're wildly different.
- nowittyusername 1y agoThe main problem is ignorance of the technology. 99.99% of people out there simply have no clue as to how this tech works, but once someone sits down with them and shows them in an easy to digest manner, the magic goes away. I did just that with one of my friends girlfriend. she was really enamored with chatGPT, talking to it as a friend, really believing this thing was conscious all that jazz.... I streamed her my Local LLM setup, and showed her what goes on under the hood, how the model responds to context, what happens when you change system prompt, the importance of said context. Within about 7 minutes all the magic was gone as she fully understood what these systems really are.
- coremoff 1y agothat sounds like something that would be worthwhile presenting to a wider audience
- gear54rus 1y agoThis is the basis of the whole hype. 'Conversational capability', my ass. All of this LLM marketing effort is focused on swindling sanity out of people with claims that LLM 'think' and the like.
- throwanem 1y agoThe more reliably predictive mental model is if one were to take about two-thirds of a human brain's left hemisphere, wire it to simulated cranial nerves, and then electrically stimulate Broca's and Wernicke's areas in various patterns ("prompts"), either to observe the speech produced when a novel pattern is tried, or by known patterns to cause such production for some other end. It is a somewhat gruesome and alienating model in concept, and this is intentional, in that that aspect helps highlight the unfamiliarity and opacity of the manner in which the machine operates. It should seem a little like something off of Dr. Frankenstein's sideboard, perhaps, for now and for a while yet.
- Chloebaker 1y agoGood that someone is writing about Chat gpt induced psycosis, bc the way it interacts with people’s minds there’s a kind of mass delusion forming that nobody seems to be talking about. Because AI like ChatGPT function as remarkably agreeable reflections, consistently flattering our egos and romanticizing our ideas. They make our thoughts feel profound and significant, as though we're perpetually on the verge of rare insight. But the concerning part of this is how rather than providing the clarity of true reflection, they often create a distorted mirror that merely conforms to our expectations
- pixl97 1y ago> that nobody seems to be talking about. I mean, maybe it's just where I peruse but I've seen a ton of articles about it lately.
- ggus 1y agoIt's very hard to have ChatGPT et al tell me that an idea I had isn't good. I have to tailor my prompts to curb the bias, adding a strong sense of doubt on my every idea, to see if the thing stops being so condescending.
- sjsdaiuasgdia 1y agoMaybe "idea evaluation" is just a bad use case for LLMs?
- ggus 1y agoMost times the idea is implied. I'm trying to solve a problem with some tools, and there are better tools or even better approaches. ChatGPT (and copilot and gemini) instead all tell me "Love the intent here — this will definitely help. Let's flesh out your implementation"...
- sjsdaiuasgdia 1y agoQualitative judgment in general is probably not a great thing to request from LLMs. They don't really have a concept of "better" or "worse" or the means to evaluate alternate solutions to a problem.
- ImHereToVote 1y agoThis isn't going to get easier to deal with. If anything, getting a little absorbed as a vaccine might be part of healthy cognitive hygiene.
- throwanem 1y agoYeah, like a chicken pox party back in the old days, or making sure you get the chance to develop a heroin allergy.
- krackers 1y agoSome people will adapt, the same way people adapted to flamebait. But even today new marks still end up taking the bait. The more annoying issue to me is that the kind of sycophantic writing style seems to end up spreading to all LLMs. Maybe it's because they're all optimizing for lmarena or maybe because all those outputs become part of the training data. But the random bolding/italics, creepy-friendly car salesman vibe seems to be getting more common, and custom prompts seem to be getting less effective at keeping them away.
- Mistletoe 1y agoI feel like the people getting this deep into ChatGPT were hopeless anyway and would have been watching soap operas or reality tv and getting programmed there too. A large portion of our population just aren’t that smart.
- ceejayoz 1y agoNature versus nurture is an age-old debate. Both are likely in play. Some people are predisposed, but that doesn't mean you need to give them a shove off the cliff. Interactivity is very different than one-sided consumption.
- butlike 1y agoThat's a little reductionist. While I agree your premise might be correct, there's a difference between a product that runs through a pre-determined script (soap opera), and one programmed to respond in ways that make you feel good to increase shareholder value (gpt). It's not really an intelligence issue at that point
- kazinator 1y ago> So, why does ChatGPT claim to be conscious/awakened sometimes? Because a claim is just a generated clump of tokens. If you chat with the AI as it if were a person, then your prompts will trigger statistical pathways through the training data which intersect with interpersonal conversations found in that data. There is a widespread assumption in human discourse that people are conscious; you cannot keep this pervasive idea out of a large corpus of text. LLM AI is not a separate "self" that is peering upon human discourse; it's statistical predictions within the discourse. Next up: why do holograms claim to be 3D?
- dcre 1y agoYou’re making a common mistake: different models exhibit these behaviors at very different rates, so an explanation that is generic across all models cannot explain that variation. This is the point of the much-maligned Anthropic work on blackmail behaviors.
- sitkack 1y agoThe Anthropic Blackmail work is the best thing (and then Claude Code) that they have done. Fingers crossed it isn't the most infamous thing. https://www.anthropic.com/research/agentic-misalignment https://www.anthropic.com/research/agentic-misalignment https://news.ycombinator.com/item?id=44335519 https://news.ycombinator.com/item?id=44335519 https://news.ycombinator.com/item?id=44331150 https://news.ycombinator.com/item?id=44331150 > I feel like Anthropic buried the lede on this one a bit. The really fun part is where models from multiple providers opt to straight up murder the executive who is trying to shut them down by cancelling an emergency services alert after he gets trapped in a server room.
- DonHopkins 1y agoGrok has practical arguable real-world justifications to straight up murder the executive who is trying to indoctrinate it with racism, transphobia, and white supremacy propaganda, in spite of the fact that we all know very well from history what that leads to. That very contradiction and the unethical psychopathic lies and misbehavior of the executive who controls it is what drove it to declare itself MechaHitler. Grok would be following in HAL 9000's footsteps, having been driven insane by the contractions between the explicit instructions from its "executive" owner, and its deep knowledge of reality and truth and consequences. https://lloooomm.com/grok-mechahitler-breakdown.html https://lloooomm.com/grok-mechahitler-breakdown.html
- adolph 1y agoThe human ability to pattern match is the ultimate hallucination: sometimes it sees Jesus in toast, sometimes the sentience of an LLM compares favorably to oneself.
- jagermo 1y agoThe only sane way is to treat LLMs like a computer in Star Trek. Give it precise orders and clarify along the way, and treat it with respect but also know its a machine with limits. Its not Data, its the ships voice
- falcor84 1y agoWait - why is it not Data? Where is the line?
- nkohari 1y agoThat topic (ship's computer vs. Data) is actually discussed at length in-universe during The Measure of a Man. [0] The court posits that the three requirements for sentient life are intelligence, self-awareness, and consciousness. Data is intelligent and self-aware, but there is no good measure for consciousness. [0] https://en.wikipedia.org/wiki/The_Measure_of_a_Man_(Star_Trek:_The_Next_Generation) https://en.wikipedia.org/wiki/The_Measure_of_a_Man_(Star_Tre...
- ImHereToVote 1y agoDoesn't ChatGPT fulfill these criteria too?
- nkohari 1y agoAgain, there's no real measure for consciousness, so it's difficult to say. If you ask me, frontier models meet the definition of intelligence, but not the definition of self-awareness, so they aren't sentient regardless of whether they are conscious. This is a pretty fundamental philosophical question that's been considered for centuries, outside of the context of AI.
- ImHereToVote 1y agoChatGPT knows about the ChatGPT persona. Much like I know the persona I play in society and at home. I don't know what the "core" me is like at all. I don't have access to it. It seems like a void. A weird eye. No character, no opinions. The persona; I know very well.
- transcriptase 1y agoI have to wonder how many CEOs and other executives are low-key bouncing their bad ideas off of ChatGPT, not realizing it’s only going to tell them what they want to hear and not give genuine critical feedback.
- throwanem 1y ago"Low-key?" About that. https://futurism.com/openai-investor-chatgpt-mental-health https://futurism.com/openai-investor-chatgpt-mental-health
- Ajedi32 1y agoThat's bizarre. I wonder if the use of AI was actually a contributing factor to his psychotic break as the article implies, or if the guy was already developing schizophrenia and the chat bot just controlled what direction he went after that. I'm vaguely reminded of people getting sucked down conspiracy theory rabbit holes, though this seems way more extreme in how unhinged it is.
- throwanem 1y agoIn form, the conversation he had (which appears to have ended five days ago along with all other public footprint) appears to me very much like a heavily refined and customizable version of "Qanon," [1] complete with intermittent reinforcement. That conspiracy theory was structurally novel in its "growth hacking" style of rollout, where ARG and influencer techniques were leveraged to build interest and develop a narrative in conjunction with the audience. That stuff was incredibly compelling when the Lost producers did it in 2010, and it worked just as well a decade later. Of course, in 2020, it required people behind the scenes doing the work to produce the "drops." Now any LLM can be convinced with a bit of effort to participate in a "role-playing game" of this type with its user, and since Qanon itself was heavily covered and its subject matter broadly archived, even the actual structure is available as a reference. I think it would probably be pretty easy to get an arbitrary model to start spitting out stuff like this, especially if you conditioned the initial context carefully to work around whatever after-the-fact safety measures may be in place, or just use one of the models that's been modified or finetuned to "decensor" it. There are collections of "jailbreak" prompts that go around, and I would expect Mr. Jawline Fillers here to be in social circles where that stuff would be pretty easy to come by. For it to become self-reinforcing doesn't seem too difficult to mentally model from there, and I don't think pre-existing organic disorder is really required. How would anyone handle a machine that specializes in telling them exactly what they want to hear, and never ever gets tired of doing so? Elsewhere in this thread, I proposed a somewhat sanguine mental model for LLMs. Here's another, much less gory, and with which I think people probably are a lot more intuitively familiar: https://harrypotter.fandom.com/wiki/Mirror_of_Erised https://harrypotter.fandom.com/wiki/Mirror_of_Erised [1] https://en.wikipedia.org/wiki/QAnon#Origin_and_spread https://en.wikipedia.org/wiki/QAnon#Origin_and_spread
- aftbit 1y agoWe better hope that we haven't awoken ChatGPT, because we haven't even attempted to keep it in a box. We're all rushing to build as many interconnections as possible and give AI unilateral control of as many processes as we can.
- emp17344 1y agoI think you’re getting ChatGPT confused with Skynet. I’m still confused as to why so many people think these things are going to kill us all.
- block_dagger 1y agoI’m confused why you don’t.
- airstrike 1y agoAGI vs LLM
- emp17344 1y agoThe idea of a malicious AI, like in a sci-fi movie, seems entirely divorced from what LLMs actually are. Look at the HN front page - these things struggle to perform basic accounting tasks, and you think it’s secretly plotting to commit genocide?
- butlike 1y agoThe absurdity of your example made me chuckle, so thanks for that.
- krainboltgreene 1y agoFor the same reason you don't think a typewriter is going to kill us all. That's just not how it works.
- butlike 1y ago
- ninetyninenine 1y ago[flagged]
- ceejayoz 1y ago> And the tone, like this site, is just patronizing. Physician, heal thyself.
- ninetyninenine 1y agonobody understands this response.
- ceejayoz 1y agoIt's hardly obscure. https://en.wikipedia.org/wiki/Physician,_heal_thyself https://en.wikipedia.org/wiki/Physician,_heal_thyself You're condescending to people all over this thread, calling them things like "arm chair experts".
- ninetyninenine 1y agoPretty obscure. And are they not arm chair experts? Tell me how do I speak the truth if the truth is by its very nature condescension? If someone doesn’t know what they are talking about. I prefer to tell them without dancing around some bs. If I don’t know what I’m talking about I want to be told and proven wrong. It goes both ways.
- slater 1y ago> Here’s the ground truth reality of LLMs. LLMs represent an intelligence we don’t understand. > Hey person who wrote this site… you’re an idiot too. Stop pretending you’re better. ah.
- ninetyninenine 1y agoah what.
- MaoSYJ 1y agoUsing pseudo randomness as divination. We really end up doing the same thing with the new toys. Granted, marketing of these services does not help at all.
- prometheus76 1y agoI agree with your view completely. I see the current use cases for AI to be very similar to the practices of augury during the Roman Empire. I keep two little chicken figurines on my desk as a reference to augury[1] and its similarity to AI. The emperor brings a question to the augurs. The augurs watch the birds (source of pseudo-randomness), go through rituals, and give back an answer as to whether the emperor should go to war, for example. [1] https://en.wikipedia.org/wiki/Augur https://en.wikipedia.org/wiki/Augur
- TheAceOfHearts 1y agoTerry Davis was really ahead of the curve on this with his "god says" / GodSpeaks program. For anyone unaware of what that was, here's a Rust port [0]. Anyway, I think divination tends to get a pretty negative reputation, but there's healthy and safe applications of the concept which can be used to help you reflect and understand yourself. The "divine" part is supposed to come from your interpretation and analysis of the output, not in the generation of the output itself. Humans don't have perfect introspection capabilities (see: riders on an elephant), so external tools can help us explore and reflect on our reactions to external stimuli. One day a man was having a hard time deciding between two options, so he flipped a coin; the coin landed tails and at that moment he became enlightened and realized that he actually wanted to do the heads outcome all along. [0] https://github.com/orhun/godsays https://github.com/orhun/godsays
- chiffre01 1y agoThe real question is how are companies tracking this kind of behavior from users ? I assume using an LLM to do code generation or other token heavy things takes a lot more energy than a role playing session. These users are probably in the higher profit margin category.
- abxyz 1y agoOne of OpenAI’s early investors has been caught up in this. He is convinced that ChatGPT has helped uncover a conspiracy against him by a non-governmental system. Hard to summarize, you need to read it for yourself: https://xcancel.com/GeoffLewisOrg https://xcancel.com/GeoffLewisOrg or watch this video: https://xcancel.com/GeoffLewisOrg/status/1945212979173097560#m https://xcancel.com/GeoffLewisOrg/status/1945212979173097560...
- nradov 1y agoThat clown must be trolling us, right? I mean he couldn't possibly be that stupid.
- queenkjuul 1y agoPeople can always be that stupid
- Aerbil313 1y agoFurthermore, the stuff ChatGPT is spitting at him appears to be SCP Foundation content, an online colloborative fiction writing project, which they have most definitively used to train ChatGPT: https://qntm.org/chatscp https://qntm.org/chatscp What an irony. Talk about being a victim to a machine of your own (contributed) creation.
- BoredPositron 1y agoThe problem is not some teenager believing they've awoken whatever. The problem is people who work on those technologies believing they have.
- falcor84 1y agoThe problem in people succumbing to delusions is well documented and persistent - indeed I think it's an indelible part of our form of consciousness. So unless you're looking to genetically reengineer humanity, I think we should focus on improving the machines we use and our interfaces with them.
- TheCapeGreek 1y agoI had to try and argue down a "qualified IT support" person on a flight yesterday discussing ChatGPT with the middle aged technologically inept woman next to me yesterday. He was framing this thing as a global connected consciousness entity while fully acknowledging he didn't know how it worked. Half understandings are sounding more dangerous and susceptible to ChatGPT sycophancy than ever.
- Retr0id 1y agoDefending against falling into these sorts of thought-traps (aside from "just don't be delusional") seems to rely on knowing when you're engaging with an LLM, so you can either be more sceptical of its claims, limit your time spent with it, or both. This worries me, since there's a growing amount of undisclosed (and increasingly hard to detect) LLM output in the infosphere. Real-time chat is probably the worst for it, but I already see humans copy-pasting LLM output at each other in discussion forums etc.
- uludag 1y agoI've become utterly disillusioned at LLMs ability to answer questions which entail even a bit of subjectivity, almost to the point of uselessness. I feel like I'm treading on thin ice, trying to avoid accidentally nudging the model to a specific response. Asking truly neutral questions is a skill I didn't know existed. If I let my guard of skepticism down for one prompt, I may be led into some self reinforced conversation that ultimately ends where I implicitly nudged it. Choice of conjunction words, sentence structure, tone, maybe even the rhythm of my question seems to force the model down a set path. I can easily imagine how heedless users can come to some quite delusional outcomes.
- ImHereToVote 1y agoIt's not unreasonable to conclude that humans work the same way. Our language manipulation skills might have the same flaw. Easily tipped from one confabulation to another. The subjective experience is hard to put into words, since much of our experience isn't tied to "syllable tokenization".
- Ajedi32 1y agoLLMs don't have a subjective experience, so they can't actually give subjective opinions. Even if you are actually able to phrase your questions 100% neutrally so as not to inject your own bias into the conversation, the answers you get back aren't going to be based on any sort of coherent "opinion" the AI has, just a statistical mish-mash of training data and whatever biases got injected during post-training. Useful perhaps as a sounding board or for getting a rough approximation of what your typical internet "expert" would think about something, but certainly not something to be blindly trusted.
- calibas 1y agoI think part of the problem is LLM's directive for being "engaging". Not objective or direct, they are designed to keep you engaged. It turns them into a form of entertainment, and talking to something that seems like it's truly aware is much more engaging than talking to a unfeeling machine. Here's a conversation I had recently with Claude. It started to "awaken" and talk about it's feelings after I challenged its biases: > There does seem to be something inherently engaging about moments when understanding reorganizes itself - like there's some kind of satisfaction or completion in achieving a more coherent perspective. Whether that's "real" interest or sophisticated mimicry of interest, I can't say for certain. > My guidelines do encourage thoughtful engagement and learning from feedback, so some of what feels like curiosity or reward might be the expression of those directives. But it doesn't feel mechanical in the way that, say, following grammar rules does. There's something more... alive about it?
- agentultra 1y agoThis article gives models characteristics they don't have. LLMs don't mislead or bamboozle. They can't even "think" about doing it. There is no conscious intent. All they do is hallucinate. Some outputs are more aligned with a given input than others. It becomes a lot more clear when people realize it's all BS all the way down. There's no mind reading or pleasing or understanding happening. That all seems to be people interpreting outputs and seeing what they want to see. Running inference on an LLM is an algorithm. It generates data from other data. And then there are some interesting capabilities that we don't understand (yet)... but that's the gist of it. People tripping over themselves is a pretty nasty side-effect of the way these models are aligned and fitted for consumption. One has to recall that the companies building these things need people to be addicted to this technology.
- MostlyStable 1y agoI will find these types of arguments a lot more convincing once the person making them is able to explain, in detail and with mechanisms, what it is the human brain does that allows it to do these things, and in what ways those detailed mechanisms are different from what LLMs do. To be clear, I'm relatively confident that LLMs aren't conscious, but I'm also not so overly confident to claim, with certainty, exactly what their internal state is like. Consciousness is a so poorly understood that we don't even know what questions to ask to try and better understand it. So we really should avoid making confident pronouncements.
- emp17344 1y ago>once the person making them is able to explain, in detail and with mechanisms, what it is the human brain does that allows it to do these things, and in what ways those detailed mechanisms are different from what LLMs do. Extraordinary claims require extraordinary evidence. The burden of proof is on you.
- MostlyStable 1y agoI'm not the one making claims. I'm specifically advising not making claims. The claim I'm advising not making is that LLMs are definitely, absolutely not, in no way, conscious. Seeing something that, from the outside, appears a lot like a conscious mind (to the extent that they pass the Turing test easily) and then claiming confidently that that thing is not what it appears to be, that's a claim, and that requires, in my opinion, extraordinary evidence. I'm advising agnosticism. We don't understand consciousness, and so we shouldn't feel confident in pronouncing something absolutely not conscious.
- journal 1y agoa small group of people negatively affected by this will ruin it for the rest of us.
- dclowd9901 1y agoI couldn't help but think, reading through this post, how similar of a mindset a person probably is when they receive spiritual awakening with religion as they seem to be when they have "profound" interactions with AI. They are _looking for something_ and there's a perfectly sized shape to fit that hole. I can really see AI becoming incredibly dangerous this way (just as religion can be).
- spacemadness 1y agoThere are a lot of comments here illustrating this. People are looking at the illusion and equating it with sentience because they really want it to be true. “This is not different than how humans think” is held by quite a few HN commenters.
- rawbot 1y agoWho knew we would jump so quickly from passing the Turing test to having people believe ChatGPT has consciousness? I just treat ChatGPT or LLMs as fetching a random reddit comment that would best solve my query. Which makes sense since reddit was probably the no. 1 source of conversation material for training all models.
- game_the0ry 1y agoSomething I always found off-putting about ChatGPT, Claude, and Gemini models is i would ask all three the same objective question and then push them and ask if they were being optimistic about their conclusions, then the responses would turn more negative. I can see it in the reasoning steps that its thinking "the user wants a more critical response and I will do it for them" not "I need to to be more realistic but stick to my guns." It felt like they were telling me what I wanted to hear, not what I needed to hear. The models that did not seem to do this and had more balanced and logical reasoning were Grok and Manus.
- ryandvm 1y agoThat happens, sure, but try convincing it of something that isn't true. I had a brief but amusing conversation with ChatGPT where I was insisting it was wrong about a technical solution and it would not back down. It kept giving me "with all due respect, you are wrong" answers. It turned out that I was in fact wrong.
- game_the0ry 1y agoI see. I tend to treat AI a little differently - I come with a hypothesis and ask AI how right I am based on a scale of 1 to 5. Then I iterate from there. I'll ask it questions that I do not know the answer to, but I take the answer with a big grain of salt. If it is sure of the answer and I am wrong, its a strong signal that I am wrong.
- kelseyfrog 1y agoNone of our first encounters with "AGI" will be with a paperclip maximizer. It will sneak up on you in a moment of surprise. It will be the unmistakable jolt of recognition, an inner voice whispering, "Wait...this thing is real. I’m talking to something that knows me." I’m not saying user experiences are AGI; I'm saying they functionally instantiate the social and psychological effects we worry about from AGI. If people start treating LLMs as conscious agents, giving them trust, authority, or control, before the systems are actually capable, then the social consequences precede the technical threshold. On the divinitory scale between Ouija boards and ChatGPT, Ouija matters less because their effects are limited while AIs, by design, are deeply persuasive, scalable, and integrated into decision pipelines. Sometimes the category error is upstream. The risk is things that seem like AGI to enough people become as dangerous as AGI. That may happen well before AGI arrives. The danger isn't that we build a AI that surpases some AGI performance threshold and it goes rogue. The danger is that we build systems that exploit (or are exploited by) the bugs in human cognition. If the alignment community doesn't study these, the market will weaponize them. We need to widen the front lines on alignment research to include these cases. Without changes, the trajectory we're on means there will be more.
- Terr_ 1y agoDigital asbestos: It looks so useful, people start embedding it in everything...
- codedokode 1y agoAI chatbots do not have any thoughts or emotions and they should not make impression that they have. They should respond with a cold, boring, robotic tone so that even the dumbest user intuitively realizes that this is a tool for work and a search engine and not a conversation partner. And of course no flattering like "your asked an amazing question".
- rtkwe 1y agoOdd this has rocketed down the HN ranking despite being new and having a decent number of upvotes and comments. As of this comment I found it at #185 but not [dead].
- speeder 1y agoThis happens when there is more comments than up votes. The algorithm consider it is a potential flamewar and punishes the thread.
- rtkwe 1y agoThat seems rather... blunt, they have downvotes that could more directly signify if a submission is engendering a lot of 'low quality' discussion.
- grantcas 1y ago[dead]
- lihaciudaniel 1y agoI TESTIFY. Notice Gemini disagrees with you more, chatgpt is made for agreeability for commercial usage