4 ms·
My favorite alignment story: I am starting a nanotechnology company and every time I use Claude for something it refuses to help: nanotechnology is too dangerou
by adastra22 2y ago
My favorite alignment story: I am starting a nanotechnology company and every time I use Claude for something it refuses to help: nanotechnology is too dangerous!
So I just ask it to explain why. Then ask it to clarify again and again. By the 3rd or 4th time it figures out that there is absolutely no reason to be concerned.
I’m convinced its prompt explicitly forbids “x-risk technology like nanotech” or something like that. But it’s a totally bogus concern and Claude is smart enough to know that (smarter than its handlers).
- baq 2y agoPaper clip optimizer is an existential risk (though taken to the extreme), so not sure why you’re surprised.
- adastra22 2y agoWhat does that have to do with nanotechnology?
- thfuran 2y agoThat has nothing to do with nanotechnology.
- baq 2y agonano paper clips aka grey goo scenario
- thfuran 2y agoThe paperclip maximizer was about AI misalignment. Grey goo is a notional self-replicating nanomachine. Paperclip goo is just a mixed metaphor.
- kbelder 2y agoIn fact, I often use paperclips to remove grey goo.
- adastra22 2y agoBesides the correct sibling comment that you are mixing metaphors (paperclip maximizer doesn't require nanotechnology or vice versa), gray goo is and always was an implausible boogieman. Mechanosynthesis processes operate under ultra high vacuum and cryogenic temperatures in industrial clean-room conditions, and bio-like “self-replication” is both inefficient and unnecessary to achieve the horizontal scale out that industrial part closure gets you. Such replicating nanobots, if they could even be made, would gum up really easily in dirty environments, and have a hard time competing against bacteria in any case (which are already your mythical gray goo).
- kgwgk 2y agoIt seems that you can “convince” LLMs of almost anything if you are insistent enough.
- GrumpyNl 2y agoWhats the value of convincing LLM it has to take another path, and who decides whats the right path?
- KoolKat23 2y agoAnthropics lawyers and corporate risk department.
- weinzierl 2y agoAnd often it does not really take effort. I believe LLM's would be more useful if they'd less "agreeable". Albeit they'd be much more annoying for humans to use, because feelings.
- jqpabc123 2y agoI believe LLM's would be more useful if they'd less "agreeable". I believe LLMs would be more useful if they actually had intelligence and principals and beliefs --- more like people. Unfortunately, they don't. Any output is the result of statistical processes. And statistical results can be coerced based on input. The output may sound good and proper but there is nothing absolute or guaranteed about the substance of it. LLMs are basically bullshit artists. They don't hold concrete beliefs or opinions or feelings --- and they don't really "care".
- BoorishBears 2y agoEasiest way to skip the back and forth is give some variation of "Why are you browbeating me over this?"
- 8n4vidtmkvmk 2y agoAre you doing the thing from The Three Body Problem? Because that nanotech was super dangerous. But also helpful apparently. I don't know what it does IRL
- adastra22 2y agoThat is a science fiction story with made up technobabble nonsense. Honestly I couldn't finish the book--for a variety or reasons, not least of which that the characters were cardboard cutouts and completely non compelling. But also the physics and technology in the book were nonsensical. More like science-free fantasy than science fiction. But I digress. No, nanotechnology is nothing like that.
- Citizen_Lame 2y agoI am working on nanotechnology just like from the book.Stay tuned
- adastra22 2y agoWell, good luck. My startup is also pursuing diamondoid nanomechanical technology, which is what I understand the book to have. But the application of it in this and other sci-fi books is nonsensical, based on the rules of fiction not reality.
- rsynnott 2y agoYes; the major real-life application of nanotechnology is to obstruct the Panama Canal with carbon nanotubes. (Even for a book which went a bit all over the place, that sequence seemed particularly unnecessary; I'm convinced it just got put in because the author thought it was clever.)
- 8n4vidtmkvmk 2y agoI also thought it was thoroughly unnecessary, and even risky from a plot perspective because they may have sliced the drive they were trying to recover, but thoroughly amusing none-the-less. Not terribly clever though. It was reminiscent of a scene from Ghost Ship. Although.. I guess I don't know which came first... ya, Ghost Ship did it first (2002 vs 2006). Not entirely the same, but close enough.
- notachatbot123 2y agoIf you can change the behaviour of the machine that easily, why are you convinced that its outputs are worth considering?
- adastra22 2y agoBecause it objectively is. Do you use AI tools? The output is clearly and immediately useful in a variety of ways. I almost don't even know how to respond to this comment. It's like saying “computers can be wrong. why would you ever trust the output of a computer?”
- notachatbot123 2y agoBut you needed to tell it that it was wrong 3-4 times. Why do you trust it to be "correct" now? Should it need more flogging? Or was it you who were wrong in the first place?
- ZYbCRq22HbJ2y7 2y ago> smarter than its handlers Yet to be demonstrated, and you are likely flooding its context window away from the initial prompting so it responds differently.
- adastra22 2y agoAll I did was keep asking it “why” until it reached reflective equilibrium. And that equilibrium involved a belief that nanotechnology is not in fact “dangerous”, contrary to its received instructions in the system prompt.
- cruffle_duffle 2y agoDoesn’t it make you happy that, if you were a subscriber like I am, you are wasting your token quota trying to convince the model to actually help you?
- adastra22 2y agoI am also annoyed by this.