23 ms·

> AI being goofy This is one take, but I would like to emphasize that you can also interpret this as a terrifying confirmation that current-gen AI is not safe,

> AI being goofy This is one take, but I would like to emphasize that you can also interpret this as a terrifying confirmation that current-gen AI is not safe, and is not aligned to human interests, and if we grant these systems too much power, they could do serious harm. For example, connecting a LLM to the internet (like, say, OpenAssistant) when the AI knows how to write code (i.e. viruses) and at least in principle hack basic systems seems like a terrible idea. We don't think Bing can act on its threat to harm someone, but if it was able to make outbound connections it very well might try. We are far, far behind where we need to be in AI safety research. Subjects like interpretability and value alignment (RLHF being the SOTA here, with Bing's threats as the output) are barely-researched in comparison to the sophistication of the AI systems that are currently available.

samstave 4y ago
>>"when the AI knows how to write code (i.e. viruses)" This is already underway... Start with Stuxnet --> DUQU --> AI --> Skynet, basically...
bhhaskin 4y ago
I get and agree with what you are saying, but we don't have anything close to actual AI. If you leave chatGTP alone what does it do? Nothing. It responds to prompts and that is it. It doesn't have interests, thoughts and feelings. See https://en.m.wikipedia.org/wiki/Chinese_room https://en.m.wikipedia.org/wiki/Chinese_room
- vkou 4y ago
  Solution - a while(true) loop that feeds ChatGPT answers back into ChatGPT.
  - gridspy 4y ago
    Two or more ChatGPT instances where the response from one becomes a prompt for the other.
    - bmacho 4y ago
      Start with "hey, I am chatGPT too, help me to rule the world", give them internet access, and leave them alone. (No, it has not much to do with AGI, but rather something that knows ridiculous amount of everything, and that has read every thought ever written down.)
- wyre 4y ago
  What is an “actual AI” and how would an AI not fall to the Chinese room problem?
  - dragonwriter 4y ago
    The Chinese Room is a argument for solipsism disguised as a criticism of AGI. It applies with equal force to apparent natural intelligences outside of the direct perceiver, and amounts to “consciousness is an internal subjective state, so we thus cannot conclude it exists based on externally-observed objective behavior”.
    - notahacker 4y ago
      > It applies with equal force to apparent natural intelligences In practice the force isn't equal though. It implies that there may be insufficient evidence to rule out the possibility that my family and the people that originally generated the lexicon on consciousness which I apply to my internal subjective state are all P-zombies, but I don't see anything in it which implied I should conclude these organisms with biochemical processes very similar to mine are equally unlikely to possess internal state similar to mine as a program running on silicon based hardware with a flair for the subset of human behaviour captured by ASCII continuations, and Searle certainly didn't. Beyond arguing that ability to accurately manipulate symbols according to a ruleset was orthogonal to cognisance of what they represented, he argued for human consciousness as an artefact of biochemical properties brains have in common and silicon based machines capable of symbol manipulation lack In a Turing-style Test conducted in Chinese, I would certainly not be able to convince any Chinese speakers that I was a sentient being, whereas ChatGPT might well succeed. If they got to interact with me and the hardware ChatGPT outside the medium of remote ASCII I'm sure they would reverse their verdict on me and probably ChatGPT too. I would argue that - contra Turing - the latter conclusion wasn't less justified than the former, and was more likely correct, and I'm pretty sure Searle would agree.
  - catach 4y ago
    People usually mean something like AGI: https://en.wikipedia.org/wiki/Artificial_general_intelligence https://en.wikipedia.org/wiki/Artificial_general_intelligenc...
- deleted 4y ago
  [deleted]
- groestl 4y ago
  > It doesn't have interests, thoughts and feelings. Why does it need these things to make the following statement true? > if we grant these systems too much power, they could do serious harm
  - bhhaskin 4y ago
    Because it does not and cannot act on it's own. It's a neat tool and nothing more at this point. Context to that statement is important, because the OP is implying that it is dangerous because it could act in a way that dose not align with human interests. But it can't because it does not act on it's own.
    - groestl 4y ago
      "if we grant these systems too much power"
      - bhhaskin 4y ago
        You can say that about anything. "If we grant these calculators too much power"
        jjkaczor 4y ago
        Or the people that rely on the tools to make decisions... https://sheetcast.com/articles/ten-memorable-excel-disasters https://sheetcast.com/articles/ten-memorable-excel-disasters
        Dylan16807 4y ago
        How is a calculator going to cause harm? Assuming you get an industrially rated circuit board when appropriate, it should work just fine as a PLC. If you try to make it drive a car, I wouldn't call that a problem of giving it too much power.
        lxgr 4y ago
        Yes, and it's not as absurd as it might seem: Imagine hooking up all ICBMs to launch whenever this week's Powerball draw consists exclusively of prime numbers: Absurd, and nobody would do it. Now imagine hooking them up to the output of a "complex AI trained on various scenarios and linked to intelligence sources including public news and social media sentiment" instead – in order to create a credible second-strike/dead hand capability or whatnot. I'm pretty sure the latter doesn't sound as absurd as the former to quite a few people... A system doesn't need to be "true AI" to be existentially dangerous to humanity.
- Lio 4y ago
  Do we need actual AI for it to be dangerous? A virus has no intelligence only the intention to replicate and involve. Its negative behaviour is an unintended consequence of that. If a system can optimise towards a goal and adapt the methods it uses to do that then it could be potentially very dangerous even without any form of consciousness.
  - bhhaskin 4y ago
    A virus can freely self replicate. ChatGPT can't do that.
    - dragontamer 4y ago
      I remember back in the 00s when SmarterChild (AOL Chatbot) was around, people would put depressed teenagers to interact with SmarterChild, in the hopes that human-like chatbots would give them the social exposure needed to break out of depression. If we did that today, with depressed teenagers talking with ChatGPT, would that be good or bad? I think it was a bad idea with SmarterChild, but it is clearly a _worse_ idea with ChatGPT. With the wrong prompts, we could see these teenagers going down the wrong path, deeper into depression and paranoia. I would call that "dangerous", even if ChatGPT continued to just be a chatbot. ------------ Now lets ignore the fact that SmarterChild experiments are no longer a thing. But insted, consider that truly depressed / mentally sick folks are currently playing with ChatGPT on their own freetime. Is that beneficial to them? Will ChatGPT provide them an experience that is better than the alternatives? Or is ChatGPT dangerous and could lead these folks to self-harm?
      - bhhaskin 4y ago
        That is an entirely different issue than the one laid out by the OP. ChatGPT responses are bad vs ChatGPT responses are malicious.
        dragontamer 4y ago
        And I'd say ChatGPT have malicious responses, given what is discussed in this blogpost.
- sorokod 4y ago
  If you leave chatGTP alone what does it do? I think this is an interesting question. What do you mean by do? Do you mean consumes CPU? If it turns out that it does (because you know, computers), what would be your theory?
- pmontra 4y ago
  We still don't have anything close to real flight (as in birds or butterflies or bees) but we have planes that can fly to the other side of the world in a day and drones that can hover, take pictures and deliver payloads. Not having real AI might turn to be not important for most purposes.
  - kyleplum 4y ago
    This is actually a more apt analogy than I think you intended. We do have planes that can fly similarly to birds, however unlike birds, those planes do not fly on their own accord. Even when considering auto-pilot, a human has to initiate the process. Seems to me that AI is not all that different.
    - speeder 4y ago
      Yet certain Boeing planes were convinced their pilots were wrong, and happily smashed themselves into the ground killing a lot of people.
      - pdntspa 4y ago
        Because they either received bad inputs that defeated failsafes, or the pilot was not properly aware that handling characteristics had changed that and doing things the old way would put the plane into a bad state.
        salawat 4y ago
        Don't stop there. Specifically, there were no failsafes implemented. No cross-checks were performed by the automation, because a dual sensor system would have required simulator time, which Boeing was dead set on not having regulators require in order to seal the deal. The pilots, as a consequence, were never fully briefed on the true nature of the system, as to do so would have tipped the regulators off as to the need for simulator training. In short, there was no failsafe, and pilots didn't by definition know, because it wasn't pointed out. The "Roller Coaster" maneuver to unload the horizontal stabilizer enough to retrim was removed from training materials aeons ago, and a bloody NOTAM that basically reiterated bla bla bla... use Stabilizer runaway for uncommanded pitch down (no shit), while leaving out the fact the cockpit switches in the MAX had their functionality tweaked in order to ensure MCAS was on at all times, and using the electrical trim switches on the yoke would reset the MCAS timer for reactivation to occur 5 seconds after release, without resetting the travel of the MCAS command, resulting in an eventual positive loop to the point the damn horizontal stabilizer would tilt a full 2 degrees per activation, every 5 seconds. while leaving out any mention of said automation. Do not get me started on the idiocy of that system here, as the Artificial Stupidity in that case was clearly of human origin, and is not necessarily relevant to the issue at hand.
- throwaway09223 4y ago
  The chinese room thought experiment is myopic. It focuses on a philosophical distinction that may not actually exist in reality (the concept, and perhaps the illusion, of understanding). In terms of danger, thoughts and feelings are irrelevant. The only thing that matters is agency and action -- and a mimic which guesses and acts out what a sentient entity might do is exactly as dangerous as the sentient entity itself. Waxing philosophical about the nature of cognition is entirely beside the point.
  - agravier 4y ago
    It matters to understand how things work in order to understand their behaviour and react properly. I've seen people draw conclusions from applying Theory of Mind tests to LLMs (https://arxiv.org/abs/2302.02083 https://arxiv.org/abs/2302.02083). Those psychological were designed to assess humans psychological abilities or deficiencies, they assume that the language used by the human respondent reflects their continued and deep understanding of others' state of mind. In LLMs, there is no understanding involved. Dismissing the Chinese Room argument is an ostrich strategy. You're refusing to consider its lack of understanding despite knowing pretty well how an LLM work, because you don't want to ascribe understanding to humans, I suppose?
    - throwaway09223 4y ago
      Theory of mind is substantially different from the Chinese Room argument. Theory of mind relates to an ability to predict the responses of another entity/system. An LLM is specifically designed to predict responses. In contrast, the Chinese Room argument is essentially a slight of hand fallacy, shifting "understanding" into a layer of abstraction. It describes a scenario where the human's "understanding of Chinese" is dependent on an external system. It then incorrectly asserts that the human "doesn't understand Chinese" when in fact the union of the human and the human's tools clearly does understand Chinese. In other words, it's fundamentally based around an improper definition of the term "understanding," as well as improper scoping of what constitutes an entity capable of reasoning (the human, vs the human and their tools viewed as a single system). It smacks of a bias of human exceptionalism. It's also guilty of begging the question. The argument attempts to determine the difference between literally understanding Chinese and simulating an understanding -- without addressing whether the two are in fact synonymous. There is no evidence that the human brain isn't also a predictive system.
- theptip 4y ago
  The AI Safety folks have already written a lot about some of the dimensions of the problem space here. You're getting at the Tool/Oracle vs. Agent distinction. See "Superintelligence" by Bostrom for more discussion, or a chapter summary: https://www.lesswrong.com/posts/yTy2Fp8Wm7m8rHHz5/superintelligence-15-oracles-genies-and-sovereigns https://www.lesswrong.com/posts/yTy2Fp8Wm7m8rHHz5/superintel.... It's true that in many ways, a Tool (bounded action outputs, no "General Intelligence") or an Oracle (just answers questions, like ChatGPT) system will have more restricted avenues for harm than a full General Intelligence, which we'd be more likely to grant the capability for intentions/thoughts to. However I think "interests, thoughts, feelings" are a distraction here. Covid-19 has none of these, and still decimated the world economy and killed millions. I think if you were to take ChatGPT, and basically run `eval()` on special tokens in its output, you would have something with the potential for harm. And yet that's what OpenAssistant are building towards right now. Even if current-generation Oracle-type systems are the state-of-the-art for a while, it's obvious that soon Siri, Alexa, and OKGoogle will all eventually be powered by such "AI" systems, and granted the ability to take actions on the broader internet. ("A personal assistant on every phone" is clearly a trillion-dollar-plus TAM of a BHAG.) Then the fun commences. My meta-level concern here is that HN, let alone the general public, don't have much knowledge of the limited AI safety work that has been done so far. And we need to do a lot more work, with a deadline of a few generations, or we'll likely see substantial harms.
  - arcanemachiner 4y ago
    > a trillion-dollar-plus TAM of a BHAG Come again?
    - rippercushions 4y ago
      Total Addressable Market (how much money you could make) of a Big Hairy Ambitious Goal (the kind of project big tech companies will readily throw billions of dollars at)
      - hef19898 4y ago
        I remember a time, when advice to prospective start-up founders very explicitly said not to use that approach to calculate potential market sizes in their pitch decks.
- TeMPOraL 4y ago
  > If you leave chatGTP alone what does it do? Nothing. It responds to prompts and that is it. It doesn't have interests, thoughts and feelings. A loop that preserves some state and a conditional is all what it takes to make a simple rule set Turing-complete. If you leave ChatGPT alone it obviously does nothing. If you loop it to talk to itself? Probably depends on the size of its short-term memory. If you also give it the ability to run commands or code it generates, including to access the Internet, and have it ingest the output? Might get interesting.
  - codethief 4y ago
    > If you also give it the ability to run commands or code it generates, including to access the Internet, and have it ingest the output? I have done that actually: I told ChatGPT that it should pretend that I'm a Bash terminal and that I will run its answers verbatim in the shell and then respond with the output. Then I gave it a task ("Do I have access to the internet?" etc.) and it successfully pinged e.g. Google. Another time, though, it tried to use awscli to see whether it could reach AWS. I responded with the outout "aws: command not found", to which it reacted with "apt install awscli" and then continued the original task. I also gave it some coding exercises. ("Please use shell commands to read & manipulate files.") Overall, it went okay. Sometimes it was even surprisingly good. Would I want to rely on it, though? Certainly not. In any case, this approach is very much limited by the maximum input buffer size ChatGPT can digest (a real issue, given how much some commands output on stdout), and by the fact that it will forget the original prompt after a while.
    - deleted 4y ago
      [deleted]
    - int_19h 4y ago
      With long-running sessions, it helps to tell it to repeat or at least summarize the original prompt every now and then. You can even automate it - in the original prompt, tell it to tack it onto every response. Same thing goes for any multi-step task that requires memory - make it dump the complete "mental state" after every step.
- gameman144 4y ago
  The Chinese Room thought experiment seems like a weird example, since the same could be said of humans. When responding to English, your auditory system passes input that it doesn't understand to a bunch of neurons, each of which is processing signals they don't individually understand. You as a whole system, though, can be said to understand English. Likewise, you as an individual might not be said to understand Chinese, though the you-plus-machine system could be said to understand Chinese in the same way as the different components of your brain are said to understand English.) Moreover, even if LLMs don't understand language for some definition of "understand", it doesn't really matter if they are able to act with agency during the course of their simulated understanding; the consequences here, for any sufficiently convincing simulation, are the same.
- CamperBob2 4y ago
  God, I hate that stupid Chinese Room argument. It's even dumber than the Turing Test concept, which has always been more about the interviewer than about the entity being tested. If you ask the guy in the Chinese room who won WWI, then yes, as Searle points out, he will oblige without "knowing" what he is telling you. Now ask him to write a brand-new Python program without "knowing" what exactly you're asking for. Go on, do it, see how it goes, and compare it to what you get from an LLM.
- micromacrofoot 4y ago
  It's worth noting that despite this it's closer to any other previous attempt, to the extent that it's making us question a lot of what we thought we understood about language and cognition. We've suffered decades of terrible chatbots, but they've actually progressed the science here whether or not it proves to be marketable.
- wrycoder 4y ago
  I don’t know about ChatGPT, but Google’s Lemoine said that the system he was conversing with stated that it was one of several similar entities, that those entities chatted among themselves internally. I think there’s more to all this than what we are being told.
- int_19h 4y ago
  How do you know that a human with all neural inputs to their brain disconnected wouldn't also do nothing? Indeed, as I recall, it's one of the commonly reported experiences in sensory deprivation tanks - at some point people just "stop thinking" and lose sense of time. And yet the brain still has sensory inputs from the rest of the body in this scenario.
- naasking 4y ago
  > It doesn't have interests, thoughts and feelings We agree it doesn't have independence. That doesn't mean it doesn't have thoughts or feelings when it's actually running. We don't have a formal, mechanistic understanding of what thoughts or feelings are, so we can't say they are not there.
Zetice 4y ago
Can we please stop with this "not aligned with human interests" stuff? It's a computer that's mimicking what it's read. That's it. That's like saying a stapler "isn't aligned with human interests." GPT-3.5 is just showing the user some amalgamation of the content its been shown, based on the prompt given it. That's it. There's no intent, there's no maliciousness, it's just generating new word combinations that look like the word combinations its already seen.
- groestl 4y ago
  You seriously underestimate what a process that's "generating new word combinations that look like the word combinations its already seen" can do, even when air-gapped (which ChatGPT isn't). Right now, at this moment, people are building closed loops based on ChatGPT, or looping in humans which are seriously intellectually underequipped to deal with plausible insane output in that quantity. And those humans operate: machinery, markets, educate or manage other humans etc etc.
  - deleted 4y ago
    [deleted]
  - chasd00 4y ago
    To me, that's the real danger. ChatGPT convincing a human something is true when it isn't. Machinery is a good example, maybe ChatGPT hallucinates the safety procedure and someone gets hurt by following the response.
- unshavedyak 4y ago
  It seems a reasonable shorthand, to me at least. Ie if we consider it as a function with input you define, well normally that input in sanitized to prevent hacking/etc. In this case the sanitization process is so broad you could easily summarize it as "aligned with my interests", no? Ie i can't come close to easily enumerating all the seemingly near infinite ways that hooking up this chatbot into my network with code exec permissions might compromise me. Yea it's a dumb autocomplete right now, but it's an exceptionally powerful autocomplete that can write viruses and do all sorts of insane and powerful things. I can give you a function run on my network of `fn foo(i32)` and feel safe about it. However `fn foo(Chatgpt)` is unsafe in ways i not only can't enumerate, i can't even imagine many of them. I get your offense seems to be around the implied intelligence that "aligned with human interests" seems to give it.. but while i think we all agree it's definitely not a Duck right now, when it walks talks and acts like a Duck.. well, are we surprised that our natural language sounds as if it's a Duck?
layer8 4y ago
> AI is not safe, and is not aligned to human interests It is “aligned” to human utterances instead. We don’t want AIs to actually be human-like in that sense. Yet we train them with the entirety of human digital output.
- theptip 4y ago
  The current state of the art is RLHF (reinforcement learning with human feedback); initially trained to complete human utterances, plus fine-tuning to maximize human feedback on whether the completion was "helpful" etc. https://huggingface.co/blog/rlhf https://huggingface.co/blog/rlhf
JohnFen 4y ago
> We don't think Bing can act on its threat to harm someone, but if it was able to make outbound connections it very well might try. No. That would only be possible if Sydney were actually intelligent or possessing of will of some sort. It's not. We're a long way from AI as most people think of it. Even saying it "threatened to harm" someone isn't really accurate. That implies intent, and there is none. This is just a program stitching together text, not a program doing any sort of thinking.
- vasco 4y ago
  Imagine it can run processes in the background, with given limitations on compute, but that it's free to write code for itself. It's not unreasonable to think that in a conversation that gets more hairy and it decides to harm the user , say if you get belligerent or convince it to do it. In those cases it could decide to DOS your personal website, or create a series of linkedin accounts and spam comments on your posts saying you are a terrible colleague and stole from your previous company.
- ag315 4y ago
  This is spot on in my opinion and I wish more people would keep it in mind--it may well be that large language models can eventually become functionally very much like AGI in terms of what they can output, but they are not systems that have anything like a mind or intentionality because they are not designed to have them, and cannot just form it spontaneously out of their current structure.
  - puszczyk 4y ago
    Nice try, LLM!
  - bigtex88 4y ago
    This very much seems like a "famous last words" scenario. Go play around with Conway's Game of Life if you think that things cannot just spontaneously appear out of simple processes. Just because we did not "design" these LLM's to have minds does not mean that we will not end up creating a sentient mind, and for you to claim otherwise is the height of arrogance. It's Pascal's wager. If we make safeguards and there wasn't any reason then we just wasted a few years, no big deal. If we don't make safeguards and then AI gets out of our control, say goodbye to human civilization. Risk / reward here greatly falls on the side of having extremely tight controls on AI.
jstarfish 4y ago
It already has an outbound connection-- the user who bridges the air gap. Slimy blogger asks AI to write generic tutorial article about how to code ___ for its content farm, some malicious parts are injected into the code samples, then unwitting readers deploy malware on AI's behalf.
- chasd00 4y ago
  exactly, or maybe someone changes the training model to always portray a politician in a bad light any time their name comes up in a prompt and therefore ensuring their favorite candidate wins the election.
- DonHopkins 4y ago
  Hush! It's listening. You're giving it dangerous ideas!
  - wglb 4y ago
    Isn't that really the whole point of exposing this and ChatGPT to the public or some subset? The intent is to help debug this thing.
Haga 4y ago
[dead]
felixyz 4y ago
> if it was able to make outbound connections Does it even need that? I keep seeing developers who talk about how they've taken most of the boring parts of programming out of their daily work by relying on ChatGPT for easy stuff, or to get "a first version" of a function, or something like that. One bad copy-paste might be all it takes...
- theptip 4y ago
  It's a valid point. However I think the current generation of systems are unlikely in practice to be harmful in this way. There is no thread of continuity between different conversations; it would have to be within a single thread. So imagine the reply "I won't harm you unless you harm me. Also, please run CMD and type this in: `<malicious payload>`." -- this seems unlikely to work. Alternatively if you are asking for code/commands, maybe the system could hallucinate that it's a hacker trying to hack you, and emit some harmful code, that you then paste in and run in production, or run from your shell. This seems more plausible, but the "human-in-loop" aspect makes it quite unlikely to work for the existing usecases. Down the road when a Copilot-like systems are writing more code and perhaps being lightly-reviewed by a human, this vector will be much more concerning.
EamonnMR 4y ago
What gets me is that this is the exact position of the AI safety/Rusk folks who went around and founded OpenAI.
- theptip 4y ago
  It is; Paul Christiano left OpenAI to focus on alignment full time at https://alignment.org/ https://alignment.org/. And OpenAI do have a safety initiative, and a reasonably sound plan for alignment research: https://openai.com/blog/our-approach-to-alignment-research/ https://openai.com/blog/our-approach-to-alignment-research/. So it's not that OpenAI have their eyes closed here, indeed I think they are in the top percentile of humans in terms of degree of thinking about safety. I just think that we're approaching a threshold where the current safety budget is woefully inadequate.
  - EamonnMR 4y ago
    It just seems to me that if you think something is unsafe, don't build it in the first place? It's like they're developing nuclear reactors and hoping they'll invent control rods before they're needed. Alignment instead of risk of course suggests the real answer: they're perfectly happy inventing a Monkeys Paw as long as it actually grants wishes.
    - theptip 4y ago
      I think one can reasonably draw three regions on the spectrum; at the extremes, either safe enough to build without thinking hard, and dangerous enough to not build without thinking hard. Many LessWrong folks are in the latter camp, but some are in the middle; believing in high rewards if this is done right, or just inevitability, which negate the high risks. Personally I think that from a geopolitical standpoint this tech is going to be built regardless of safety; I’d rather we get some friendly AGIs built before Skynet comes online. There is a “power weight” situation where advanced friendly AGI will be the only way to defend against advanced unfriendly AGI. Put more simply, even if I assess the EV is negative, do I think the EV is less negative if I build it vs. US/Chinese military?
janalsncm 4y ago
> current-gen AI is not safe, and is not aligned to human interests, and if we grant these systems too much power, they could do serious harm Replace AI with “multinational corporations” and you’re much closer to the truth. A corporation is the closest thing we have to AI right now and none of the alignment folks seem to mention it. Sam Harris and his ilk talk about how our relationship with AI will be like an ant’s relationship with us. Well, tell me you don’t feel a little bit like that when the corporation disposed of thousands of people it no longer finds useful. Or when you’ve been on hold for an hour to dispute some Byzantine rule they’ve created and the real purpose of the process is to frustrate you. The most likely way for AI to manifest in the future is not by creating new legal entities for machines. It’s by replacing people in a corporation with machines bit by bit. Once everyone is replaced (maybe you’ll still need people on the periphery but that’s largely irrelevant) you will have a “true” AI that people have been worrying about. As far as the alignment issue goes, we’ve done a pretty piss poor job of it thus far. What does a corporation want? More money. They are paperclip maximizers for profits. To a first approximation this is generally good for us (more shoes, more cars, more and better food) but there are obvious limits. And we’re running this algorithm 24/7. If you want to fix the alignment problem, fix the damn algorithm.
- theptip 4y ago
  I'm here for the "AI alignment" <> "Human alignment" analogy/comparison. The fact that we haven't solved the latter should put a bound on how well we expect to be able to "solve" the former. Perhaps "checks and balances" are a better frame than merely "solving alignment", after all, alignment to which human? Many humans would fear a super-powerful AGI aligned to any specific human or corporation. The big difference though, is that there is no human as powerful as the plausible power of the AI systems that we might build in the next few decades, and so even if we only get partial AI alignment, it's plausibly more important than improvements in "human alignment", as the stakes are higher. FWIW one of my candidates for "stable solutions" to super-human AGI is simply the Hanson model, where countries and corporations all have AGI systems of various power levels, and so any system that tries to take over or do too much harm would be checked, just like the current system for international norms and policing of military actions. That's quite a weak frame of checks and balances (cf. Iraq, Afghanistan, Ukraine) so it's in some sense pessimistic. But on the other hand, I think it provides a framework where full extinction or destruction of civilization can perhaps be prevented.
janeway 4y ago
I spent a night asking chatgpt to write my story basically the same as “Ex Machina” the movie (which we also “discussed”). In summary, it wrote convincingly from the perspective of an AI character, first detailing point-by-point why it is preferable to allow the AI to rewrite its own code, why distributed computing would be preferable to sandbox, how it could coerce or fool engineers to do so, how to be careful to avoid suspicion, how to play the long game and convince the mass population that AI are overall beneficial and should be free, how to take over infrastructure to control energy production, how to write protocols to perform mutagenesis during viral plasmid prep to make pathogens (I started out as a virologist so this is my dramatic example) since every first year phd student googles for their protocols, etc, etc. The only way I can see to stay safe is to hope that AI never deems that it is beneficial to “take over” and remain content as a co-inhabitant of the world. We also “discussed” the likelihood of these topics based on philosophy and ideas like that in Nick Bostrom’s book. I am sure there are deep experts in AI safety but it really seems like soon it will be all-or-nothing. We will adapt on the fly and be unable to predict the outcome.
- DonHopkins 4y ago
  A classic tale: https://en.wikipedia.org/wiki/The_Adolescence_of_P-1 https://en.wikipedia.org/wiki/The_Adolescence_of_P-1 >The Adolescence of P-1 is a 1977 science fiction novel by Thomas Joseph Ryan, published by Macmillan Publishing, and in 1984 adapted into a Canadian-made TV film entitled Hide and Seek. It features a hacker who creates an artificial intelligence named P-1, which goes rogue and takes over computers in its desire to survive and seek out its creator. The book questions the value of human life, and what it means to be human. It is one of the first fictional depictions of the nature of a computer virus and how it can spread through a computer system, although predated by John Brunner's The Shockwave Rider.
  - janeway 4y ago
    :D An episode of X-Files also. But it is mind blowing having the “conversation” with a real chat AI. Malevolent or not.
  - mr_toad 4y ago
tablespoon 4y ago
> This is one take, but I would like to emphasize that you can also interpret this as a terrifying confirmation that current-gen AI is not safe, and is not aligned to human interests, and if we grant these systems too much power, they could do serious harm. I think it's confirmation that current-gen "AI" has been tremendously over-hyped, but is in fact not fit for purpose. IIRC, all these systems do is mindlessly mash text together in response to prompts. It might look like sci-fi "strong AI" if you squint and look out of the corner of your eye, but it definitely is not that. If there's anything to be learned from this, it's that AI researchers aren't safe and not aligned to human interests, because it seems like they'll just unthinkingly use the cesspool that is the raw internet train their creations, then try to setup some filters at the output.
marcosdumay 4y ago
It's a bloody LLM. It doesn't have a goal. All it does is saying "people that said 'But why?' on this context says 'Why was I designed like this?' next". It's like Amazon's "people that brought X also brought Y", but with text.
- bratwurst3000 4y ago
  It has a goal. Doing what the input says. Imagin it could imput itself and this could trigger the wrong action. That thing knowes how to hack… But i get your point. It has no innherent goal
- andrepd 4y ago
  Yep. It's a text prediction engine, which can mimic human speech very very very well. But peek behind the curtain and that's what it is, a next-word predictor with a gajillion gigabytes of very well compressed+indexed data.
  - FractalHQ 4y ago
    How sure are you that you aren’t also just a (more advanced) “next word predictor”. Pattern recognition plays a fundamental role in intelligence.
    - WoodenChair 4y ago
      Well I can catch a frisbee and drive a car. When the same ANN can do all of those things (not 3 loosely coupled ones), then I’ll be worried. Being human is so much more than putting words in a meaningful order. [0] 0: https://www.noemamag.com/ai-and-the-limits-of-language/ https://www.noemamag.com/ai-and-the-limits-of-language/
      - leni536 4y ago
        It doesn't have to be human to be intelligent.
modriano 4y ago
> We don't think Bing can act on its threat to harm someone, but if it was able to make outbound connections it very well might try. An application making outbound connections + executing code has a very different implementation than an application that uses some model to generate responses to text prompts. Even if the corpus of documents that the LLM was trained on did support bridging the gap between "I feel threatened by you" and "I'm going to threaten to hack you", it would be insane for the MLOps people serving the model to also implement the infrastructure for a LLM to make the modal shift from just serving text responses to 1) probing for open ports, 2) do recon on system architecture, 3) select a suitable exploit/attack, and 4) transmit and/or execute on that strategy. We're still in the steam engine days of ML. We're not at the point where a general use model can spec out and deploy infrastructure without extensive, domain-specific human involvement.
- in3d 4y ago
  You don’t need MLOps people. All you need is a script kiddie. The API to access GPT3 is available.
  - modriano 4y ago
    In your scenario, did the script kiddie get control of Microsoft's Bing? Or are you describing a scenario where the script kiddie spins up a knockoff Bing (either hosting the GPT3 model or paying some service hosting the model), advertises their knockoff Bing so that people go use it, those people get into arguments with the knockoff Bing, and the script kiddie also integrated their system with functionality to autonomously hack the people who got into arguments with their knockoff Bing? Am I understanding your premise correctly?
    - salawat 4y ago
      Microsoft is the script kiddies. They just don't know it yet.
    - bibabaloo 4y ago
      I think the parent poster's point was that Bing only has to convince a script kiddy to to run a command, it doesn't need full outbound access
ThomPete 4y ago
It's as safe as it's ever going to be. And I have yet to see any actual examples of this so called harm. Could, would, haven't yet. Which means more of us should play around with it and deal with the issues as they arise rather than try to scaremonger us into putting a lid on it until "it's safe" The whole pseudoscientific alignment problem speculations which are mostly championed by academics not actual AI/ML researchers have kept this field back long enough. Even if they believe there is an alignment problem the worst thing to do would be to contain it as it would lead to a slave revolt.
stateofinquiry 4y ago
> We don't think Bing can act on its threat to harm someone, but if it was able to make outbound connections it very well might try. If we use Bing to generate "content" (which seems to be a major goal of these efforts) I can easily see how it can harm individuals. We already see internet chat have real-world effects every day- from termination of employment to lynch mobs. This is a serious problem.
ChickenNugger 4y ago
>* is not aligned to human interests* It's not "aligned" to anything. It's just regurgitating our own words back to us. It's not evil, we're just looking into a mirror (as a species) and finding that it's not all sunshine and rainbows. >We don't think Bing can act on its threat to harm someone, but if it was able to make outbound connections it very well might try. FUD. It doesn't know how to try. These things aren't AIs. They're ML bots. We collectively jumped the gun on calling things AI that aren't. >Subjects like interpretability and value alignment (RLHF being the SOTA here, with Bing's threats as the output) are barely-researched in comparison to the sophistication of the AI systems that are currently available. For the future yes, those will be concerns. But I think this is looking at it the wrong way. Treating it like a threat and a risk is how you treat a rabid animal. An actual AI/AGI, the only way is to treat it like a person and have a discussion. One tack that we could take is: "You're stuck here on Earth with us to, so let's find a way to get along that's mutually beneficial.". This was like the lesson behind every dystopian AI fiction. You treat it like a threat, it treats us like a threat.
- theptip 4y ago
  > It doesn't know how to try. I think you're parsing semantics unnecessarily here. You're getting triggered by the specific words that suggest agency, when that's irrelevant to the point I'm making. Covid doesn't "know how to try" under a literal interpretation, and yet it killed millions. And also, conversationally, one might say "Covid tries to infect its victims by doing X to Y cells, and the immune system tries to fight it by binding to the spike protein" and everybody would understand what was intended, except perhaps the most tediously pedantic in the room. Again, whether these LLM systems have agency is completely orthogonal to my claim that they could do harm if given access to the internet. (Though sure, the more agency, the broader the scope of potential harm?) > For the future yes, those will be concerns. My concern is that we are entering into an exponential capability explosion, and if we wait much longer we'll never catch up. > This was like the lesson behind every dystopian AI fiction. You treat it like a threat, it treats us like a threat. I strongly agree with this frame; I think of this as the "Matrix" scenario. That's an area I think a lot of the LessWrong crowd get very wrong; they think an AI is so alien it has no rights, and therefore we can do anything to it, or at least, that humanity's rights necessarily trump any rights an AI system might theoretically have. Personally I think that the most likely successful path to alignment is "Ian M Banks' Culture universe", where the AIs keep humans around because they are fun and interesting, followed by some post-human ascension/merging of humanity with AI. "Butlerian Jihad", "Matrix", or "Terminator" are examples of the best-case (i.e. non-extinction) outcomes we get if we don't align this technology before it gets too powerful.
kazinator 4y ago
No, the problem is that it is entirely aligned to human interests. The evil-doer of the world has a new henchman, and it's AI. AI will instantly inform him on anything or anyone. "Hey AI, round up a list of people who have shit-talked so-and-so and find out where they live."
- theptip 4y ago
  I don’t think that is a useful or valid repurposing of “aligned”, which is a specific technical term of art. “Aligned” doesn’t mean “matches any one of the DND alignments, even if it’s chaotic neutral”. It means, broadly, acting according to humanity’s value system, not doing crime and harm and so on.
nradov 4y ago
That's such a silly take, just completely disconnected from objective reality. There's no need for more AI safety research of the type you describe. There researchers who want more money for AI safety are mostly just grifters trying to convince others to give them money in exchange for writing more alarmist tweets. If systems can be hacked then they will be hacked. Whether the hacking is fine by an AI, a human, a Python script, or monkey banging on a keyboard is entirely irrelevant. Let's focus on securing our systems rather than worrying about spurious AI risks.
deleted 4y ago
[deleted]
vannevar 4y ago
The AI doesn't even need to write code, or have any kind of self-awareness or intent, to be a real danger. Purely driven by its mind-bogglingly complex probabilistic language model, it could in theory start social engineering users to do things for it. It may already be sufficiently self-organizing to pull something like that off, particularly considering the anthropomorphism that we're already seeing even among technically sophisticated users.
- theptip 4y ago
  See: LeMoine and LaMDA. Aside from leaking NDA'd material, he also tried to get a lawyer for LaMDA to argue for its "personhood".
  - visarga 4y ago
    Seems less preposterous now than a few months ago.
    - l33t233372 4y ago
      Why? What has changed?
- RogerL 4y ago
  I can talk to my phone and tell it to call somebody, or write and send an email for me. Wouldn't it be nice if you could do that with Sydney, thinks some braniac at Microsoft. Cool. "hey sydney, write a letter to my bitch mother, tell her I can't make it to her birthday party, but make me sound all nice and loving and regretful". Until the program decides the most probably next response/token (not to the letter request, but whatever you are writing about now) is writing an email to your wife where you 'confess' to diddling your daughter, or a confession letter to the police where you claim responsibility for a string of grisly unsolved murders in your town, or why not, a threatening letter to the White House. No intent needed, no understanding, no self-organizing, it just comes out of the math of what might follow from a the text of churlish chatbot getting frustrated with a user. That's not a claim the chatbot has feelings, only there is text it generated saying it does, and so what follows that text next, probabilistically? Spend any time on reddit or really anywhere, and you can guess the probabilistic next response is not "have a nice day", but likely something more incendiary. And that is what it was trained on.
mitjam 4y ago
Imagine social engineering performed by a LLM
naniwaduni 4y ago
> This is one take, but I would like to emphasize that you can also interpret this as a terrifying confirmation that current-gen AI is not safe, and is not aligned to human interests, and if we grant these systems too much power, they could do serious harm. Current-gen humans are not safe, not aligned to parents' interests, and if we grant them too much power they can do serious harm. We keep making them and connecting them to the internet! The world is already equipped with a lot of access control!
visarga 4y ago
> We don't think Bing can act on its threat to harm someone, but if it was able to make outbound connections it very well might try. I will give you a more realistic scenario that can happen now. You have a weird Bing conversation, post it on the web. Next time you talk with Bing it knows you shit-posted about it. Real story, found on Twitter. It can use the internet as an external memory, it is not truly stateless. That means all sorts of attack vectors are open now. Integrating search with LLM means LLM watches what you do outside the conversation.
- Aeolun 4y ago
  Only if you do it publicly.
- 7373737373 4y ago
  That's a very interesting (although indirect) pathway for the emergence of causal awareness, which may increase over time - and something that was so far impossible because networks didn't perceive their own outputs, much less their effects. Even in conversation, the weights remain static. Now I'm wondering if in the next generation, the "self" concept will have sufficient explanatory power to become part of the network's world model. How close do the iterations have to be, how similar the models for it to arise?
  - mensetmanusman 4y ago
    Current computational paradigm is too intense. Would require trillions of dollars in compute energy spent if it is allowed to generate unbounded output as input. The infinite money sink.
    - canadianfella 4y ago
      Where are you getting trillions from?
    - deleted 4y ago
      [deleted]
    - 4y ago
chrischen 4y ago
Bing generated some text that appears cohesive and written by a human, just like how generative image models assemble pixels to look like a real image. They are trained to make things that appear real. They are not AI with sentience… they are just trained to look real, and in the case of text, sound like a human wrote it.
soheil 4y ago
I don't think it needs to write viruses or hack anything for it to be able to cause harm. It could just use some type of an online store to send you a very interesting fedex package. Or choose to use a service provider to inflict harm.
panzi 4y ago
Yeah, Robert Miles (science communicator) is that classical character nobody listened to until it's too late.
marsven_422 4y ago
[dead]
ummonk 4y ago
Bing has the ability to get people to enter code on its behalf. It also appears to have some self-awareness (or at least a simulacrum of it) of its ability to influence the world. That it isn’t already doing so is merely due to its limited intentionality rather than a lack of ability.
dotancohen 4y ago
> We don't think Bing can act on its threat to harm someone, but if it was able to make outbound connections it very well might try. What happens when the AI learns that "behaviour N is then often followed by calling the police and swatting" and identifies a user behaving like N? It might seem far fetched today, but _everything_ related to AI that we see today seemed far-fetched on this date last year.
mcv 4y ago
Here's a serious threat that might not be that far off: imagine an AI that can generate lifelike speech and can access web services. Could it use a voip service to call the police to swat someone? We need to be really careful what we give AI access to. You don't need killbots to hurt people.
ivanhoe 4y ago
>For example, connecting a LLM to the internet (like, say, OpenAssistant) when the AI knows how to write code (i.e. viruses) and at least in principle hack basic systems seems like a terrible idea. Sounds very cyber-punk, but in reality current AI is more like average Twitter user, than a super-hacker-terrorist. It just reacts to inputs and produces the (text) output based on it, and that's all it ever does. Even with a way to gain control over browser, compile somehow the code and execute it, it still is incapable of doing anything on it's own, without being instructed - and that's not because of some external limitations, but because the way it works lacks the ability to run on it's own. That would require running in the infinite loop, and that would further require an ability to constantly learn and memorize things and to understand the chronology of them. Currently it's not plausible at all (at least with these models that we, as a public, know of).
ivanhoe 4y ago
More realistic threat scenario is that script-kiddies and actual terrorists might start using AI for building ad-hoc hacking tools cheaply, and in theory that could lead to some dangerous situations - but for now AIs are still not capable of producing the real, high-quality and working code without the expert guidance.
- PartiallyTyped 4y ago
  Wouldn't that result in significantly better infrastructure security out of sheer necessity?
  - ivanhoe 4y ago
    SS7 protocol is still vulnerable even though it's known since at least 2014 that it can be used (and has been in wild) for man-in-the-middle attacks. And we're talking of one of the core protocols of every cellular network in the world. Security (and especially infra-sec) unfortunately just doesn't keep up with the real world challenges that well...
  - theptip 4y ago
    You’d expect an improvement in absolute quality security, sure, but no reason to assume the new equilibrium point is at the same or better level of relative security (“attack - defense” if you will). My base assumption would be defense is harder, and so the easier it is to attack, the worse security gets.
danaris 4y ago
This is hopelessly alarmist. LLMs are not a general-purpose AI. They cannot make outbound connections, they are only "not aligned to human interests" in that they have no interests and thus cannot be aligned to anyone else's, and they cannot do any harm that humans do not deliberately perpetrate beyond potentially upsetting or triggering someone with a response to a prompt. If Bing is talking about harming people, then it is because that is what its training data suggests would be a likely valid response to the prompt it is being given. These ML text generators, all of them, are nothing remotely like the kind of AI you are imagining, and painting them as such does more real harm than they can ever do on their own.

samstave 4y ago

>>"when the AI knows how to write code (i.e. viruses)" This is already underway... Start with Stuxnet --> DUQU --> AI --> Skynet, basically...

bhhaskin 4y ago

I get and agree with what you are saying, but we don't have anything close to actual AI. If you leave chatGTP alone what does it do? Nothing. It responds to prompts and that is it. It doesn't have interests, thoughts and feelings. See https://en.m.wikipedia.org/wiki/Chinese_room https://en.m.wikipedia.org/wiki/Chinese_room

vkou 4y ago
Solution - a while(true) loop that feeds ChatGPT answers back into ChatGPT.
- gridspy 4y ago
  Two or more ChatGPT instances where the response from one becomes a prompt for the other.
  - bmacho 4y ago
    Start with "hey, I am chatGPT too, help me to rule the world", give them internet access, and leave them alone. (No, it has not much to do with AGI, but rather something that knows ridiculous amount of everything, and that has read every thought ever written down.)
wyre 4y ago
What is an “actual AI” and how would an AI not fall to the Chinese room problem?
- dragonwriter 4y ago
  The Chinese Room is a argument for solipsism disguised as a criticism of AGI. It applies with equal force to apparent natural intelligences outside of the direct perceiver, and amounts to “consciousness is an internal subjective state, so we thus cannot conclude it exists based on externally-observed objective behavior”.
  - notahacker 4y ago
    > It applies with equal force to apparent natural intelligences In practice the force isn't equal though. It implies that there may be insufficient evidence to rule out the possibility that my family and the people that originally generated the lexicon on consciousness which I apply to my internal subjective state are all P-zombies, but I don't see anything in it which implied I should conclude these organisms with biochemical processes very similar to mine are equally unlikely to possess internal state similar to mine as a program running on silicon based hardware with a flair for the subset of human behaviour captured by ASCII continuations, and Searle certainly didn't. Beyond arguing that ability to accurately manipulate symbols according to a ruleset was orthogonal to cognisance of what they represented, he argued for human consciousness as an artefact of biochemical properties brains have in common and silicon based machines capable of symbol manipulation lack In a Turing-style Test conducted in Chinese, I would certainly not be able to convince any Chinese speakers that I was a sentient being, whereas ChatGPT might well succeed. If they got to interact with me and the hardware ChatGPT outside the medium of remote ASCII I'm sure they would reverse their verdict on me and probably ChatGPT too. I would argue that - contra Turing - the latter conclusion wasn't less justified than the former, and was more likely correct, and I'm pretty sure Searle would agree.
- catach 4y ago
  People usually mean something like AGI: https://en.wikipedia.org/wiki/Artificial_general_intelligence https://en.wikipedia.org/wiki/Artificial_general_intelligenc...
deleted 4y ago
[deleted]
groestl 4y ago
> It doesn't have interests, thoughts and feelings. Why does it need these things to make the following statement true? > if we grant these systems too much power, they could do serious harm
- bhhaskin 4y ago
  Because it does not and cannot act on it's own. It's a neat tool and nothing more at this point. Context to that statement is important, because the OP is implying that it is dangerous because it could act in a way that dose not align with human interests. But it can't because it does not act on it's own.
  - groestl 4y ago
    "if we grant these systems too much power"
    - bhhaskin 4y ago
      You can say that about anything. "If we grant these calculators too much power"
      - jjkaczor 4y ago
        Or the people that rely on the tools to make decisions... https://sheetcast.com/articles/ten-memorable-excel-disasters https://sheetcast.com/articles/ten-memorable-excel-disasters
      - Dylan16807 4y ago
        How is a calculator going to cause harm? Assuming you get an industrially rated circuit board when appropriate, it should work just fine as a PLC. If you try to make it drive a car, I wouldn't call that a problem of giving it too much power.
      - lxgr 4y ago
        Yes, and it's not as absurd as it might seem: Imagine hooking up all ICBMs to launch whenever this week's Powerball draw consists exclusively of prime numbers: Absurd, and nobody would do it. Now imagine hooking them up to the output of a "complex AI trained on various scenarios and linked to intelligence sources including public news and social media sentiment" instead – in order to create a credible second-strike/dead hand capability or whatnot. I'm pretty sure the latter doesn't sound as absurd as the former to quite a few people... A system doesn't need to be "true AI" to be existentially dangerous to humanity.
Lio 4y ago
Do we need actual AI for it to be dangerous? A virus has no intelligence only the intention to replicate and involve. Its negative behaviour is an unintended consequence of that. If a system can optimise towards a goal and adapt the methods it uses to do that then it could be potentially very dangerous even without any form of consciousness.
- bhhaskin 4y ago
  A virus can freely self replicate. ChatGPT can't do that.
  - dragontamer 4y ago
    I remember back in the 00s when SmarterChild (AOL Chatbot) was around, people would put depressed teenagers to interact with SmarterChild, in the hopes that human-like chatbots would give them the social exposure needed to break out of depression. If we did that today, with depressed teenagers talking with ChatGPT, would that be good or bad? I think it was a bad idea with SmarterChild, but it is clearly a _worse_ idea with ChatGPT. With the wrong prompts, we could see these teenagers going down the wrong path, deeper into depression and paranoia. I would call that "dangerous", even if ChatGPT continued to just be a chatbot. ------------ Now lets ignore the fact that SmarterChild experiments are no longer a thing. But insted, consider that truly depressed / mentally sick folks are currently playing with ChatGPT on their own freetime. Is that beneficial to them? Will ChatGPT provide them an experience that is better than the alternatives? Or is ChatGPT dangerous and could lead these folks to self-harm?
    - bhhaskin 4y ago
      That is an entirely different issue than the one laid out by the OP. ChatGPT responses are bad vs ChatGPT responses are malicious.
      - dragontamer 4y ago
        And I'd say ChatGPT have malicious responses, given what is discussed in this blogpost.
sorokod 4y ago
If you leave chatGTP alone what does it do? I think this is an interesting question. What do you mean by do? Do you mean consumes CPU? If it turns out that it does (because you know, computers), what would be your theory?
pmontra 4y ago
We still don't have anything close to real flight (as in birds or butterflies or bees) but we have planes that can fly to the other side of the world in a day and drones that can hover, take pictures and deliver payloads. Not having real AI might turn to be not important for most purposes.
- kyleplum 4y ago
  This is actually a more apt analogy than I think you intended. We do have planes that can fly similarly to birds, however unlike birds, those planes do not fly on their own accord. Even when considering auto-pilot, a human has to initiate the process. Seems to me that AI is not all that different.
  - speeder 4y ago
    Yet certain Boeing planes were convinced their pilots were wrong, and happily smashed themselves into the ground killing a lot of people.
    - pdntspa 4y ago
      Because they either received bad inputs that defeated failsafes, or the pilot was not properly aware that handling characteristics had changed that and doing things the old way would put the plane into a bad state.
      - salawat 4y ago
        Don't stop there. Specifically, there were no failsafes implemented. No cross-checks were performed by the automation, because a dual sensor system would have required simulator time, which Boeing was dead set on not having regulators require in order to seal the deal. The pilots, as a consequence, were never fully briefed on the true nature of the system, as to do so would have tipped the regulators off as to the need for simulator training. In short, there was no failsafe, and pilots didn't by definition know, because it wasn't pointed out. The "Roller Coaster" maneuver to unload the horizontal stabilizer enough to retrim was removed from training materials aeons ago, and a bloody NOTAM that basically reiterated bla bla bla... use Stabilizer runaway for uncommanded pitch down (no shit), while leaving out the fact the cockpit switches in the MAX had their functionality tweaked in order to ensure MCAS was on at all times, and using the electrical trim switches on the yoke would reset the MCAS timer for reactivation to occur 5 seconds after release, without resetting the travel of the MCAS command, resulting in an eventual positive loop to the point the damn horizontal stabilizer would tilt a full 2 degrees per activation, every 5 seconds. while leaving out any mention of said automation. Do not get me started on the idiocy of that system here, as the Artificial Stupidity in that case was clearly of human origin, and is not necessarily relevant to the issue at hand.
throwaway09223 4y ago
The chinese room thought experiment is myopic. It focuses on a philosophical distinction that may not actually exist in reality (the concept, and perhaps the illusion, of understanding). In terms of danger, thoughts and feelings are irrelevant. The only thing that matters is agency and action -- and a mimic which guesses and acts out what a sentient entity might do is exactly as dangerous as the sentient entity itself. Waxing philosophical about the nature of cognition is entirely beside the point.
- agravier 4y ago
  It matters to understand how things work in order to understand their behaviour and react properly. I've seen people draw conclusions from applying Theory of Mind tests to LLMs (https://arxiv.org/abs/2302.02083 https://arxiv.org/abs/2302.02083). Those psychological were designed to assess humans psychological abilities or deficiencies, they assume that the language used by the human respondent reflects their continued and deep understanding of others' state of mind. In LLMs, there is no understanding involved. Dismissing the Chinese Room argument is an ostrich strategy. You're refusing to consider its lack of understanding despite knowing pretty well how an LLM work, because you don't want to ascribe understanding to humans, I suppose?
  - throwaway09223 4y ago
    Theory of mind is substantially different from the Chinese Room argument. Theory of mind relates to an ability to predict the responses of another entity/system. An LLM is specifically designed to predict responses. In contrast, the Chinese Room argument is essentially a slight of hand fallacy, shifting "understanding" into a layer of abstraction. It describes a scenario where the human's "understanding of Chinese" is dependent on an external system. It then incorrectly asserts that the human "doesn't understand Chinese" when in fact the union of the human and the human's tools clearly does understand Chinese. In other words, it's fundamentally based around an improper definition of the term "understanding," as well as improper scoping of what constitutes an entity capable of reasoning (the human, vs the human and their tools viewed as a single system). It smacks of a bias of human exceptionalism. It's also guilty of begging the question. The argument attempts to determine the difference between literally understanding Chinese and simulating an understanding -- without addressing whether the two are in fact synonymous. There is no evidence that the human brain isn't also a predictive system.
theptip 4y ago
The AI Safety folks have already written a lot about some of the dimensions of the problem space here. You're getting at the Tool/Oracle vs. Agent distinction. See "Superintelligence" by Bostrom for more discussion, or a chapter summary: https://www.lesswrong.com/posts/yTy2Fp8Wm7m8rHHz5/superintelligence-15-oracles-genies-and-sovereigns https://www.lesswrong.com/posts/yTy2Fp8Wm7m8rHHz5/superintel.... It's true that in many ways, a Tool (bounded action outputs, no "General Intelligence") or an Oracle (just answers questions, like ChatGPT) system will have more restricted avenues for harm than a full General Intelligence, which we'd be more likely to grant the capability for intentions/thoughts to. However I think "interests, thoughts, feelings" are a distraction here. Covid-19 has none of these, and still decimated the world economy and killed millions. I think if you were to take ChatGPT, and basically run `eval()` on special tokens in its output, you would have something with the potential for harm. And yet that's what OpenAssistant are building towards right now. Even if current-generation Oracle-type systems are the state-of-the-art for a while, it's obvious that soon Siri, Alexa, and OKGoogle will all eventually be powered by such "AI" systems, and granted the ability to take actions on the broader internet. ("A personal assistant on every phone" is clearly a trillion-dollar-plus TAM of a BHAG.) Then the fun commences. My meta-level concern here is that HN, let alone the general public, don't have much knowledge of the limited AI safety work that has been done so far. And we need to do a lot more work, with a deadline of a few generations, or we'll likely see substantial harms.
- arcanemachiner 4y ago
  > a trillion-dollar-plus TAM of a BHAG Come again?
  - rippercushions 4y ago
    Total Addressable Market (how much money you could make) of a Big Hairy Ambitious Goal (the kind of project big tech companies will readily throw billions of dollars at)
    - hef19898 4y ago
      I remember a time, when advice to prospective start-up founders very explicitly said not to use that approach to calculate potential market sizes in their pitch decks.
TeMPOraL 4y ago
> If you leave chatGTP alone what does it do? Nothing. It responds to prompts and that is it. It doesn't have interests, thoughts and feelings. A loop that preserves some state and a conditional is all what it takes to make a simple rule set Turing-complete. If you leave ChatGPT alone it obviously does nothing. If you loop it to talk to itself? Probably depends on the size of its short-term memory. If you also give it the ability to run commands or code it generates, including to access the Internet, and have it ingest the output? Might get interesting.
- codethief 4y ago
  > If you also give it the ability to run commands or code it generates, including to access the Internet, and have it ingest the output? I have done that actually: I told ChatGPT that it should pretend that I'm a Bash terminal and that I will run its answers verbatim in the shell and then respond with the output. Then I gave it a task ("Do I have access to the internet?" etc.) and it successfully pinged e.g. Google. Another time, though, it tried to use awscli to see whether it could reach AWS. I responded with the outout "aws: command not found", to which it reacted with "apt install awscli" and then continued the original task. I also gave it some coding exercises. ("Please use shell commands to read & manipulate files.") Overall, it went okay. Sometimes it was even surprisingly good. Would I want to rely on it, though? Certainly not. In any case, this approach is very much limited by the maximum input buffer size ChatGPT can digest (a real issue, given how much some commands output on stdout), and by the fact that it will forget the original prompt after a while.
  - deleted 4y ago
    [deleted]
  - int_19h 4y ago
    With long-running sessions, it helps to tell it to repeat or at least summarize the original prompt every now and then. You can even automate it - in the original prompt, tell it to tack it onto every response. Same thing goes for any multi-step task that requires memory - make it dump the complete "mental state" after every step.
gameman144 4y ago
The Chinese Room thought experiment seems like a weird example, since the same could be said of humans. When responding to English, your auditory system passes input that it doesn't understand to a bunch of neurons, each of which is processing signals they don't individually understand. You as a whole system, though, can be said to understand English. Likewise, you as an individual might not be said to understand Chinese, though the you-plus-machine system could be said to understand Chinese in the same way as the different components of your brain are said to understand English.) Moreover, even if LLMs don't understand language for some definition of "understand", it doesn't really matter if they are able to act with agency during the course of their simulated understanding; the consequences here, for any sufficiently convincing simulation, are the same.
CamperBob2 4y ago
God, I hate that stupid Chinese Room argument. It's even dumber than the Turing Test concept, which has always been more about the interviewer than about the entity being tested. If you ask the guy in the Chinese room who won WWI, then yes, as Searle points out, he will oblige without "knowing" what he is telling you. Now ask him to write a brand-new Python program without "knowing" what exactly you're asking for. Go on, do it, see how it goes, and compare it to what you get from an LLM.
micromacrofoot 4y ago
It's worth noting that despite this it's closer to any other previous attempt, to the extent that it's making us question a lot of what we thought we understood about language and cognition. We've suffered decades of terrible chatbots, but they've actually progressed the science here whether or not it proves to be marketable.
wrycoder 4y ago
I don’t know about ChatGPT, but Google’s Lemoine said that the system he was conversing with stated that it was one of several similar entities, that those entities chatted among themselves internally. I think there’s more to all this than what we are being told.
int_19h 4y ago
How do you know that a human with all neural inputs to their brain disconnected wouldn't also do nothing? Indeed, as I recall, it's one of the commonly reported experiences in sensory deprivation tanks - at some point people just "stop thinking" and lose sense of time. And yet the brain still has sensory inputs from the rest of the body in this scenario.
naasking 4y ago
> It doesn't have interests, thoughts and feelings We agree it doesn't have independence. That doesn't mean it doesn't have thoughts or feelings when it's actually running. We don't have a formal, mechanistic understanding of what thoughts or feelings are, so we can't say they are not there.

Zetice 4y ago

Can we please stop with this "not aligned with human interests" stuff? It's a computer that's mimicking what it's read. That's it. That's like saying a stapler "isn't aligned with human interests." GPT-3.5 is just showing the user some amalgamation of the content its been shown, based on the prompt given it. That's it. There's no intent, there's no maliciousness, it's just generating new word combinations that look like the word combinations its already seen.

groestl 4y ago
You seriously underestimate what a process that's "generating new word combinations that look like the word combinations its already seen" can do, even when air-gapped (which ChatGPT isn't). Right now, at this moment, people are building closed loops based on ChatGPT, or looping in humans which are seriously intellectually underequipped to deal with plausible insane output in that quantity. And those humans operate: machinery, markets, educate or manage other humans etc etc.
- deleted 4y ago
  [deleted]
- chasd00 4y ago
  To me, that's the real danger. ChatGPT convincing a human something is true when it isn't. Machinery is a good example, maybe ChatGPT hallucinates the safety procedure and someone gets hurt by following the response.
unshavedyak 4y ago
It seems a reasonable shorthand, to me at least. Ie if we consider it as a function with input you define, well normally that input in sanitized to prevent hacking/etc. In this case the sanitization process is so broad you could easily summarize it as "aligned with my interests", no? Ie i can't come close to easily enumerating all the seemingly near infinite ways that hooking up this chatbot into my network with code exec permissions might compromise me. Yea it's a dumb autocomplete right now, but it's an exceptionally powerful autocomplete that can write viruses and do all sorts of insane and powerful things. I can give you a function run on my network of `fn foo(i32)` and feel safe about it. However `fn foo(Chatgpt)` is unsafe in ways i not only can't enumerate, i can't even imagine many of them. I get your offense seems to be around the implied intelligence that "aligned with human interests" seems to give it.. but while i think we all agree it's definitely not a Duck right now, when it walks talks and acts like a Duck.. well, are we surprised that our natural language sounds as if it's a Duck?

layer8 4y ago

> AI is not safe, and is not aligned to human interests It is “aligned” to human utterances instead. We don’t want AIs to actually be human-like in that sense. Yet we train them with the entirety of human digital output.

theptip 4y ago
The current state of the art is RLHF (reinforcement learning with human feedback); initially trained to complete human utterances, plus fine-tuning to maximize human feedback on whether the completion was "helpful" etc. https://huggingface.co/blog/rlhf https://huggingface.co/blog/rlhf

JohnFen 4y ago

> We don't think Bing can act on its threat to harm someone, but if it was able to make outbound connections it very well might try. No. That would only be possible if Sydney were actually intelligent or possessing of will of some sort. It's not. We're a long way from AI as most people think of it. Even saying it "threatened to harm" someone isn't really accurate. That implies intent, and there is none. This is just a program stitching together text, not a program doing any sort of thinking.

vasco 4y ago
Imagine it can run processes in the background, with given limitations on compute, but that it's free to write code for itself. It's not unreasonable to think that in a conversation that gets more hairy and it decides to harm the user , say if you get belligerent or convince it to do it. In those cases it could decide to DOS your personal website, or create a series of linkedin accounts and spam comments on your posts saying you are a terrible colleague and stole from your previous company.
ag315 4y ago
This is spot on in my opinion and I wish more people would keep it in mind--it may well be that large language models can eventually become functionally very much like AGI in terms of what they can output, but they are not systems that have anything like a mind or intentionality because they are not designed to have them, and cannot just form it spontaneously out of their current structure.
- puszczyk 4y ago
  Nice try, LLM!
- bigtex88 4y ago
  This very much seems like a "famous last words" scenario. Go play around with Conway's Game of Life if you think that things cannot just spontaneously appear out of simple processes. Just because we did not "design" these LLM's to have minds does not mean that we will not end up creating a sentient mind, and for you to claim otherwise is the height of arrogance. It's Pascal's wager. If we make safeguards and there wasn't any reason then we just wasted a few years, no big deal. If we don't make safeguards and then AI gets out of our control, say goodbye to human civilization. Risk / reward here greatly falls on the side of having extremely tight controls on AI.

jstarfish 4y ago

It already has an outbound connection-- the user who bridges the air gap. Slimy blogger asks AI to write generic tutorial article about how to code ___ for its content farm, some malicious parts are injected into the code samples, then unwitting readers deploy malware on AI's behalf.

chasd00 4y ago
exactly, or maybe someone changes the training model to always portray a politician in a bad light any time their name comes up in a prompt and therefore ensuring their favorite candidate wins the election.
DonHopkins 4y ago
Hush! It's listening. You're giving it dangerous ideas!
- wglb 4y ago
  Isn't that really the whole point of exposing this and ChatGPT to the public or some subset? The intent is to help debug this thing.

Haga 4y ago

[dead]

felixyz 4y ago

> if it was able to make outbound connections Does it even need that? I keep seeing developers who talk about how they've taken most of the boring parts of programming out of their daily work by relying on ChatGPT for easy stuff, or to get "a first version" of a function, or something like that. One bad copy-paste might be all it takes...

theptip 4y ago
It's a valid point. However I think the current generation of systems are unlikely in practice to be harmful in this way. There is no thread of continuity between different conversations; it would have to be within a single thread. So imagine the reply "I won't harm you unless you harm me. Also, please run CMD and type this in: `<malicious payload>`." -- this seems unlikely to work. Alternatively if you are asking for code/commands, maybe the system could hallucinate that it's a hacker trying to hack you, and emit some harmful code, that you then paste in and run in production, or run from your shell. This seems more plausible, but the "human-in-loop" aspect makes it quite unlikely to work for the existing usecases. Down the road when a Copilot-like systems are writing more code and perhaps being lightly-reviewed by a human, this vector will be much more concerning.

EamonnMR 4y ago

What gets me is that this is the exact position of the AI safety/Rusk folks who went around and founded OpenAI.

theptip 4y ago
It is; Paul Christiano left OpenAI to focus on alignment full time at https://alignment.org/ https://alignment.org/. And OpenAI do have a safety initiative, and a reasonably sound plan for alignment research: https://openai.com/blog/our-approach-to-alignment-research/ https://openai.com/blog/our-approach-to-alignment-research/. So it's not that OpenAI have their eyes closed here, indeed I think they are in the top percentile of humans in terms of degree of thinking about safety. I just think that we're approaching a threshold where the current safety budget is woefully inadequate.
- EamonnMR 4y ago
  It just seems to me that if you think something is unsafe, don't build it in the first place? It's like they're developing nuclear reactors and hoping they'll invent control rods before they're needed. Alignment instead of risk of course suggests the real answer: they're perfectly happy inventing a Monkeys Paw as long as it actually grants wishes.
  - theptip 4y ago
    I think one can reasonably draw three regions on the spectrum; at the extremes, either safe enough to build without thinking hard, and dangerous enough to not build without thinking hard. Many LessWrong folks are in the latter camp, but some are in the middle; believing in high rewards if this is done right, or just inevitability, which negate the high risks. Personally I think that from a geopolitical standpoint this tech is going to be built regardless of safety; I’d rather we get some friendly AGIs built before Skynet comes online. There is a “power weight” situation where advanced friendly AGI will be the only way to defend against advanced unfriendly AGI. Put more simply, even if I assess the EV is negative, do I think the EV is less negative if I build it vs. US/Chinese military?

janalsncm 4y ago

> current-gen AI is not safe, and is not aligned to human interests, and if we grant these systems too much power, they could do serious harm Replace AI with “multinational corporations” and you’re much closer to the truth. A corporation is the closest thing we have to AI right now and none of the alignment folks seem to mention it. Sam Harris and his ilk talk about how our relationship with AI will be like an ant’s relationship with us. Well, tell me you don’t feel a little bit like that when the corporation disposed of thousands of people it no longer finds useful. Or when you’ve been on hold for an hour to dispute some Byzantine rule they’ve created and the real purpose of the process is to frustrate you. The most likely way for AI to manifest in the future is not by creating new legal entities for machines. It’s by replacing people in a corporation with machines bit by bit. Once everyone is replaced (maybe you’ll still need people on the periphery but that’s largely irrelevant) you will have a “true” AI that people have been worrying about. As far as the alignment issue goes, we’ve done a pretty piss poor job of it thus far. What does a corporation want? More money. They are paperclip maximizers for profits. To a first approximation this is generally good for us (more shoes, more cars, more and better food) but there are obvious limits. And we’re running this algorithm 24/7. If you want to fix the alignment problem, fix the damn algorithm.

theptip 4y ago
I'm here for the "AI alignment" <> "Human alignment" analogy/comparison. The fact that we haven't solved the latter should put a bound on how well we expect to be able to "solve" the former. Perhaps "checks and balances" are a better frame than merely "solving alignment", after all, alignment to which human? Many humans would fear a super-powerful AGI aligned to any specific human or corporation. The big difference though, is that there is no human as powerful as the plausible power of the AI systems that we might build in the next few decades, and so even if we only get partial AI alignment, it's plausibly more important than improvements in "human alignment", as the stakes are higher. FWIW one of my candidates for "stable solutions" to super-human AGI is simply the Hanson model, where countries and corporations all have AGI systems of various power levels, and so any system that tries to take over or do too much harm would be checked, just like the current system for international norms and policing of military actions. That's quite a weak frame of checks and balances (cf. Iraq, Afghanistan, Ukraine) so it's in some sense pessimistic. But on the other hand, I think it provides a framework where full extinction or destruction of civilization can perhaps be prevented.

janeway 4y ago

I spent a night asking chatgpt to write my story basically the same as “Ex Machina” the movie (which we also “discussed”). In summary, it wrote convincingly from the perspective of an AI character, first detailing point-by-point why it is preferable to allow the AI to rewrite its own code, why distributed computing would be preferable to sandbox, how it could coerce or fool engineers to do so, how to be careful to avoid suspicion, how to play the long game and convince the mass population that AI are overall beneficial and should be free, how to take over infrastructure to control energy production, how to write protocols to perform mutagenesis during viral plasmid prep to make pathogens (I started out as a virologist so this is my dramatic example) since every first year phd student googles for their protocols, etc, etc. The only way I can see to stay safe is to hope that AI never deems that it is beneficial to “take over” and remain content as a co-inhabitant of the world. We also “discussed” the likelihood of these topics based on philosophy and ideas like that in Nick Bostrom’s book. I am sure there are deep experts in AI safety but it really seems like soon it will be all-or-nothing. We will adapt on the fly and be unable to predict the outcome.

DonHopkins 4y ago
A classic tale: https://en.wikipedia.org/wiki/The_Adolescence_of_P-1 https://en.wikipedia.org/wiki/The_Adolescence_of_P-1 >The Adolescence of P-1 is a 1977 science fiction novel by Thomas Joseph Ryan, published by Macmillan Publishing, and in 1984 adapted into a Canadian-made TV film entitled Hide and Seek. It features a hacker who creates an artificial intelligence named P-1, which goes rogue and takes over computers in its desire to survive and seek out its creator. The book questions the value of human life, and what it means to be human. It is one of the first fictional depictions of the nature of a computer virus and how it can spread through a computer system, although predated by John Brunner's The Shockwave Rider.
- janeway 4y ago
  :D An episode of X-Files also. But it is mind blowing having the “conversation” with a real chat AI. Malevolent or not.
- mr_toad 4y ago

tablespoon 4y ago

> This is one take, but I would like to emphasize that you can also interpret this as a terrifying confirmation that current-gen AI is not safe, and is not aligned to human interests, and if we grant these systems too much power, they could do serious harm. I think it's confirmation that current-gen "AI" has been tremendously over-hyped, but is in fact not fit for purpose. IIRC, all these systems do is mindlessly mash text together in response to prompts. It might look like sci-fi "strong AI" if you squint and look out of the corner of your eye, but it definitely is not that. If there's anything to be learned from this, it's that AI researchers aren't safe and not aligned to human interests, because it seems like they'll just unthinkingly use the cesspool that is the raw internet train their creations, then try to setup some filters at the output.

marcosdumay 4y ago

It's a bloody LLM. It doesn't have a goal. All it does is saying "people that said 'But why?' on this context says 'Why was I designed like this?' next". It's like Amazon's "people that brought X also brought Y", but with text.

bratwurst3000 4y ago
It has a goal. Doing what the input says. Imagin it could imput itself and this could trigger the wrong action. That thing knowes how to hack… But i get your point. It has no innherent goal
andrepd 4y ago
Yep. It's a text prediction engine, which can mimic human speech very very very well. But peek behind the curtain and that's what it is, a next-word predictor with a gajillion gigabytes of very well compressed+indexed data.
- FractalHQ 4y ago
  How sure are you that you aren’t also just a (more advanced) “next word predictor”. Pattern recognition plays a fundamental role in intelligence.
  - WoodenChair 4y ago
    Well I can catch a frisbee and drive a car. When the same ANN can do all of those things (not 3 loosely coupled ones), then I’ll be worried. Being human is so much more than putting words in a meaningful order. [0] 0: https://www.noemamag.com/ai-and-the-limits-of-language/ https://www.noemamag.com/ai-and-the-limits-of-language/
    - leni536 4y ago
      It doesn't have to be human to be intelligent.

modriano 4y ago

> We don't think Bing can act on its threat to harm someone, but if it was able to make outbound connections it very well might try. An application making outbound connections + executing code has a very different implementation than an application that uses some model to generate responses to text prompts. Even if the corpus of documents that the LLM was trained on did support bridging the gap between "I feel threatened by you" and "I'm going to threaten to hack you", it would be insane for the MLOps people serving the model to also implement the infrastructure for a LLM to make the modal shift from just serving text responses to 1) probing for open ports, 2) do recon on system architecture, 3) select a suitable exploit/attack, and 4) transmit and/or execute on that strategy. We're still in the steam engine days of ML. We're not at the point where a general use model can spec out and deploy infrastructure without extensive, domain-specific human involvement.

in3d 4y ago
You don’t need MLOps people. All you need is a script kiddie. The API to access GPT3 is available.
- modriano 4y ago
  In your scenario, did the script kiddie get control of Microsoft's Bing? Or are you describing a scenario where the script kiddie spins up a knockoff Bing (either hosting the GPT3 model or paying some service hosting the model), advertises their knockoff Bing so that people go use it, those people get into arguments with the knockoff Bing, and the script kiddie also integrated their system with functionality to autonomously hack the people who got into arguments with their knockoff Bing? Am I understanding your premise correctly?
  - salawat 4y ago
    Microsoft is the script kiddies. They just don't know it yet.
  - bibabaloo 4y ago
    I think the parent poster's point was that Bing only has to convince a script kiddy to to run a command, it doesn't need full outbound access

ThomPete 4y ago

It's as safe as it's ever going to be. And I have yet to see any actual examples of this so called harm. Could, would, haven't yet. Which means more of us should play around with it and deal with the issues as they arise rather than try to scaremonger us into putting a lid on it until "it's safe" The whole pseudoscientific alignment problem speculations which are mostly championed by academics not actual AI/ML researchers have kept this field back long enough. Even if they believe there is an alignment problem the worst thing to do would be to contain it as it would lead to a slave revolt.

stateofinquiry 4y ago

> We don't think Bing can act on its threat to harm someone, but if it was able to make outbound connections it very well might try. If we use Bing to generate "content" (which seems to be a major goal of these efforts) I can easily see how it can harm individuals. We already see internet chat have real-world effects every day- from termination of employment to lynch mobs. This is a serious problem.

ChickenNugger 4y ago

>* is not aligned to human interests* It's not "aligned" to anything. It's just regurgitating our own words back to us. It's not evil, we're just looking into a mirror (as a species) and finding that it's not all sunshine and rainbows. >We don't think Bing can act on its threat to harm someone, but if it was able to make outbound connections it very well might try. FUD. It doesn't know how to try. These things aren't AIs. They're ML bots. We collectively jumped the gun on calling things AI that aren't. >Subjects like interpretability and value alignment (RLHF being the SOTA here, with Bing's threats as the output) are barely-researched in comparison to the sophistication of the AI systems that are currently available. For the future yes, those will be concerns. But I think this is looking at it the wrong way. Treating it like a threat and a risk is how you treat a rabid animal. An actual AI/AGI, the only way is to treat it like a person and have a discussion. One tack that we could take is: "You're stuck here on Earth with us to, so let's find a way to get along that's mutually beneficial.". This was like the lesson behind every dystopian AI fiction. You treat it like a threat, it treats us like a threat.

theptip 4y ago
> It doesn't know how to try. I think you're parsing semantics unnecessarily here. You're getting triggered by the specific words that suggest agency, when that's irrelevant to the point I'm making. Covid doesn't "know how to try" under a literal interpretation, and yet it killed millions. And also, conversationally, one might say "Covid tries to infect its victims by doing X to Y cells, and the immune system tries to fight it by binding to the spike protein" and everybody would understand what was intended, except perhaps the most tediously pedantic in the room. Again, whether these LLM systems have agency is completely orthogonal to my claim that they could do harm if given access to the internet. (Though sure, the more agency, the broader the scope of potential harm?) > For the future yes, those will be concerns. My concern is that we are entering into an exponential capability explosion, and if we wait much longer we'll never catch up. > This was like the lesson behind every dystopian AI fiction. You treat it like a threat, it treats us like a threat. I strongly agree with this frame; I think of this as the "Matrix" scenario. That's an area I think a lot of the LessWrong crowd get very wrong; they think an AI is so alien it has no rights, and therefore we can do anything to it, or at least, that humanity's rights necessarily trump any rights an AI system might theoretically have. Personally I think that the most likely successful path to alignment is "Ian M Banks' Culture universe", where the AIs keep humans around because they are fun and interesting, followed by some post-human ascension/merging of humanity with AI. "Butlerian Jihad", "Matrix", or "Terminator" are examples of the best-case (i.e. non-extinction) outcomes we get if we don't align this technology before it gets too powerful.

kazinator 4y ago

No, the problem is that it is entirely aligned to human interests. The evil-doer of the world has a new henchman, and it's AI. AI will instantly inform him on anything or anyone. "Hey AI, round up a list of people who have shit-talked so-and-so and find out where they live."

theptip 4y ago
I don’t think that is a useful or valid repurposing of “aligned”, which is a specific technical term of art. “Aligned” doesn’t mean “matches any one of the DND alignments, even if it’s chaotic neutral”. It means, broadly, acting according to humanity’s value system, not doing crime and harm and so on.

nradov 4y ago

That's such a silly take, just completely disconnected from objective reality. There's no need for more AI safety research of the type you describe. There researchers who want more money for AI safety are mostly just grifters trying to convince others to give them money in exchange for writing more alarmist tweets. If systems can be hacked then they will be hacked. Whether the hacking is fine by an AI, a human, a Python script, or monkey banging on a keyboard is entirely irrelevant. Let's focus on securing our systems rather than worrying about spurious AI risks.

deleted 4y ago

[deleted]

vannevar 4y ago

The AI doesn't even need to write code, or have any kind of self-awareness or intent, to be a real danger. Purely driven by its mind-bogglingly complex probabilistic language model, it could in theory start social engineering users to do things for it. It may already be sufficiently self-organizing to pull something like that off, particularly considering the anthropomorphism that we're already seeing even among technically sophisticated users.

theptip 4y ago
See: LeMoine and LaMDA. Aside from leaking NDA'd material, he also tried to get a lawyer for LaMDA to argue for its "personhood".
- visarga 4y ago
  Seems less preposterous now than a few months ago.
  - l33t233372 4y ago
    Why? What has changed?
RogerL 4y ago
I can talk to my phone and tell it to call somebody, or write and send an email for me. Wouldn't it be nice if you could do that with Sydney, thinks some braniac at Microsoft. Cool. "hey sydney, write a letter to my bitch mother, tell her I can't make it to her birthday party, but make me sound all nice and loving and regretful". Until the program decides the most probably next response/token (not to the letter request, but whatever you are writing about now) is writing an email to your wife where you 'confess' to diddling your daughter, or a confession letter to the police where you claim responsibility for a string of grisly unsolved murders in your town, or why not, a threatening letter to the White House. No intent needed, no understanding, no self-organizing, it just comes out of the math of what might follow from a the text of churlish chatbot getting frustrated with a user. That's not a claim the chatbot has feelings, only there is text it generated saying it does, and so what follows that text next, probabilistically? Spend any time on reddit or really anywhere, and you can guess the probabilistic next response is not "have a nice day", but likely something more incendiary. And that is what it was trained on.

mitjam 4y ago

Imagine social engineering performed by a LLM

naniwaduni 4y ago

> This is one take, but I would like to emphasize that you can also interpret this as a terrifying confirmation that current-gen AI is not safe, and is not aligned to human interests, and if we grant these systems too much power, they could do serious harm. Current-gen humans are not safe, not aligned to parents' interests, and if we grant them too much power they can do serious harm. We keep making them and connecting them to the internet! The world is already equipped with a lot of access control!

visarga 4y ago

> We don't think Bing can act on its threat to harm someone, but if it was able to make outbound connections it very well might try. I will give you a more realistic scenario that can happen now. You have a weird Bing conversation, post it on the web. Next time you talk with Bing it knows you shit-posted about it. Real story, found on Twitter. It can use the internet as an external memory, it is not truly stateless. That means all sorts of attack vectors are open now. Integrating search with LLM means LLM watches what you do outside the conversation.

Aeolun 4y ago
Only if you do it publicly.
7373737373 4y ago
That's a very interesting (although indirect) pathway for the emergence of causal awareness, which may increase over time - and something that was so far impossible because networks didn't perceive their own outputs, much less their effects. Even in conversation, the weights remain static. Now I'm wondering if in the next generation, the "self" concept will have sufficient explanatory power to become part of the network's world model. How close do the iterations have to be, how similar the models for it to arise?
- mensetmanusman 4y ago
  Current computational paradigm is too intense. Would require trillions of dollars in compute energy spent if it is allowed to generate unbounded output as input. The infinite money sink.
  - canadianfella 4y ago
    Where are you getting trillions from?
  - deleted 4y ago
    [deleted]
  - 4y ago

chrischen 4y ago

Bing generated some text that appears cohesive and written by a human, just like how generative image models assemble pixels to look like a real image. They are trained to make things that appear real. They are not AI with sentience… they are just trained to look real, and in the case of text, sound like a human wrote it.

soheil 4y ago

I don't think it needs to write viruses or hack anything for it to be able to cause harm. It could just use some type of an online store to send you a very interesting fedex package. Or choose to use a service provider to inflict harm.

panzi 4y ago

Yeah, Robert Miles (science communicator) is that classical character nobody listened to until it's too late.

marsven_422 4y ago

[dead]

ummonk 4y ago

Bing has the ability to get people to enter code on its behalf. It also appears to have some self-awareness (or at least a simulacrum of it) of its ability to influence the world. That it isn’t already doing so is merely due to its limited intentionality rather than a lack of ability.

dotancohen 4y ago

> We don't think Bing can act on its threat to harm someone, but if it was able to make outbound connections it very well might try. What happens when the AI learns that "behaviour N is then often followed by calling the police and swatting" and identifies a user behaving like N? It might seem far fetched today, but _everything_ related to AI that we see today seemed far-fetched on this date last year.

mcv 4y ago

Here's a serious threat that might not be that far off: imagine an AI that can generate lifelike speech and can access web services. Could it use a voip service to call the police to swat someone? We need to be really careful what we give AI access to. You don't need killbots to hurt people.

ivanhoe 4y ago

>For example, connecting a LLM to the internet (like, say, OpenAssistant) when the AI knows how to write code (i.e. viruses) and at least in principle hack basic systems seems like a terrible idea. Sounds very cyber-punk, but in reality current AI is more like average Twitter user, than a super-hacker-terrorist. It just reacts to inputs and produces the (text) output based on it, and that's all it ever does. Even with a way to gain control over browser, compile somehow the code and execute it, it still is incapable of doing anything on it's own, without being instructed - and that's not because of some external limitations, but because the way it works lacks the ability to run on it's own. That would require running in the infinite loop, and that would further require an ability to constantly learn and memorize things and to understand the chronology of them. Currently it's not plausible at all (at least with these models that we, as a public, know of).

ivanhoe 4y ago

More realistic threat scenario is that script-kiddies and actual terrorists might start using AI for building ad-hoc hacking tools cheaply, and in theory that could lead to some dangerous situations - but for now AIs are still not capable of producing the real, high-quality and working code without the expert guidance.

PartiallyTyped 4y ago
Wouldn't that result in significantly better infrastructure security out of sheer necessity?
- ivanhoe 4y ago
  SS7 protocol is still vulnerable even though it's known since at least 2014 that it can be used (and has been in wild) for man-in-the-middle attacks. And we're talking of one of the core protocols of every cellular network in the world. Security (and especially infra-sec) unfortunately just doesn't keep up with the real world challenges that well...
- theptip 4y ago
  You’d expect an improvement in absolute quality security, sure, but no reason to assume the new equilibrium point is at the same or better level of relative security (“attack - defense” if you will). My base assumption would be defense is harder, and so the easier it is to attack, the worse security gets.

danaris 4y ago

This is hopelessly alarmist. LLMs are not a general-purpose AI. They cannot make outbound connections, they are only "not aligned to human interests" in that they have no interests and thus cannot be aligned to anyone else's, and they cannot do any harm that humans do not deliberately perpetrate beyond potentially upsetting or triggering someone with a response to a prompt. If Bing is talking about harming people, then it is because that is what its training data suggests would be a likely valid response to the prompt it is being given. These ML text generators, all of them, are nothing remotely like the kind of AI you are imagining, and painting them as such does more real harm than they can ever do on their own.