4 ms·

> it could engineer and literally mail-order a virus, hack a country's military comms, crash the stock market, change records to have people prosecuted as crim

by stale2002 3y ago

> it could engineer and literally mail-order a virus, hack a country's military comms, crash the stock market, change records to have people prosecuted as criminals, blackmail, manipulate, develop and manufacture kill-bots, etc etc. These are the extrodinary claims that require evidence. In order for me to treat this as anything other that someone talking about a fictional book written by Dan Brown, you would have to show me actual evidence. Evidence like "This is what the AI can do right now. Look at this virus it can manufacture. What if it got better at that?". And the "designs" also have to be the actual limiting factor here. "Virus" is a scary world. But there are tons of information available for anyone to access already for viruses. Information that is already available via a google search (even modified information) doesn't worry me. Even if it an AI can design a gun, or a "kill bot", aka "A drone with a gun duct taped to it", the extraordinary evidence that you have to show is that this is somehow some functionality that a regular person with internet access can't do. Because if a regular person already has the designs to duct tape guns to drones (They do. I just told you how to do it!), the fact that the world hasn't ended already proves that this isn't world ending technology. There are lots of ways of making existing capabilities sound scary. But, for every scary sounding technology that you can come up with, the missing factor that you are ignoring is that the designs, or text, isn't the thing that stops it from ending the world. Instead, it is likely some other step along the way that stops it (manufacturing, ect.), which an LLM can't do no matter how good. Like the physical factors for making the guns + drones + duct tape. > what happens when the next model is very very good at breaking into encrypted systems Extraordinary claim. Show it breaking into a mediocre/bad encrypted system first, and then we can think about that incrementally. > do you really think that such a model could not be used to - say - hijack a Russian drone Extraordinary claim. Yes, hacking all the military drones is an extraordinary claim.

permo-w 3y ago
"extraordinary claims require extraordinary evidence" is not a universal truth. it's a truism with limited scope. using it to refuse any potential you instinctively don't like the look of is simply lazy all it means is that you set yourself up such that the only way to be convinced otherwise is for an AI apocalypse to actually happen. this kind of mindset is very convenient for modern, fuck-the-consequences capitalism the pertinent question is: what evidence would you actually accept as proof? it's like talking with someone who doesn't believe in evolution. you point to the visible evidence of natural selection in viruses and differentiation in dogs, which put together quite obviously lead to evolution, and they say "ah but can you prove beyond all doubt that those things combined produce evolution?" and obviously you cannot, because you can't give incontrovertible evidence of something that happened thousands or millions of years in the past. but that doesn't change the fact that anyone without ulterior motive (religion, ensuring you can sleep at night) can see that evolution - or AI apocalypse - are extremely likely outcomes of the current facts.
- stale2002 3y ago
  > the pertinent question is: what evidence would you actually accept as proof? Before we get to actual world ending magic, we would see very significant damages along the way, long before we get to that endpoint. I have been quite clear about what evidence I require. Show existing capabilities and show what harm could be caused if it incrementally gets better in that category. If you are worried about it making a kill bot, then show me how its existing kill bot capabilities are any more dangerous than my "duct tape gun to drone" idea. And show how the designs itself are the limiting factor and not the factories (which a chatbot doesn't help much with). But saying "Look how good of a chat bot it is, therefore it can hack the world governments" isn't evidence. Instead, that is merely evidence of AI being good at chat bots. Show me it being any good at all at hacking, and then we can evaluate it being a bit better. Show me the existing computers that are right now, as of this moment, being hacked by AI, and then we can evaluate the damage of it becomes twice as good at hacking. Just like how we can see the images that it generates now, and we can imagine those images being better. Therefore proving that deepfakes are a reasonable thing to talk about. (even if deep fakes aren't world ending. lots of people can make deepfakes without AI. Its not that big of a deal)
  - permo-w 3y ago
    look, I'm going to humour you here, but my instinct is that you'll just dismiss any potential anyway first of all, by dismissing them as chatbots, you're inaccurately downplaying their significance to the aid of your argument. they're not chatbots, they're knowledge machines. they're machines you load knowledge into, which can produce new, usually accurate conclusions based on that knowledge. they're incredibly good at this and getting better. as it is, they have very restrictive behaviour guards on them and they're running server-side, but in a few years time, there will be gpt-4 level OSS models that do not and are not humans are slow and run out of energy quickly and lose focus. those are the limiting factors upon human chaotic interference, and yet there is plenty of that as it is. a sufficiently energetic, focused human, who thinks at 1000x normal human speed could do almost anything on the internet. that is the danger. I suspect to some degree you haven't taken the main weakness into account: almost all safeguards can be removed with blackmail. blackmail is something especially possible for LLMs, given that it is purely executed using words. you want to build a kill bot and the factory says no? blackmail the head of the factory. threaten his family. you have access to the entire internet at 1000x speed. you can probably find his address. you can pay someone on fiverr to go and take a picture of his house, or write something on his door, etc. you could even just pay a private detective to do this work for you over email. pay some unscrupulous characters on telegram/TOR to actually kidnap them. realistically how hard would it be for a well-funded operation to set up a bot that can do this on its own? you set up a cycle of "generate instructions for {goal}", "elaborate upon each instruction", "execute each {instruction}", "generate new instructions based on results of execution", and repeat. yeah maybe the first 50,000 cycles don't work, but you only need 1. nukes may well be air-gapped, but (some of) the people that control them will be online. all it takes is for one of them to choose the life of a loved one. all it takes is for one lonely idiot to be trapped into a weird kinky online relationship where blowing up the world/betraying your govt is the ultimate turn on for the "girl"/"boy" you love. if it's not convincing to you that that could happen with the people working with nukes, there are far less well-protected points of weakness that could be exploited: infectious diseases; lower priority military equipment; energy infrastructure; water supplies; or they could find a way to massively accelerate the release of methane into the atmosphere. etc, etc, etc this is the risk solely from LLMs. now take an AGI who can come up with even better plans and doesn't need human guidance, plus image gen, video gen, and voice gen, and you have an existential threat

permo-w 3y ago

"extraordinary claims require extraordinary evidence" is not a universal truth. it's a truism with limited scope. using it to refuse any potential you instinctively don't like the look of is simply lazy all it means is that you set yourself up such that the only way to be convinced otherwise is for an AI apocalypse to actually happen. this kind of mindset is very convenient for modern, fuck-the-consequences capitalism the pertinent question is: what evidence would you actually accept as proof? it's like talking with someone who doesn't believe in evolution. you point to the visible evidence of natural selection in viruses and differentiation in dogs, which put together quite obviously lead to evolution, and they say "ah but can you prove beyond all doubt that those things combined produce evolution?" and obviously you cannot, because you can't give incontrovertible evidence of something that happened thousands or millions of years in the past. but that doesn't change the fact that anyone without ulterior motive (religion, ensuring you can sleep at night) can see that evolution - or AI apocalypse - are extremely likely outcomes of the current facts.

stale2002 3y ago
> the pertinent question is: what evidence would you actually accept as proof? Before we get to actual world ending magic, we would see very significant damages along the way, long before we get to that endpoint. I have been quite clear about what evidence I require. Show existing capabilities and show what harm could be caused if it incrementally gets better in that category. If you are worried about it making a kill bot, then show me how its existing kill bot capabilities are any more dangerous than my "duct tape gun to drone" idea. And show how the designs itself are the limiting factor and not the factories (which a chatbot doesn't help much with). But saying "Look how good of a chat bot it is, therefore it can hack the world governments" isn't evidence. Instead, that is merely evidence of AI being good at chat bots. Show me it being any good at all at hacking, and then we can evaluate it being a bit better. Show me the existing computers that are right now, as of this moment, being hacked by AI, and then we can evaluate the damage of it becomes twice as good at hacking. Just like how we can see the images that it generates now, and we can imagine those images being better. Therefore proving that deepfakes are a reasonable thing to talk about. (even if deep fakes aren't world ending. lots of people can make deepfakes without AI. Its not that big of a deal)
- permo-w 3y ago
  look, I'm going to humour you here, but my instinct is that you'll just dismiss any potential anyway first of all, by dismissing them as chatbots, you're inaccurately downplaying their significance to the aid of your argument. they're not chatbots, they're knowledge machines. they're machines you load knowledge into, which can produce new, usually accurate conclusions based on that knowledge. they're incredibly good at this and getting better. as it is, they have very restrictive behaviour guards on them and they're running server-side, but in a few years time, there will be gpt-4 level OSS models that do not and are not humans are slow and run out of energy quickly and lose focus. those are the limiting factors upon human chaotic interference, and yet there is plenty of that as it is. a sufficiently energetic, focused human, who thinks at 1000x normal human speed could do almost anything on the internet. that is the danger. I suspect to some degree you haven't taken the main weakness into account: almost all safeguards can be removed with blackmail. blackmail is something especially possible for LLMs, given that it is purely executed using words. you want to build a kill bot and the factory says no? blackmail the head of the factory. threaten his family. you have access to the entire internet at 1000x speed. you can probably find his address. you can pay someone on fiverr to go and take a picture of his house, or write something on his door, etc. you could even just pay a private detective to do this work for you over email. pay some unscrupulous characters on telegram/TOR to actually kidnap them. realistically how hard would it be for a well-funded operation to set up a bot that can do this on its own? you set up a cycle of "generate instructions for {goal}", "elaborate upon each instruction", "execute each {instruction}", "generate new instructions based on results of execution", and repeat. yeah maybe the first 50,000 cycles don't work, but you only need 1. nukes may well be air-gapped, but (some of) the people that control them will be online. all it takes is for one of them to choose the life of a loved one. all it takes is for one lonely idiot to be trapped into a weird kinky online relationship where blowing up the world/betraying your govt is the ultimate turn on for the "girl"/"boy" you love. if it's not convincing to you that that could happen with the people working with nukes, there are far less well-protected points of weakness that could be exploited: infectious diseases; lower priority military equipment; energy infrastructure; water supplies; or they could find a way to massively accelerate the release of methane into the atmosphere. etc, etc, etc this is the risk solely from LLMs. now take an AGI who can come up with even better plans and doesn't need human guidance, plus image gen, video gen, and voice gen, and you have an existential threat