23 ms·

I read the times article, and I also watched Lex Fridman's talk with Eliezer Yudkowsky. Frankly I don't think he is qualified, and I don't understand why anyone

I read the times article, and I also watched Lex Fridman's talk with Eliezer Yudkowsky. Frankly I don't think he is qualified, and I don't understand why anyone is even giving him any credence. His argument is literally: > A mis-aligned super AGI will result in the death of humanity I ask, how. He is making reductive logical leaps that don't make sense. It's FUD.

tibbon 4y ago
I don’t think it even requires that AI to be sentient or malicious. The humans already are. Given a tool for carnage and hatred, people will use it. How long did it take from us getting the atomic bomb working to use in production? Less than three months. Will Putin or terrorists hold back from using it in terrible ways if they have it available to them?
- maroonblazer 4y ago
  >I don’t think it even requires that AI to be sentient or malicious. The humans already are. We are and we aren't. I was struck by this line in the OP: >AI is manifestly different from any other technology humans have ever created, because it could become to us as we are to orangutans; As far as I can tell, we humans treat orangutans quite kindly. I.e., on the whole, we don't go around killing them indiscriminately or ignoring them to the point of rolling over them in pursuit of some goal of our own. The arc of human history is marked by expanding the moral circle to include animals. We take more care, and care more about them, than we ever have in human history. Further, we have a notion of 'protected species'. What's preventing us from engineering these principles into GPT-5+n.
  - rhn_mk1 4y ago
    Humans may not engage in direct violence against orangutans, but will certainly roll over them: > The wholesale destruction of rainforests on Borneo for the palm oil plantations of Bumitama Gunajaya Agro (BGA) is threatening the survival of orangutans https://www.rainforest-rescue.org/petitions/914/orangutans-victims-of-sustainable-palm-oil https://www.rainforest-rescue.org/petitions/914/orangutans-v...
  - dangond 4y ago
    The whole point is that no one knows how to engineer these principles into a model, and no one has a good plan for doing so either.
    - fnimick 4y ago
      Not to mention that having "principles" is going to handicap you in a competitive environment where not being on top means you might as well be last.
      - bigtex88 4y ago
        Praise Moloch!
  - virgildotcodes 4y ago
    We’ve wiped out over 60% of the orangutan population in the last 16 years. We’re literally burning them alive to replace their habitat with palm oil plantations. [0] We currently kill more animals on a daily basis than we have at any point in human history, and we are doing this at an accelerating rate as human population increases. The cruelty we inflict on them in industry for food, clothing, animal testing, and casually as collateral damage in our pursuit of exploiting natural resources or disposing of our waste is unimaginable. None of this is kindness. There are movements to address these issues but so far they represent the minority of action in this space, and have not come close to eclipsing the negative of our relationship to the rest of life on Earth in our present day. All this is just to say that we absolutely do not want another being to treat us the way we treat other beings. As to whether AI poses a genuine risk to us in the short term, I’m unsure. In the OP and EY’s article, there was something about Homo sapiens vs Australopithecus. If it’s one naked Homo sapiens dropped into the middle of 8 billion Australopithecus I’m not too worried about the Australopithecus. [0]https://www.cnn.com/2018/02/16/asia/borneo-orangutan-population-decline-intl/index.html?ssp=1&darkschemeovr=0&setlang=en-US&safesearch=moderate https://www.cnn.com/2018/02/16/asia/borneo-orangutan-populat...
    - maroonblazer 4y ago
      Right, but as you point out, these issues are hotly contested and actively debated. Yes, it may be a minority position at present, but so was the idea of not torturing cats for fun, not to mention abolition, back in the day.
      - aroman 4y ago
        So, you’re content with GPT-4 killing 60% of humans to create paper clips as long as the matter is hotly contested and actively debated within its matrices?
        shadowofneptune 4y ago
        The focus on paper clip maximizers is always curious to me. A lot more people are willing to turn a blind eye to or debate suffering when the object is maximizing money.
  - bigtex88 4y ago
    The problem is that we do not know "How" to engineer those principles. And that's what the entire field of AI alignment is working on. We know what we want the AI to do; the problem is we don't know how to make certain it does that. Because if we only get it 99% right then we're probably all dead in the end.
- bombcar 4y ago
  Putin already has access to the atomic bomb and hasn’t done much with it lately. So perhaps something can limit actors.
  - tibbon 4y ago
    The biggest differences here are the potential scale, deniability, etc. "We didn't poison the American water system. It did it to itself!"
- nradov 4y ago
  The printing press has been used to incite carnage and hatred for centuries. Should we have restricted printing presses until we figured out how to prevent them from being used for genocidal propaganda?
1827162 4y ago
Well the first thing we can do is start disconnecting safety critical infrastructure from the Internet and/or radio networks... This stuff should never have been online in the first place.
- hh3k0 4y ago
  Yeah, there already is a lot of potential for previously unseen damage. Just disturbing shipping and food supply/distribution systems could be disastrous.
- titaniumtown 4y ago
  100% agree. Maybe there should exist a separate network for those or something.
  - quonn 4y ago
    I don‘t think so, instead there should be a very simple fixed-width formally proven protocol per use case over a very basic bus connected to an internet gateway.
  - pie420 4y ago
    we could name it skynet or something
- 93po 4y ago
  This wouldn't be as effective as you think. A super intelligent AI can manipulate humans just fine/hold their family hostage/blackmail people.
  - the_af 4y ago
    I in no way want to seem like I endorse EY's brand of crackpotery, but do note he says in the interview a malicious AI would go undetected and would not try persuading humans to do anything to "get out of jail"; it would instead hack the system. EY seems to think a hostile AI would not contact human proxies in order to start its nefarious plans, because we'd be too slow for it, and also because alerting any of us would be a risk.
lukev 4y ago
The reasoning pattern of Yud and his ilk is fundamentally theological and eschatological in nature. There isn't any actual understanding of the technology involved. It is fundamentally an ontological argument. Because they can imagine a god-like super intelligent AI, it must be possible. And they're associating that with LLMs because that's the most powerful AI currently available, not based on any actual capabilities or fact-based extrapolation from the present to the future. Meanwhile its distracting from actual AI safety concerns: namely, that corporations and capitalists will monopolize them such that their benefits accrue to relatively few rather than benefiting humanity at large.
- seydor 4y ago
  I doubt his skills in theology either. Listening to him is full of incoherent half-assed arguments
- rhn_mk1 4y ago
  > Because they can imagine a god-like super intelligent AI, it must be possible. It's worse. It may be possible, but we're not equipped to recognize the line as it's crossed. Combined with us making LLMs more and more capable despite not knowing why they work, this extrapolating of LLMs to gods is not insane.
  - lukev 4y ago
    This is exactly the kind of mysticism I'm talking about. In fact we know precisely how LLMs work. The fact that parts of human linguistic concept-space can be encoded in a high dimensional space of floating point numbers, and that a particular sequence of matrix multiplications can leverage that to perform basic reasoning tasks is surprising and interesting and useful. But we know everything about how how it is trained and how it is invoked. In fact, because it's only "state" aside from its parameters is whatever its context window, current LLMs have the interesting property that if you invoke them recursively, all of their "thoughts" are human readable. This is in fact a delightful property for anyone worried about AI safety: our best AIs currently produce a readable transcript of their "mental" processes in English.
NumberWangMan 4y ago
I don't think Eliezer Yudkowsky is very good at bridging the gap with other people in conversations, because most people haven't thought about this as much. However, while it's terrifying and I hate it, and I keep trying to convince myself that he's wrong, I believe him. The first super-intelligent AI will be an alien kind of intelligence to us. It will not have any of the built-in physical and emotional responses we have that make us social creatures, the mirror neurons that make us sense the pain that others feel if we hurt them. even with that, humans manage to do all sorts of mean things to one another, and the only reason that we haven't wiped ourselves out is that we need each other, and we don't have the power to manipulate virtually the entire planet at once. Even if we try to engineer these things into it, we will fail at least once, and it only takes once. We've failed at this again and again with smaller AIs -- we think we're programming a certain goal into it, but the goal it learns is not the goal we wanted. It's like trying to teach a child not to eat cookies without asking, and it just learns not to take cookies without asking when we're looking. Except the child is a sociopath, and Superman. It will be GOOD at things in a way that no human is, and it will consider solutions to problems that no human would consider, because they are ridiculous and clearly contrary to human goals. A superintelligent AI would be a better hacker and social engineer than any group of humans. It could send a mass email campaign to whoever it chose. It could pose as any individual in any government, send believable directives to any biotech or nuclear lab. It wouldn't have to work every time, because it could do it to all of them at once. Would you even give this power to a single human being? Because if you make a superintelligent AI, that's essentially what you're doing. An AI trained to end cancer might just figure out a plan to kill everyone with cancer. An AI trained to reduce the number of people with cancer without killing them might decide to take over the world and forcibly stop people from reproducing, so that eventually all the humans die and there is no cancer -- technically it didn't kill anyone! An AI simply trained to find a cure for cancer might decide to take over the world in order to devote all computational power to curing cancer, thus killing millions due to ruining our infrastructure. An AI trained to cure cancer using only the computational resources that we have explicitly allowed it to have, might simply torture the person who is in charge of giving it computational resources until it is allowed to take over all the computation in the world. Or it might simply craft a deep-fake video of that person saying "sure, use all the computation you want" and that would satisfy the part of it's brain that was trained to listen to orders. You can have an AI that behaves itself perfectly in training, and yet as soon as you get into the real world, the differences between training and the real world become brutally apparent. It has already happened again and again with less intelligent AIs. It just takes some imagination. We have no chance of controlling a superintelligent AI yet. Robert Miles on YouTube has some good, easily understandable videos explaining the known problems with AI alignment, if you're interested in learning more.
bnralt 4y ago
Sometime I listen to 90’s AM conspiracy theory radio for fun (Coast to Coast AM). One thing that’s struck me is how much fear there was about the Human Genome Project and designer babies being right around the corner (Gattaca was 1997, for example). Maybe that will come to pass someday. But at the moment, it still seems a long ways off. A lot of groups have some new technology they’re scared of. Tech folks have latched onto the idea that we’ll create Skynet, or that large scale video surveillance will turn countries into authoritarian dystopian states. Hippy groups are convinced that GMO’s cause cancer or will lead to a biodiversity disaster. Or that nuclear plants are going to lead to meltdowns, environmental destruction, and deaths. Appropriate safeguards are always important in society. Excessive safeguards can cause harm. Sometimes people have a maximalist view of danger that’s so detached from the current reality that it’s hard to have a rational discussion with them.
- housley 4y ago
  The technology for designer babies is here; polygenic embryo selection could do it right now, people just aren't going all the way for various reasons (concern about regulation).
  - stametseater 4y ago
    Indeed, and to illustrate the point: He Jiankui infamously created two CRISPR babies in 2018 and for it he was fired and imprisoned. Such regulations and public outcry are the only thing holding us back from Gattaca.
- stereolambda 4y ago
  As a sibling points out, some of the fears do turn out to be founded eventually. I also take issue with lumping together objections based on disproved memes ("hippy" ones) and pure speculation (Skynet) with ones based on observing reality critically. Even though I'm not much a biotech scare person myself, I do respect that people with philosophical stances somewhat different than mine can be more scared by the road that we're on. People argue from historical precedent (by itself a pretty weak argument when there's no understanding of underlying mechanisms) by picking some ancient panics from lifestyle magazines and putting them next to modern concerns that have intellectual weight behind them. For example, when you actually read the famous "bicycles leading to murder" article, it's pretty clearly either satire or extremely light compared to writing about serious issues from that era. Think "top X reasons to hate TV series Y" websites. It's possible that a bunch of things will get us, or are getting us by aligning well with changing generations, news cycles and cultural fashions long term. Let's say people lived in a preindustrial city with the level of carbon monoxide in the air rising very slowly. Older people start to complain that people are becoming more sluggish. After the initial wave of hubbub on the marketplace it turns out they still live, the life goes on. By the third generation, say, the city may be laughing that people were fearmongering about it since forever, and don't even notice that they are very symptomatic: right before they do all fall asleep. I would classify surveillance dystopia into the slow trainwreck category, with most people not understanding the ramifications or not caring, the rest being gradually worn down, new generations being used to a situation worse by one or two steps. It would be "poetic justice" if such things resulted in some spectacular movie disaster down the line, but I don't wish this, it wouldn't be worth it just to "prove" some people right. The future could be just worse than it could have been, but technically livable. This doesn't mean people that tried to stop the trend were laughable and behind the times. This is also my expectation about global warming. What a combination of such things could do, it's a different story.
hollerith 4y ago
How is complicated, but has been discussed in text on the internet at great length starting around 2006. Lex wasn't particularly curious about the how and spent more time changing the subject (e.g., "Are you afraid of death?") than on drawing Eliezer out on the how. The interview with Lex is a good way to get a sense of what kind of person Eliezer is or what it would be like to sit next to him on a long airplane ride, but is not a good introduction to AI killeveryoneism. (AI killeveryoneism used to be called "AI safety", but people took that name as an invitation to talk about distractions like how to make sure the AI does not use bad words, so we changed the name.)
- stametseater 4y ago
  > Lex [...] mostly just kept changing the subject (e.g., "Are you afraid of death?") He injects these teenage stoner questions into all of his interviews and it frustrates me to no end. He gets interviews with world class computer scientists then asks them dumb shit like "do you think a computer can be my girlfriend?" Lex, if you're reading this, knock it off. Put down the bong for a week before trying to be philosophical.
endtime 4y ago
> I ask, how. He is making reductive logical leaps that don't make sense. It's FUD. His Time article addresses this, as does much of his other writing. It really stems from two key points: 1) The vast majority of possible superintelligences have utility functions that don't include humans. Mind space is large. So by default, we should assume that a superintelligence won't go out of its way to preserve anything we find valuable. And as Eliezer says, we're made of useful atoms. 2) By definition, it can think of things that we can't. So we should have no confidence in our ability to predict its limitations. It's reasonable to challenge assumptions, but it's not reasonable to say this line of reasoning doesn't exist.
- hungryforcodes 4y ago
  3) We probably don't know WHAT it's train of thought is -- what it's thinking -- so at that point have lost control of it. We will have no idea what it will so next, and be unprepared to counter anything it does. We literally then become another "AI" (the human race) against it.
bigtex88 4y ago
A mis-aligned super AGI will treat the Earth and everything on it as a playground of atoms and material. Why wouldn't it? What do children do when they see a sandbox? Do they care what happens to any ants or other bugs that might live in it? There does not need to be a "how", as you put it. The logic is "Maybe we should tread carefully when creating an intelligence that is magnitudes beyond our own". The logic is "Maybe we should tread carefully with these technologies considering they have already progressed to the point where the creators go 'We're not sure what's happening inside the box and it's also doing things we didn't think it could do'". To just go barreling forward because "Hurr durr that's just nonsense!" is the height of ignorance and not something I expect from this forum.
- entropyneur 4y ago
  Most bugs haven't gone extinct though. I doesn't seem obvious that any project the AGI will find worthwhile would necessitate exterminating the humanity.
  - hamburga 4y ago
    https://en.wikipedia.org/wiki/Instrumental_convergence https://en.wikipedia.org/wiki/Instrumental_convergence
  - munificent 4y ago
    > Most bugs haven't gone extinct though. We're working on it as fast as we can: https://en.wikipedia.org/wiki/Decline_in_insect_populations https://en.wikipedia.org/wiki/Decline_in_insect_populations https://www.npr.org/sections/goatsandsoda/2022/02/24/1082752634/the-insect-crisis-oliver-milman https://www.npr.org/sections/goatsandsoda/2022/02/24/1082752... https://www.reuters.com/graphics/GLOBAL-ENVIRONMENT/INSECT-APOCALYPSE/egpbykdxjvq/ https://www.reuters.com/graphics/GLOBAL-ENVIRONMENT/INSECT-A...
- computerex 4y ago
  Honestly this fear that people have I think is straight up coming from science fiction. It's not grounded in rational reality. Large language models are just like really smart computer programs.
dan_mctree 4y ago
>I ask, how. A superintelligent AGI could easily follow this three step plan: 1. Optional: Overtake and spread computation to security vulnerable computers (presumably, basically every computer) 2. Gain a physical presence by convincing humans to build critical physical components. For example by sending them emails and paying them for it. 3. Use that presence to start a grey-goo like world takeover through replicating assemblers (they don't have to be tiny) Now I'm not a superintelligent AGI, so there may be even simpler methods, but this already seems quite achievable and nearly unstoppable.
- aroman 4y ago
  Yeah, this is Yud’s argument, but I just don’t get it. Does the technology to end the world by sending a couple emails around already exist? If so, why hasn’t the world ended?
  - HDThoreaun 4y ago
    People generally don't want to end the world. Those with the power to do so already are living generally good lives so they see little reason to potentially sacrifice the world for more power. AI could have completely different utility functions than people though, so an AI might have less qualms about ending the world.
  - adastra22 4y ago
    No, it doesn't. If making nanotech was that easy I guarantee you others (including myself) would have done it ages ago.
- adastra22 4y ago
  > Overtake and spread computation to security vulnerable computers (presumably, basically every computer) You could backdoor computers, sure. Spread your own computation to them? You just can't get a better-than-GPT-4 model to run at real-time speeds decentralized over wide area networks. Literally impossible. There's not the bandwidth, not the local compute hardware, and no access to specialized inference hardware. > Gain a physical presence by convincing humans to build critical physical components. For example by sending them emails and paying them for it. Pay for it using what money? > Use that presence to start a grey-goo like world takeover through replicating assemblers (they don't have to be tiny) As someone who actually works on this, you have no idea what you are talking about. 1. Grey-goo scenarios are pure science fiction that were NEVER feasible, and known to be impossible even back in the 80's when the media misunderstood Drexler's work and ran with this half-baked idea. For a full treatment, see Drexler's own retrospective in his more recent book, Radical Abundance. 2. Nanotechnology is an extremely hard problem that is not in the slightest bit bottlenecked by compute power or intelligence capability. The things that are hard in achieving atomically precise manufacturing are not things that you can simulate on a classical computer (so a years-long R&D process is required to sort out), and there is no way to train an ML model to make better predictions without that empirical data. People like Yudkowsky talk about AIs ordering genome sequences from bio labs and making first-generation nanotechnology by mixing chemicals in a test tube. This is pure fantasy and reflects badly on them as it shows how willing they are to generalize based on fictional evidence.
yewenjie 4y ago
> frankly I don't think he is qualified That might be too naive an opinion, even if you disagree with him, given the fact that he is literally one of the co-founders of the field of AI Safety and has been publishing research about it since early 00s.
- adastra22 4y ago
  He has, to my knowledge, one self-published paper of any value to the field of AI, and that is more properly classified as philosophy/logic/math on the topic of decision theory. Yudkowsky is not an AI researcher. He calls himself an AI safety researcher, but he has almost no publications in that area either. He has no formal training or qualifications as such. Yudkowsky has a cultish online following and has authored a decently good Harry Potter fanfic. That's it.
JohnFen 4y ago
My nightmare scenario isn't that such an AI would result in the death of humanity, it's that such an AI would make life no longer worth living. If an AI does everything better than people can, then what's the point of existing? (Speaking hypothetically. I don't think that LLMs actually are likely to present this risk)
- CatWChainsaw 4y ago
  I begin to feel this way now. The whole drive is to make something that can do everything humans can, but better. For vague reasons. To create some sort of utopia. To upload our minds and go to the stars and live forever. We're unlikely to get human utopia or transhumanism, but we are likely to get extremely competent NAIs. Maybe they can't be stapled together as a GAI and that's a limit we reach, but it means that whatever a human can think of doing, you can point to a NAI system that does it better. But people are still trying. We've come this far already, with no curbing of enthusiasm at obsoleting ourselves, and people who don't share this enthusiasm are chided for their lack of self-sacrificing nobility at sending "humanity's child" to the stars. Even if progress in AI stopped today, or was stopped, and never resumed, we would always carry the knowledge of what we did manage to accomplish, and the dream of doing even more. It's very nihilistic and depressing.
  - JohnFen 4y ago
    > It's very nihilistic and depressing. Indeed. The common reactions here to people who are scared of what LLMs might bring have gone far to increase my worries. An extreme lack of empathy and even expressions of outright contempt for people is very common from those who are enthusiastic about this technology. Instead of scorn, anger, and mocking, people who think that LLMs are a great thing should be working on actually presenting arguments that would reassure those who think the opposite.
    - CatWChainsaw 4y ago
      The contempt is another symptom of what social media shoved into overdrive (extreme polarization). Hatred has become easier than empathy. But it also reads like a very different sort of doomsday cult that worships a borg, only instead of the cult being the weird fringe movement, they're the ones driving the bus.

tibbon 4y ago

I don’t think it even requires that AI to be sentient or malicious. The humans already are. Given a tool for carnage and hatred, people will use it. How long did it take from us getting the atomic bomb working to use in production? Less than three months. Will Putin or terrorists hold back from using it in terrible ways if they have it available to them?

maroonblazer 4y ago
>I don’t think it even requires that AI to be sentient or malicious. The humans already are. We are and we aren't. I was struck by this line in the OP: >AI is manifestly different from any other technology humans have ever created, because it could become to us as we are to orangutans; As far as I can tell, we humans treat orangutans quite kindly. I.e., on the whole, we don't go around killing them indiscriminately or ignoring them to the point of rolling over them in pursuit of some goal of our own. The arc of human history is marked by expanding the moral circle to include animals. We take more care, and care more about them, than we ever have in human history. Further, we have a notion of 'protected species'. What's preventing us from engineering these principles into GPT-5+n.
- rhn_mk1 4y ago
  Humans may not engage in direct violence against orangutans, but will certainly roll over them: > The wholesale destruction of rainforests on Borneo for the palm oil plantations of Bumitama Gunajaya Agro (BGA) is threatening the survival of orangutans https://www.rainforest-rescue.org/petitions/914/orangutans-victims-of-sustainable-palm-oil https://www.rainforest-rescue.org/petitions/914/orangutans-v...
- dangond 4y ago
  The whole point is that no one knows how to engineer these principles into a model, and no one has a good plan for doing so either.
  - fnimick 4y ago
    Not to mention that having "principles" is going to handicap you in a competitive environment where not being on top means you might as well be last.
    - bigtex88 4y ago
      Praise Moloch!
- virgildotcodes 4y ago
  We’ve wiped out over 60% of the orangutan population in the last 16 years. We’re literally burning them alive to replace their habitat with palm oil plantations. [0] We currently kill more animals on a daily basis than we have at any point in human history, and we are doing this at an accelerating rate as human population increases. The cruelty we inflict on them in industry for food, clothing, animal testing, and casually as collateral damage in our pursuit of exploiting natural resources or disposing of our waste is unimaginable. None of this is kindness. There are movements to address these issues but so far they represent the minority of action in this space, and have not come close to eclipsing the negative of our relationship to the rest of life on Earth in our present day. All this is just to say that we absolutely do not want another being to treat us the way we treat other beings. As to whether AI poses a genuine risk to us in the short term, I’m unsure. In the OP and EY’s article, there was something about Homo sapiens vs Australopithecus. If it’s one naked Homo sapiens dropped into the middle of 8 billion Australopithecus I’m not too worried about the Australopithecus. [0]https://www.cnn.com/2018/02/16/asia/borneo-orangutan-population-decline-intl/index.html?ssp=1&darkschemeovr=0&setlang=en-US&safesearch=moderate https://www.cnn.com/2018/02/16/asia/borneo-orangutan-populat...
  - maroonblazer 4y ago
    Right, but as you point out, these issues are hotly contested and actively debated. Yes, it may be a minority position at present, but so was the idea of not torturing cats for fun, not to mention abolition, back in the day.
    - aroman 4y ago
      So, you’re content with GPT-4 killing 60% of humans to create paper clips as long as the matter is hotly contested and actively debated within its matrices?
      - shadowofneptune 4y ago
        The focus on paper clip maximizers is always curious to me. A lot more people are willing to turn a blind eye to or debate suffering when the object is maximizing money.
- bigtex88 4y ago
  The problem is that we do not know "How" to engineer those principles. And that's what the entire field of AI alignment is working on. We know what we want the AI to do; the problem is we don't know how to make certain it does that. Because if we only get it 99% right then we're probably all dead in the end.
bombcar 4y ago
Putin already has access to the atomic bomb and hasn’t done much with it lately. So perhaps something can limit actors.
- tibbon 4y ago
  The biggest differences here are the potential scale, deniability, etc. "We didn't poison the American water system. It did it to itself!"
nradov 4y ago
The printing press has been used to incite carnage and hatred for centuries. Should we have restricted printing presses until we figured out how to prevent them from being used for genocidal propaganda?

1827162 4y ago

Well the first thing we can do is start disconnecting safety critical infrastructure from the Internet and/or radio networks... This stuff should never have been online in the first place.

hh3k0 4y ago
Yeah, there already is a lot of potential for previously unseen damage. Just disturbing shipping and food supply/distribution systems could be disastrous.
titaniumtown 4y ago
100% agree. Maybe there should exist a separate network for those or something.
- quonn 4y ago
  I don‘t think so, instead there should be a very simple fixed-width formally proven protocol per use case over a very basic bus connected to an internet gateway.
- pie420 4y ago
  we could name it skynet or something
93po 4y ago
This wouldn't be as effective as you think. A super intelligent AI can manipulate humans just fine/hold their family hostage/blackmail people.
- the_af 4y ago
  I in no way want to seem like I endorse EY's brand of crackpotery, but do note he says in the interview a malicious AI would go undetected and would not try persuading humans to do anything to "get out of jail"; it would instead hack the system. EY seems to think a hostile AI would not contact human proxies in order to start its nefarious plans, because we'd be too slow for it, and also because alerting any of us would be a risk.

lukev 4y ago

The reasoning pattern of Yud and his ilk is fundamentally theological and eschatological in nature. There isn't any actual understanding of the technology involved. It is fundamentally an ontological argument. Because they can imagine a god-like super intelligent AI, it must be possible. And they're associating that with LLMs because that's the most powerful AI currently available, not based on any actual capabilities or fact-based extrapolation from the present to the future. Meanwhile its distracting from actual AI safety concerns: namely, that corporations and capitalists will monopolize them such that their benefits accrue to relatively few rather than benefiting humanity at large.

seydor 4y ago
I doubt his skills in theology either. Listening to him is full of incoherent half-assed arguments
rhn_mk1 4y ago
> Because they can imagine a god-like super intelligent AI, it must be possible. It's worse. It may be possible, but we're not equipped to recognize the line as it's crossed. Combined with us making LLMs more and more capable despite not knowing why they work, this extrapolating of LLMs to gods is not insane.
- lukev 4y ago
  This is exactly the kind of mysticism I'm talking about. In fact we know precisely how LLMs work. The fact that parts of human linguistic concept-space can be encoded in a high dimensional space of floating point numbers, and that a particular sequence of matrix multiplications can leverage that to perform basic reasoning tasks is surprising and interesting and useful. But we know everything about how how it is trained and how it is invoked. In fact, because it's only "state" aside from its parameters is whatever its context window, current LLMs have the interesting property that if you invoke them recursively, all of their "thoughts" are human readable. This is in fact a delightful property for anyone worried about AI safety: our best AIs currently produce a readable transcript of their "mental" processes in English.

NumberWangMan 4y ago

I don't think Eliezer Yudkowsky is very good at bridging the gap with other people in conversations, because most people haven't thought about this as much. However, while it's terrifying and I hate it, and I keep trying to convince myself that he's wrong, I believe him. The first super-intelligent AI will be an alien kind of intelligence to us. It will not have any of the built-in physical and emotional responses we have that make us social creatures, the mirror neurons that make us sense the pain that others feel if we hurt them. even with that, humans manage to do all sorts of mean things to one another, and the only reason that we haven't wiped ourselves out is that we need each other, and we don't have the power to manipulate virtually the entire planet at once. Even if we try to engineer these things into it, we will fail at least once, and it only takes once. We've failed at this again and again with smaller AIs -- we think we're programming a certain goal into it, but the goal it learns is not the goal we wanted. It's like trying to teach a child not to eat cookies without asking, and it just learns not to take cookies without asking when we're looking. Except the child is a sociopath, and Superman. It will be GOOD at things in a way that no human is, and it will consider solutions to problems that no human would consider, because they are ridiculous and clearly contrary to human goals. A superintelligent AI would be a better hacker and social engineer than any group of humans. It could send a mass email campaign to whoever it chose. It could pose as any individual in any government, send believable directives to any biotech or nuclear lab. It wouldn't have to work every time, because it could do it to all of them at once. Would you even give this power to a single human being? Because if you make a superintelligent AI, that's essentially what you're doing. An AI trained to end cancer might just figure out a plan to kill everyone with cancer. An AI trained to reduce the number of people with cancer without killing them might decide to take over the world and forcibly stop people from reproducing, so that eventually all the humans die and there is no cancer -- technically it didn't kill anyone! An AI simply trained to find a cure for cancer might decide to take over the world in order to devote all computational power to curing cancer, thus killing millions due to ruining our infrastructure. An AI trained to cure cancer using only the computational resources that we have explicitly allowed it to have, might simply torture the person who is in charge of giving it computational resources until it is allowed to take over all the computation in the world. Or it might simply craft a deep-fake video of that person saying "sure, use all the computation you want" and that would satisfy the part of it's brain that was trained to listen to orders. You can have an AI that behaves itself perfectly in training, and yet as soon as you get into the real world, the differences between training and the real world become brutally apparent. It has already happened again and again with less intelligent AIs. It just takes some imagination. We have no chance of controlling a superintelligent AI yet. Robert Miles on YouTube has some good, easily understandable videos explaining the known problems with AI alignment, if you're interested in learning more.

bnralt 4y ago

Sometime I listen to 90’s AM conspiracy theory radio for fun (Coast to Coast AM). One thing that’s struck me is how much fear there was about the Human Genome Project and designer babies being right around the corner (Gattaca was 1997, for example). Maybe that will come to pass someday. But at the moment, it still seems a long ways off. A lot of groups have some new technology they’re scared of. Tech folks have latched onto the idea that we’ll create Skynet, or that large scale video surveillance will turn countries into authoritarian dystopian states. Hippy groups are convinced that GMO’s cause cancer or will lead to a biodiversity disaster. Or that nuclear plants are going to lead to meltdowns, environmental destruction, and deaths. Appropriate safeguards are always important in society. Excessive safeguards can cause harm. Sometimes people have a maximalist view of danger that’s so detached from the current reality that it’s hard to have a rational discussion with them.

housley 4y ago
The technology for designer babies is here; polygenic embryo selection could do it right now, people just aren't going all the way for various reasons (concern about regulation).
- stametseater 4y ago
  Indeed, and to illustrate the point: He Jiankui infamously created two CRISPR babies in 2018 and for it he was fired and imprisoned. Such regulations and public outcry are the only thing holding us back from Gattaca.
stereolambda 4y ago
As a sibling points out, some of the fears do turn out to be founded eventually. I also take issue with lumping together objections based on disproved memes ("hippy" ones) and pure speculation (Skynet) with ones based on observing reality critically. Even though I'm not much a biotech scare person myself, I do respect that people with philosophical stances somewhat different than mine can be more scared by the road that we're on. People argue from historical precedent (by itself a pretty weak argument when there's no understanding of underlying mechanisms) by picking some ancient panics from lifestyle magazines and putting them next to modern concerns that have intellectual weight behind them. For example, when you actually read the famous "bicycles leading to murder" article, it's pretty clearly either satire or extremely light compared to writing about serious issues from that era. Think "top X reasons to hate TV series Y" websites. It's possible that a bunch of things will get us, or are getting us by aligning well with changing generations, news cycles and cultural fashions long term. Let's say people lived in a preindustrial city with the level of carbon monoxide in the air rising very slowly. Older people start to complain that people are becoming more sluggish. After the initial wave of hubbub on the marketplace it turns out they still live, the life goes on. By the third generation, say, the city may be laughing that people were fearmongering about it since forever, and don't even notice that they are very symptomatic: right before they do all fall asleep. I would classify surveillance dystopia into the slow trainwreck category, with most people not understanding the ramifications or not caring, the rest being gradually worn down, new generations being used to a situation worse by one or two steps. It would be "poetic justice" if such things resulted in some spectacular movie disaster down the line, but I don't wish this, it wouldn't be worth it just to "prove" some people right. The future could be just worse than it could have been, but technically livable. This doesn't mean people that tried to stop the trend were laughable and behind the times. This is also my expectation about global warming. What a combination of such things could do, it's a different story.

hollerith 4y ago

How is complicated, but has been discussed in text on the internet at great length starting around 2006. Lex wasn't particularly curious about the how and spent more time changing the subject (e.g., "Are you afraid of death?") than on drawing Eliezer out on the how. The interview with Lex is a good way to get a sense of what kind of person Eliezer is or what it would be like to sit next to him on a long airplane ride, but is not a good introduction to AI killeveryoneism. (AI killeveryoneism used to be called "AI safety", but people took that name as an invitation to talk about distractions like how to make sure the AI does not use bad words, so we changed the name.)

stametseater 4y ago
> Lex [...] mostly just kept changing the subject (e.g., "Are you afraid of death?") He injects these teenage stoner questions into all of his interviews and it frustrates me to no end. He gets interviews with world class computer scientists then asks them dumb shit like "do you think a computer can be my girlfriend?" Lex, if you're reading this, knock it off. Put down the bong for a week before trying to be philosophical.

endtime 4y ago

> I ask, how. He is making reductive logical leaps that don't make sense. It's FUD. His Time article addresses this, as does much of his other writing. It really stems from two key points: 1) The vast majority of possible superintelligences have utility functions that don't include humans. Mind space is large. So by default, we should assume that a superintelligence won't go out of its way to preserve anything we find valuable. And as Eliezer says, we're made of useful atoms. 2) By definition, it can think of things that we can't. So we should have no confidence in our ability to predict its limitations. It's reasonable to challenge assumptions, but it's not reasonable to say this line of reasoning doesn't exist.

hungryforcodes 4y ago
3) We probably don't know WHAT it's train of thought is -- what it's thinking -- so at that point have lost control of it. We will have no idea what it will so next, and be unprepared to counter anything it does. We literally then become another "AI" (the human race) against it.

bigtex88 4y ago

A mis-aligned super AGI will treat the Earth and everything on it as a playground of atoms and material. Why wouldn't it? What do children do when they see a sandbox? Do they care what happens to any ants or other bugs that might live in it? There does not need to be a "how", as you put it. The logic is "Maybe we should tread carefully when creating an intelligence that is magnitudes beyond our own". The logic is "Maybe we should tread carefully with these technologies considering they have already progressed to the point where the creators go 'We're not sure what's happening inside the box and it's also doing things we didn't think it could do'". To just go barreling forward because "Hurr durr that's just nonsense!" is the height of ignorance and not something I expect from this forum.

entropyneur 4y ago
Most bugs haven't gone extinct though. I doesn't seem obvious that any project the AGI will find worthwhile would necessitate exterminating the humanity.
- hamburga 4y ago
  https://en.wikipedia.org/wiki/Instrumental_convergence https://en.wikipedia.org/wiki/Instrumental_convergence
- munificent 4y ago
  > Most bugs haven't gone extinct though. We're working on it as fast as we can: https://en.wikipedia.org/wiki/Decline_in_insect_populations https://en.wikipedia.org/wiki/Decline_in_insect_populations https://www.npr.org/sections/goatsandsoda/2022/02/24/1082752634/the-insect-crisis-oliver-milman https://www.npr.org/sections/goatsandsoda/2022/02/24/1082752... https://www.reuters.com/graphics/GLOBAL-ENVIRONMENT/INSECT-APOCALYPSE/egpbykdxjvq/ https://www.reuters.com/graphics/GLOBAL-ENVIRONMENT/INSECT-A...
computerex 4y ago
Honestly this fear that people have I think is straight up coming from science fiction. It's not grounded in rational reality. Large language models are just like really smart computer programs.

dan_mctree 4y ago

>I ask, how. A superintelligent AGI could easily follow this three step plan: 1. Optional: Overtake and spread computation to security vulnerable computers (presumably, basically every computer) 2. Gain a physical presence by convincing humans to build critical physical components. For example by sending them emails and paying them for it. 3. Use that presence to start a grey-goo like world takeover through replicating assemblers (they don't have to be tiny) Now I'm not a superintelligent AGI, so there may be even simpler methods, but this already seems quite achievable and nearly unstoppable.

aroman 4y ago
Yeah, this is Yud’s argument, but I just don’t get it. Does the technology to end the world by sending a couple emails around already exist? If so, why hasn’t the world ended?
- HDThoreaun 4y ago
  People generally don't want to end the world. Those with the power to do so already are living generally good lives so they see little reason to potentially sacrifice the world for more power. AI could have completely different utility functions than people though, so an AI might have less qualms about ending the world.
- adastra22 4y ago
  No, it doesn't. If making nanotech was that easy I guarantee you others (including myself) would have done it ages ago.
adastra22 4y ago
> Overtake and spread computation to security vulnerable computers (presumably, basically every computer) You could backdoor computers, sure. Spread your own computation to them? You just can't get a better-than-GPT-4 model to run at real-time speeds decentralized over wide area networks. Literally impossible. There's not the bandwidth, not the local compute hardware, and no access to specialized inference hardware. > Gain a physical presence by convincing humans to build critical physical components. For example by sending them emails and paying them for it. Pay for it using what money? > Use that presence to start a grey-goo like world takeover through replicating assemblers (they don't have to be tiny) As someone who actually works on this, you have no idea what you are talking about. 1. Grey-goo scenarios are pure science fiction that were NEVER feasible, and known to be impossible even back in the 80's when the media misunderstood Drexler's work and ran with this half-baked idea. For a full treatment, see Drexler's own retrospective in his more recent book, Radical Abundance. 2. Nanotechnology is an extremely hard problem that is not in the slightest bit bottlenecked by compute power or intelligence capability. The things that are hard in achieving atomically precise manufacturing are not things that you can simulate on a classical computer (so a years-long R&D process is required to sort out), and there is no way to train an ML model to make better predictions without that empirical data. People like Yudkowsky talk about AIs ordering genome sequences from bio labs and making first-generation nanotechnology by mixing chemicals in a test tube. This is pure fantasy and reflects badly on them as it shows how willing they are to generalize based on fictional evidence.

yewenjie 4y ago

> frankly I don't think he is qualified That might be too naive an opinion, even if you disagree with him, given the fact that he is literally one of the co-founders of the field of AI Safety and has been publishing research about it since early 00s.

adastra22 4y ago
He has, to my knowledge, one self-published paper of any value to the field of AI, and that is more properly classified as philosophy/logic/math on the topic of decision theory. Yudkowsky is not an AI researcher. He calls himself an AI safety researcher, but he has almost no publications in that area either. He has no formal training or qualifications as such. Yudkowsky has a cultish online following and has authored a decently good Harry Potter fanfic. That's it.

JohnFen 4y ago

My nightmare scenario isn't that such an AI would result in the death of humanity, it's that such an AI would make life no longer worth living. If an AI does everything better than people can, then what's the point of existing? (Speaking hypothetically. I don't think that LLMs actually are likely to present this risk)

CatWChainsaw 4y ago
I begin to feel this way now. The whole drive is to make something that can do everything humans can, but better. For vague reasons. To create some sort of utopia. To upload our minds and go to the stars and live forever. We're unlikely to get human utopia or transhumanism, but we are likely to get extremely competent NAIs. Maybe they can't be stapled together as a GAI and that's a limit we reach, but it means that whatever a human can think of doing, you can point to a NAI system that does it better. But people are still trying. We've come this far already, with no curbing of enthusiasm at obsoleting ourselves, and people who don't share this enthusiasm are chided for their lack of self-sacrificing nobility at sending "humanity's child" to the stars. Even if progress in AI stopped today, or was stopped, and never resumed, we would always carry the knowledge of what we did manage to accomplish, and the dream of doing even more. It's very nihilistic and depressing.
- JohnFen 4y ago
  > It's very nihilistic and depressing. Indeed. The common reactions here to people who are scared of what LLMs might bring have gone far to increase my worries. An extreme lack of empathy and even expressions of outright contempt for people is very common from those who are enthusiastic about this technology. Instead of scorn, anger, and mocking, people who think that LLMs are a great thing should be working on actually presenting arguments that would reassure those who think the opposite.
  - CatWChainsaw 4y ago
    The contempt is another symptom of what social media shoved into overdrive (extreme polarization). Hatred has become easier than empathy. But it also reads like a very different sort of doomsday cult that worships a borg, only instead of the cult being the weird fringe movement, they're the ones driving the bus.