4 ms·

Those people (like most people who talk AGI) live in a world far "above the metal." They, like the rationalist bloggers, live in a "software world," far above t

by mlsu 3y ago

Those people (like most people who talk AGI) live in a world far "above the metal." They, like the rationalist bloggers, live in a "software world," far above the constraints of physical reality. This causes an understandable blind spot: the understanding that computers are, fundamentally, physical machines. It's quite easy to imagine that you could conjure superintelligence when you are conjuring a pod of 8 quadrillion transistors in a single shell script. The illusion breaks down when you get closer to the physical reality. Reality is that those transistors are made of real materials, with real impurities, that have to be broken down and painstakingly physically, electrically, chemically, debugged, in the world below the 1's and 0's abstraction. The commonality that both sides of this silly debate share is that neither really deals with entities at the layer below abstract software (or essays). To them, the perfect world of software is fundamental reality (they spend all of their time there!), and any crystal prism that can be constructed in software can be constructed in reality. Of course, such prisms are instantly shattered when they come into contact with the real world: an impurity in the wafer, a static shock on the PCB before installation, a flood in the datacenter. This software stuff is fragile by default, it actually takes a tremendous amount of active work in the physical reality to conjure it into existence. I can do something that no language model can do, and is showing no signs of being able to do any time soon: drive a car to the datacenter, turn the door handle, and unplug the rack.

zrezzed 3y ago
> turn the door handle, and unplug the rack. There are door handles that you cannot turn, and racks you cannot unplug: ones buried deep in the Cheyenne Mountains and hidden far away in the Siberian tundra. They are protected by unfathomably powerful systems, with the support of countless people and backed by the whims global economic power. I say this as someone who likely agrees with you. I think the power of the real world, that of companies, and governments and militaries should decrease our concern with AGI gaining power itself. But I don’t think it’s as obvious as pointing to the fragility of software. Our human systems are fragile too, and subject to manipulation in not-so-different a way as data centers. I think you should not be so quick to discount the voices of many smart people shouting.
- mlsu 3y ago
  Human systems. They are protected by human systems. Human beings are the ones who press the big red button. We probably do agree. I'm not saying that this tech won't be used by bad actors. I already told my elderly relatives that if they haven't seen someone in person, they shouldn't talk on the phone. But what I'm talking about is categorically, fundamentally, not the eschaton!
  - high_5 3y ago
    Yes, they are human indeed, but those humans cannot agree on whether the plug should or shouldn't be pulled.
ethanbond 3y ago
So your argument is that we need not worry because Sutskever and Hinton aren’t aware you can destroy data centers with water or static electricity?
- skepticATX 3y ago
  The same Hinton who was certain that radiology was solved? The same Sutskever who led OpenAI when they released a neutered GPT-2 because the full model was too dangerous? The problem with the AI safety argument is that it can be boiled down to: 1) Assume that we build an all powerful god machine 2) The all powerful god machine has the ability to exterminate humanity All of their work revolves around different ways that 2) might occur. But what I, and many other people, take issue with is whether 1) is even possible, and if it is, if we’re even remotely close to building it. No one ever explains how 1) will be achieved beyond vague handwaving, because no one knows how to build a god machine.
  - deleted 3y ago
    [deleted]
  - ethanbond 3y ago
    (1) will happen if the following is true: A. Intelligence is substrate independent I.e. there’s nothing mythically special about biological neurons that allow them to “be intelligent.” B. Intelligence is economically valuable C. We don’t destroy ourselves first e.g. via nukes or asteroid We know for a fact that you can produce human-level intelligence that runs on very, very little power, and you can train that intelligence up to human-level in the course of 10-50 years of navigating the real world. Why do we know that? Because evolution did it already. Now is there something so remarkable about the tissue we happen to have in our brain that means it is literally impossible to replicate this success? I can’t seem to find what it is. Is there some reason to believe that the human brain is the absolute upper limit of what type of intelligence is possible in this universe? I can’t seem to find why that would be the case. Obviously everyone who needs to agree on the commercial incentive already does agree on the commercial incentive (see: bottomless VC and soon govt funding), so that’s a moot point. So explain how if these things are true: it’s possible, there’s immense incentives, and we have time enough to do it, that we don’t end up accomplishing it?
nl 3y ago
> I can do something that no language model can do, and is showing no signs of being able to do any time soon: drive a car to the datacenter, turn the door handle, and unplug the rack. Steven Hawking was generally intelligent and struggled to do these things. He'd also struggle to move the goal posts for AGI as much as this. (edit: to clarify I think AI Safety concerns are fever dreams of people who don't get out in the real world. But I don't think that reflects on what AGI looks like)
- mlsu 3y ago
  I'm not concerned that Stephen Hawking will destroy all matter in the light cone*. Remember, if this conversation wasn't about eschatology, "AGI" would be just another synonym for "smart, useful machines." I didn't set the terms of discussion. * or prevent humanity's technological transformation into the divine. Choose your flavor.
tim333 3y ago
Nah. We'll have robots for that.

by mlsu 3y ago

zrezzed 3y ago

> turn the door handle, and unplug the rack. There are door handles that you cannot turn, and racks you cannot unplug: ones buried deep in the Cheyenne Mountains and hidden far away in the Siberian tundra. They are protected by unfathomably powerful systems, with the support of countless people and backed by the whims global economic power. I say this as someone who likely agrees with you. I think the power of the real world, that of companies, and governments and militaries should decrease our concern with AGI gaining power itself. But I don’t think it’s as obvious as pointing to the fragility of software. Our human systems are fragile too, and subject to manipulation in not-so-different a way as data centers. I think you should not be so quick to discount the voices of many smart people shouting.

mlsu 3y ago
Human systems. They are protected by human systems. Human beings are the ones who press the big red button. We probably do agree. I'm not saying that this tech won't be used by bad actors. I already told my elderly relatives that if they haven't seen someone in person, they shouldn't talk on the phone. But what I'm talking about is categorically, fundamentally, not the eschaton!
- high_5 3y ago
  Yes, they are human indeed, but those humans cannot agree on whether the plug should or shouldn't be pulled.

ethanbond 3y ago

So your argument is that we need not worry because Sutskever and Hinton aren’t aware you can destroy data centers with water or static electricity?

skepticATX 3y ago
The same Hinton who was certain that radiology was solved? The same Sutskever who led OpenAI when they released a neutered GPT-2 because the full model was too dangerous? The problem with the AI safety argument is that it can be boiled down to: 1) Assume that we build an all powerful god machine 2) The all powerful god machine has the ability to exterminate humanity All of their work revolves around different ways that 2) might occur. But what I, and many other people, take issue with is whether 1) is even possible, and if it is, if we’re even remotely close to building it. No one ever explains how 1) will be achieved beyond vague handwaving, because no one knows how to build a god machine.
- deleted 3y ago
  [deleted]
- ethanbond 3y ago
  (1) will happen if the following is true: A. Intelligence is substrate independent I.e. there’s nothing mythically special about biological neurons that allow them to “be intelligent.” B. Intelligence is economically valuable C. We don’t destroy ourselves first e.g. via nukes or asteroid We know for a fact that you can produce human-level intelligence that runs on very, very little power, and you can train that intelligence up to human-level in the course of 10-50 years of navigating the real world. Why do we know that? Because evolution did it already. Now is there something so remarkable about the tissue we happen to have in our brain that means it is literally impossible to replicate this success? I can’t seem to find what it is. Is there some reason to believe that the human brain is the absolute upper limit of what type of intelligence is possible in this universe? I can’t seem to find why that would be the case. Obviously everyone who needs to agree on the commercial incentive already does agree on the commercial incentive (see: bottomless VC and soon govt funding), so that’s a moot point. So explain how if these things are true: it’s possible, there’s immense incentives, and we have time enough to do it, that we don’t end up accomplishing it?

nl 3y ago

> I can do something that no language model can do, and is showing no signs of being able to do any time soon: drive a car to the datacenter, turn the door handle, and unplug the rack. Steven Hawking was generally intelligent and struggled to do these things. He'd also struggle to move the goal posts for AGI as much as this. (edit: to clarify I think AI Safety concerns are fever dreams of people who don't get out in the real world. But I don't think that reflects on what AGI looks like)

mlsu 3y ago
I'm not concerned that Stephen Hawking will destroy all matter in the light cone*. Remember, if this conversation wasn't about eschatology, "AGI" would be just another synonym for "smart, useful machines." I didn't set the terms of discussion. * or prevent humanity's technological transformation into the divine. Choose your flavor.

tim333 3y ago

Nah. We'll have robots for that.