3 ms·

If we do achieve AGI through a setup which is not only neural network based, but where even the architecture by which the nodes are connected is beyond our unde

by drdeca 5y ago

If we do achieve AGI through a setup which is not only neural network based, but where even the architecture by which the nodes are connected is beyond our understanding, then, that sounds very bad for interpretability and therefore for safety.

mkaic 5y ago
Sure. I'm all for safety, but I also think that unsafe AGI, like the atom bomb, is an inevitability. It's not a matter of if, it's a matter of when we invent it and release it. The future is already murky but it's about to get a heck of a lot more so -- sentient AIs could be the thing that exalts humanity or the thing that destroys us. Break out the popcorn, because I'd wager we're going to see one of those outcomes in the next century.
riskneutral 5y ago
If we achieve AGI, then the AGI should be able to produce a formal verification proof of its own safety, similar to how humans use produce formal verification proofs for critical software using automated theorem provers today.
- Icko 5y ago
  Can you provide formal verification proof of your own safety?
- drdeca 5y ago
  Well, to have a formal proof of its own safety, we first would need a correct formal definition of what it means for it to be safe (or, rather, a formal definition of a condition which is truly a sufficient condition for it to be safe. It needn't be a necessary condition.) But, I suppose arriving at such a formal definition doesn't really require that the thing being described be particularly amenable to our understanding. However, I'm not sure that "if we ran it, if it is safe, it could prove itself to be safe" is sufficient to address the concerns. If it isn't safe, then running at all may spell doom, so, "if it is safe, it could demonstrate that after we turn it on" doesn't really address that, because it doesn't give us any assurance before we turn it on. Now, if we started with something with substantially sub-human overall intelligence, and which wasn't really agent-y (so that, if it was a little unsafe, it would at least not be catastrophically unsafe), but which was more equipped to formally prove things about itself and potential modifications of itself than humans are equipped to formally prove things about it, then we could task that thing with formally proving safety properties about itself, and of also proving safety properties about its successor, and do that before running its successor, and iterate this process to produce increasingly intelligent programs and perhaps also eventually agentic ones, while always having a safety proof of each before we run it.. But, I'm not really sure how plausible this route is? Like, even assuming we do reach AGI, safe or unsafe, I'm not sure this is a plausible route of getting there safely.
yusefnapora 5y ago
Well, we don't have anything like a coherent understanding of the architecture of biological general intelligence either, so at least this won't be a particularly new state of affairs.

by drdeca 5y ago

mkaic 5y ago

Sure. I'm all for safety, but I also think that unsafe AGI, like the atom bomb, is an inevitability. It's not a matter of if, it's a matter of when we invent it and release it. The future is already murky but it's about to get a heck of a lot more so -- sentient AIs could be the thing that exalts humanity or the thing that destroys us. Break out the popcorn, because I'd wager we're going to see one of those outcomes in the next century.

riskneutral 5y ago

If we achieve AGI, then the AGI should be able to produce a formal verification proof of its own safety, similar to how humans use produce formal verification proofs for critical software using automated theorem provers today.

Icko 5y ago
Can you provide formal verification proof of your own safety?
drdeca 5y ago
Well, to have a formal proof of its own safety, we first would need a correct formal definition of what it means for it to be safe (or, rather, a formal definition of a condition which is truly a sufficient condition for it to be safe. It needn't be a necessary condition.) But, I suppose arriving at such a formal definition doesn't really require that the thing being described be particularly amenable to our understanding. However, I'm not sure that "if we ran it, if it is safe, it could prove itself to be safe" is sufficient to address the concerns. If it isn't safe, then running at all may spell doom, so, "if it is safe, it could demonstrate that after we turn it on" doesn't really address that, because it doesn't give us any assurance before we turn it on. Now, if we started with something with substantially sub-human overall intelligence, and which wasn't really agent-y (so that, if it was a little unsafe, it would at least not be catastrophically unsafe), but which was more equipped to formally prove things about itself and potential modifications of itself than humans are equipped to formally prove things about it, then we could task that thing with formally proving safety properties about itself, and of also proving safety properties about its successor, and do that before running its successor, and iterate this process to produce increasingly intelligent programs and perhaps also eventually agentic ones, while always having a safety proof of each before we run it.. But, I'm not really sure how plausible this route is? Like, even assuming we do reach AGI, safe or unsafe, I'm not sure this is a plausible route of getting there safely.

yusefnapora 5y ago

Well, we don't have anything like a coherent understanding of the architecture of biological general intelligence either, so at least this won't be a particularly new state of affairs.