4 ms·

That's not completely fair. I just mean any AI with the capacity to self-improve. We might know how to specify goals for some of those AIs, but none of these ar

That's not completely fair. I just mean any AI with the capacity to self-improve. We might know how to specify goals for some of those AIs, but none of these are proved safe. And we have pretty strong arguments that any goal that's not proved safe is most likely unsafe, making everyone end up dead or worse. For example, if you make the AI "learn" a utility function about making cheeseburgers by using observations and reinforcement learning, and then the AI self-improves, you are most likely dead, because the learned utility function didn't include all possible caveats about not killing or torturing people to make cheeseburgers faster. And if you think you can keep applying negative reinforcement after the AI self-improves, think again.

eli_gottlieb 11y ago
>That's not completely fair. I just mean any AI with the capacity to self-improve. Yes, but you're still assuming that we're talking about an agent that can take decisions and act autonomously, rather than an inference engine that just processes data and spits out its inferences with zero autonomy whatsoever. The former is extremely, stupidly unsafe by default, so, logically, people probably won't try to build any such thing. They'll build the second sort of thing, which will mostly just be a more advanced version of today's statistical learning. Unless, of course, you're talking about ideological Singulatarians, who may well face legal sanction one of these days for deliberately trying to build the former agenty sort of thing as if that was a good idea. (Protip: If you want to talk "AI safety", we can do that on the site devoted to it, but out here in broader where nobody's damn fool enough to try to build agent-y "AI", mixing up realistic ML with the kind of agent-y "AI" you'd have to be blatantly suicidal to build is an abuse of terminology.) Besides which, "the capacity to self-improve" is actually a currently-open research problem. Currently. Once I work some things out, I've got something to report about that...
- cousin_it 11y ago
  Let me get this straight: 1) Your plan for AI safety is "no one will be stupid enough to build a self-improving AI". 2) You are currently working on self-improving AI. I'm sorry to say, but in my eyes you've just lost the right to criticize the LW/MIRI school of thought :-(
  - eli_gottlieb 11y ago
    >I'm sorry to say, but in my eyes you've just lost the right to criticize the LW/MIRI school of thought :-( You mean the one I belong to? Like I said: go on LW and talk about "AI" with assumed context. It's just out here in the rest of the world where you can't assume that everyone automatically has read the literature on AIXI/Goedel Machines/etc and considers "agenty" AI to be a real thing. >2) You are currently working on self-improving AI. Hell no! I'm working on logic and theorem-proving in the context of algorithmic information theory -- really just dicking around as a hobby. If you want "stable" self-improvement for your "AIs", you need that. It's also not, in and of itself, AI: it's logic, programming language theory, and computability theory. And if I get a result that holds up, which is an open if, I'd be happy to keep it the hell away from "AI" people. The main reason I don't consider alarmism warranted about "self-improving AI" (though I don't count any of FLI's letters as alarmism) is that I think of "an agenty AI" as something put together out of many distinct pieces. It's arranging the pieces into a whole and executing them that's unsafe, but also currently prohibitively unlikely to happen by accident. Naturalized induction and Vingean reflection wouldn't be open problems if self-improving "agenty AI" was so easy it could happen by accident. I fully agree that one does not build a self-improving agenty AI under basically any circumstances, ever, even if there's quite a lot of guns to your head and various other unlikely and terrible things have happened, as the research literature stands right now.
  - eli_gottlieb 11y ago
    By the way, good luck on your presentation to LW-Tel-Aviv tomorrow evening!

eli_gottlieb 11y ago

>That's not completely fair. I just mean any AI with the capacity to self-improve. Yes, but you're still assuming that we're talking about an agent that can take decisions and act autonomously, rather than an inference engine that just processes data and spits out its inferences with zero autonomy whatsoever. The former is extremely, stupidly unsafe by default, so, logically, people probably won't try to build any such thing. They'll build the second sort of thing, which will mostly just be a more advanced version of today's statistical learning. Unless, of course, you're talking about ideological Singulatarians, who may well face legal sanction one of these days for deliberately trying to build the former agenty sort of thing as if that was a good idea. (Protip: If you want to talk "AI safety", we can do that on the site devoted to it, but out here in broader where nobody's damn fool enough to try to build agent-y "AI", mixing up realistic ML with the kind of agent-y "AI" you'd have to be blatantly suicidal to build is an abuse of terminology.) Besides which, "the capacity to self-improve" is actually a currently-open research problem. Currently. Once I work some things out, I've got something to report about that...

cousin_it 11y ago
Let me get this straight: 1) Your plan for AI safety is "no one will be stupid enough to build a self-improving AI". 2) You are currently working on self-improving AI. I'm sorry to say, but in my eyes you've just lost the right to criticize the LW/MIRI school of thought :-(
- eli_gottlieb 11y ago
  >I'm sorry to say, but in my eyes you've just lost the right to criticize the LW/MIRI school of thought :-( You mean the one I belong to? Like I said: go on LW and talk about "AI" with assumed context. It's just out here in the rest of the world where you can't assume that everyone automatically has read the literature on AIXI/Goedel Machines/etc and considers "agenty" AI to be a real thing. >2) You are currently working on self-improving AI. Hell no! I'm working on logic and theorem-proving in the context of algorithmic information theory -- really just dicking around as a hobby. If you want "stable" self-improvement for your "AIs", you need that. It's also not, in and of itself, AI: it's logic, programming language theory, and computability theory. And if I get a result that holds up, which is an open if, I'd be happy to keep it the hell away from "AI" people. The main reason I don't consider alarmism warranted about "self-improving AI" (though I don't count any of FLI's letters as alarmism) is that I think of "an agenty AI" as something put together out of many distinct pieces. It's arranging the pieces into a whole and executing them that's unsafe, but also currently prohibitively unlikely to happen by accident. Naturalized induction and Vingean reflection wouldn't be open problems if self-improving "agenty AI" was so easy it could happen by accident. I fully agree that one does not build a self-improving agenty AI under basically any circumstances, ever, even if there's quite a lot of guns to your head and various other unlikely and terrible things have happened, as the research literature stands right now.
- eli_gottlieb 11y ago
  By the way, good luck on your presentation to LW-Tel-Aviv tomorrow evening!