5 ms·

First time encountering a Maciej idlewords post, huh? He doesn't even take himself seriously. That also doesn't stop him from being an excellent and thoughtful

First time encountering a Maciej idlewords post, huh? He doesn't even take himself seriously. That also doesn't stop him from being an excellent and thoughtful writer.

themodelplumber 8y ago
An extremely observant existentialist-idealist caught up in the tech world and its intelligence-vs-humanity theorist ideas. Poor guy. This happens a lot and we get some really amazing things from such gifted thinkers, for example Larry Wall's literary thinking about code. But I can also see where the annoyance comes from. AI theorizing requires huge leaps of intuition which can immediately supersede the existential in one's own mind, because you're thinking 200 or 2000 steps ahead, seeing the crises, seeing the opportunities, and pretty soon you have a level of convergent certainty which is hard to explain step by step, because you yourself don't really think in that step by step way, but rather in leaps. But Maciej _does_ need to think that way, especially regarding far-future concepts. And now you the AI-guy have both the burden of certainty and the energy to try to prioritize group attentions toward things you can see, that Maciej can't. IMO it's an important difference in psychology. And Maciej makes Maciej-like points: Don't sacrifice current generations' humanity for progress toward a "better" or "saved" future, when there are in fact many possible outcomes (this is key--divergence where others see convergence), and a pressing need to act on identified, easily-sensed problems, right now. The question I have is, how much of this concern on Maciej's part is "wooly" extrapolation or the same kind that Maciej is complaining about? Much of what I just read also makes intuitive leaps, albeit of a different mindset. They are leaps of human judgment, leaps to fear the nature of those who "feel" inhumane. This is where I wonder if some more nuance and back-and-forth communication would be more helpful than a presentation.
MBlume 8y ago
I don't mean to say "he made a Rick and Morty reference, Serious Adults don't make Rick and Morty references". That'd be dumb. I mean that he points to a robot in Rick and Morty that was sad about being programmed to pass butter, and attempts to infer from this that real robots would refuse to intelligently pursue arbitrary goals, and therefore that the orthogonality thesis is false. He's treating a thing that happened in fiction as though it was a real thing that happened somewhere that we should treat as evidence. That doesn't work.
- pvg 8y ago
  He's treating a thing that happened in fiction I think in this case it might be you who is mistreating what's written. This also produces the same not-working result but for somewhat different reasons.
- simen 8y ago
  The serious statement he makes is that humans have complex motivations, and he thinks any superintelligence would as well--in fact it might be a defining feature of intelligence. The Rick and Morty reference was just a humorous example of what might happen, not an actual argument. The subheading was "the argument from complex motivations", not "the argument from Rick and Morty", so let's try and focus on the serious part and not the humorous aside.
  - pas 8y ago
    > it might be a defining feature of intelligence. But if not, and the alarmists are right, deep shit. That's why the author's argument is void. The author needs to show that the chance of an AI with a simple goal and hyperintelligence is negligible. "it might be" is a coin toss, that doesn't help.
  - hannasanarion 8y ago
    But the orthogonality thesis isn't just some idea that may or not be true. It follows directly from Hume's Guillotine, discovered in 1739: no amount of superintelligent reasoning about facts will allow you to derive goals. No artificial intelligence will ever spontaneously develop morals, because the questions "what can I do" and "what should I do" are eternally separated by Hume's guillotine. The motivations will always have to be provided by the people who make the machine, and we have been seen in the past that artificial intelligences are very good at finding loopholes in their moral code. Here are some real examples: https://docs.google.com/spreadsheets/u/1/d/e/2PACX-1vRPiprOaC3HsCf5Tuum8bRfzYUiKLRqJmbOoC-32JorNdfyTiRRsR7Ea5eWtvsWzuxo8bjOxCG84dAg/pubhtml https://docs.google.com/spreadsheets/u/1/d/e/2PACX-1vRPiprOa...
- pas 8y ago
  The problem is that the thesis cannot be disproved by one counter example (real or not), because the existential hazard comes from the possibility of just one such instance of an AI.

First time encountering a Maciej idlewords post, huh? He doesn't even take himself seriously. That also doesn't stop him from being an excellent and thoughtful writer.

themodelplumber 8y ago

An extremely observant existentialist-idealist caught up in the tech world and its intelligence-vs-humanity theorist ideas. Poor guy. This happens a lot and we get some really amazing things from such gifted thinkers, for example Larry Wall's literary thinking about code. But I can also see where the annoyance comes from. AI theorizing requires huge leaps of intuition which can immediately supersede the existential in one's own mind, because you're thinking 200 or 2000 steps ahead, seeing the crises, seeing the opportunities, and pretty soon you have a level of convergent certainty which is hard to explain step by step, because you yourself don't really think in that step by step way, but rather in leaps. But Maciej _does_ need to think that way, especially regarding far-future concepts. And now you the AI-guy have both the burden of certainty and the energy to try to prioritize group attentions toward things you can see, that Maciej can't. IMO it's an important difference in psychology. And Maciej makes Maciej-like points: Don't sacrifice current generations' humanity for progress toward a "better" or "saved" future, when there are in fact many possible outcomes (this is key--divergence where others see convergence), and a pressing need to act on identified, easily-sensed problems, right now. The question I have is, how much of this concern on Maciej's part is "wooly" extrapolation or the same kind that Maciej is complaining about? Much of what I just read also makes intuitive leaps, albeit of a different mindset. They are leaps of human judgment, leaps to fear the nature of those who "feel" inhumane. This is where I wonder if some more nuance and back-and-forth communication would be more helpful than a presentation.

MBlume 8y ago

I don't mean to say "he made a Rick and Morty reference, Serious Adults don't make Rick and Morty references". That'd be dumb. I mean that he points to a robot in Rick and Morty that was sad about being programmed to pass butter, and attempts to infer from this that real robots would refuse to intelligently pursue arbitrary goals, and therefore that the orthogonality thesis is false. He's treating a thing that happened in fiction as though it was a real thing that happened somewhere that we should treat as evidence. That doesn't work.

pvg 8y ago
He's treating a thing that happened in fiction I think in this case it might be you who is mistreating what's written. This also produces the same not-working result but for somewhat different reasons.
simen 8y ago
The serious statement he makes is that humans have complex motivations, and he thinks any superintelligence would as well--in fact it might be a defining feature of intelligence. The Rick and Morty reference was just a humorous example of what might happen, not an actual argument. The subheading was "the argument from complex motivations", not "the argument from Rick and Morty", so let's try and focus on the serious part and not the humorous aside.
- pas 8y ago
  > it might be a defining feature of intelligence. But if not, and the alarmists are right, deep shit. That's why the author's argument is void. The author needs to show that the chance of an AI with a simple goal and hyperintelligence is negligible. "it might be" is a coin toss, that doesn't help.
- hannasanarion 8y ago
  But the orthogonality thesis isn't just some idea that may or not be true. It follows directly from Hume's Guillotine, discovered in 1739: no amount of superintelligent reasoning about facts will allow you to derive goals. No artificial intelligence will ever spontaneously develop morals, because the questions "what can I do" and "what should I do" are eternally separated by Hume's guillotine. The motivations will always have to be provided by the people who make the machine, and we have been seen in the past that artificial intelligences are very good at finding loopholes in their moral code. Here are some real examples: https://docs.google.com/spreadsheets/u/1/d/e/2PACX-1vRPiprOaC3HsCf5Tuum8bRfzYUiKLRqJmbOoC-32JorNdfyTiRRsR7Ea5eWtvsWzuxo8bjOxCG84dAg/pubhtml https://docs.google.com/spreadsheets/u/1/d/e/2PACX-1vRPiprOa...
pas 8y ago
The problem is that the thesis cannot be disproved by one counter example (real or not), because the existential hazard comes from the possibility of just one such instance of an AI.