5 ms·

> SIAI is tackling the world’s most important task -- the task of shaping the Singularity. The task of averting human extinction. We aren’t the only people tac

by thunk 16y ago

> SIAI is tackling the world’s most important task -- the task of shaping the Singularity. The task of averting human extinction. We aren’t the only people tackling this, but the total set is frighteningly small. Forgive me, but this seems a little on the "self-important quackery" side. There are any number of threats to our species far more urgent and reality-based than xenocide by a weakly godlike post-singularity AI. I suppose it can't be completely ruled out any more than alien invasion or the Nemesis hypothesis (though I find it pretty far fetched given the state of AGI). But "the world's most important task"? C'mon.

apsec112 16y ago
"There are any number of threats to our species far more urgent and reality-based than xenocide by a weakly godlike post-singularity AI." We work on preventing those too. Some of the risks we work to prevent: - Deliberate misuse of nanotechnology - Nuclear holocaust - We’re living in a simulation and it gets shut down - Genetically engineered biological agent - Accidental misuse of nanotechnology (“gray goo”) - Physics disasters - Misguided world government or another static social equilibrium stops technological progress - “Dysgenic” pressures - Take-over by a transcending upload - Repressive totalitarian global regime - Our potential or even our core values are eroded by evolutionary development For more information, see http://www.nickbostrom.com/existential/risks.html http://www.nickbostrom.com/existential/risks.html.
- seiji 16y ago
  I've been working on getting our simulation terminated while attracting a transcedent to upload everybody (giving us a bridge to outside the soon-to-be-terminated simulation). Also, I'm waiting for SLAC to burn through their barrier and burrow through Palo Alto.
- westbywest 16y ago
  Industrial food production is quite fragile, and the world basically has yet to experience that production in any sort of "failure mode," e.g. global blight due to excessive monocultures. The probability of such a global failure occurring within the next 100 years strikes me as near 1.0, and thus possibly more pressing than any of the events listed above.
  - jessriedel 16y ago
    This isn't an existential risk.
    - indrax 16y ago
      If there is a significant die-off in industrial nations it could lead to a major delay in the singularity, perhaps making it impossible altogether. http://www.nickbostrom.com/existential/risks.html http://www.nickbostrom.com/existential/risks.html In these terms this could be a 'crunch'.
      - jessriedel 16y ago
        Importance-wise, delays in the singularity are trivial compared to a risk of it not happening at all. So suppose we are worried that a die-off in industrial nations will actually make it impossible. Isn't this risk (that there is a major catastrophic disaster for industrialized nations and this disaster prevents a singularity from ever taking place) much smaller than the risk that the singularity goes badly when it happens? If so, then this supports the spirit of my original comment: worrying about the singularity going well is more pressing than worrying about major shocks to civilization which do not completely wipe out humanity.
      - jessriedel 16y ago
        The badness of a delayed singularity is insignificant to the badness of it never happening. So let's consider the latter. Isn't the risk that a large industrial collapse happens and it prevents a singularity from ever taking place much smaller than the risk that the singularity goes badly? If so, then this agrees with the spirit of my comment: worrying about the singularity going well is much more pressing than worrying about catastrophes that do not immediately wipe out humanity.
- SkyMarshal 16y ago
  Have you contacted Nassim Taleb yet? 'Robustifying' society against such unpredictables is his speciality. http://fooledbyrandomness.com/ http://fooledbyrandomness.com/
- coffeemug 16y ago
  Before I state my criticism of your mission, let me say that I respect what you are doing, I think it's incredibly interesting, and I think it adds spice to the world that an organization like the Singularity Institute exists. I think the world is a better place for it, and I respect you very much for spending your time exploring these issues. That being said, I think the way you state your mission appears a little immature to me and many other people, and I think that very seriously detracts from your credibility. This is very important, because with more credibility you could accomplish many more things that make the world a better place. One of the most important things about scientific research is having a concrete, well defined way to attack a problem you're setting out to solve. People are working on quantum mechanics because there are lots of concrete things they can do and test, and people aren't working on antigravity because they don't know where to begin. Your research is from the "antigravity" category of problems, i.e. problems where we don't really have an attack. This doesn't mean working on these problems isn't useful, or that you should stop, or that you should spend your time doing something else. It does mean that your research is highly philosophical - it is very interesting intellectually, it has potential to attract great young minds to work on interesting things, and it will likely produce results that will get people thinking in new directions. This is always a good thing. But it also means that you are extremely unlikely to find practical solutions to any of these problems, and it seems that everyone is recognizing this other than you. When you present yourself as an organization that is working on practical problems, instead of intellectually interesting philosophical problems, it becomes very difficult for many intelligent people to take you seriously, and I think it significantly detracts from your goal. Once again, I think what you're doing is great, I wish more people were doing it, and I sincerely wish you luck and success. I am merely trying to show you how many intelligent people that would otherwise be interested in what you're doing view your organization. Please treat this comment as a constructive argument for changing your marketing. At the very least, think seriously about slightly adjusting the wording of your mission statement. It might get more of the right people interested, which will likely accelerate your work, and make all of us better off for it.
arundelo 16y ago
Here are some arguments for why AI is different that I find convincing: http://lesswrong.com/lw/qk/that_alien_message/ http://lesswrong.com/lw/qk/that_alien_message/ http://lesswrong.com/lw/rk/optimization_and_the_singularity/ http://lesswrong.com/lw/rk/optimization_and_the_singularity/
crystalis 16y ago
The nice thing about a post-singularity AI is that it also handles all of those other threats.
deleted 16y ago
[deleted]
- Retric 16y ago
  The core problems with singularity hypothesis makes the whole issue moot. A) An infinitely intelligent machine is not capable of breaking the laws of physics so they don't get god like powers. B) Intelligence and creativity are always bound by insanity. (You can always over fit any data set.) C) Sensors limit the acquisition of new information. (You can deduce the existence of quarks from looking at a web cam pointed at a brick.) D) While processing power has grown exponentially the cost of fabricating that technology has continued to increase faster than world GDP.
  - apsec112 16y ago
    http://yudkowsky.net/singularity/power http://yudkowsky.net/singularity/power http://lesswrong.com/lw/qk/that_alien_message/ http://lesswrong.com/lw/qk/that_alien_message/ http://som.csudh.edu/fac/lpress/netapps/hout/techprogress/kurzweil.gif http://som.csudh.edu/fac/lpress/netapps/hout/techprogress/ku...
  - obfuscate 16y ago
    > While processing power has grown exponentially the cost of fabricating that technology has continued to increase faster than world GDP. Note that "Singularity" doesn't always refer to exponential growth (http://yudkowsky.net/singularity/schools http://yudkowsky.net/singularity/schools); SIAI in particular focuses on the "intelligence explosion" definition.
  - smanek 16y ago
    A) But it could discover new laws of physics. i.e., A nuclear bomb would have looked like 'god like powers' before Einstein/Relativity. B) This is nonsensical ... C) You can deduce a ludicrous amount of information from relatively little sensor input. Humans don't extract a fraction of a percent of the maximum (in the information theoretic sense) information from our environment. See http://lesswrong.com/lw/qk/that_alien_message/ http://lesswrong.com/lw/qk/that_alien_message/ D) This is trivially, factually incorrect. A 3GHz P4 is cheaper today (in absolute or relative terms) than it was when it was brand new. A new Core 2 Duo will give you better performance/dollar than a P4 when it was new (which was better than a P3, which was better than a PII, etc).
obfuscate 16y ago
> There are any number of threats to our species far more urgent and reality-based than xenocide by a weakly godlike post-singularity AI. Is this belief based on your familiarity with and assessment of the best(1) arguments(2) for and against Singularity/AI risk? If not, then what? Do you think the idea is absurd enough to confidently dismiss without engaging, and if so, why? (1) http://lesswrong.com/lw/lw/reversed_stupidity_is_not_intelligence/ http://lesswrong.com/lw/lw/reversed_stupidity_is_not_intelli... (2) e.g. http://singinst.org/AIRisk.pdf http://singinst.org/AIRisk.pdf, http://wiki.lesswrong.com/wiki/The_Hanson-Yudkowsky_AI-Foom_Debate http://wiki.lesswrong.com/wiki/The_Hanson-Yudkowsky_AI-Foom_...
- thunk 16y ago
  I took issue with the claim that shaping the Singularity is "the world's most important task" when there are so many extinction-level threats that already exist. It seems arrogant and presumptuous to claim the number one spot for a hypothetical threat that one happens to be deeply involved with. Debating the likelihood of recursively self-improving AI is tedious, usually pointless, and often goes the way of political and religious debate, which could hint at its inclusion in the participants' self definition [1], and I don't have the time or energy to get involved right now. I apologize if I came off as dismissive -- I was frusterated over my initial point. If it matters to you, I certainly haven't completely written off recursively self-improving AI. I just find it extremely unlikely in the near-to-mid future, and think there are other more pressing issues that have the distinction of currently existing. [1] http://www.paulgraham.com/identity.html http://www.paulgraham.com/identity.html
  - indrax 16y ago
    Note that Shaping the Singularity was taken as the same task as averting human extinction. The talk of AI is secondary, it comes up because many of us think that AI can offer good solutions. In looking at AI, it also seems apparent that AI is itself an existential threat. Is shaping the singularity the same as averting human extinction? If the singularity itself poses existential risks (this seems obvious,) and if extinction is inevitable without a singularity (this seems likely,) then we need a singularity that is 'safe and soon'. (though the initial wording does seem to gloss over the need to not go extinct in modern ways.) Put another way: We can do the most work with the most powerful tools. There is a great need to use very powerful tools correctly. If we can even maybe influence the work done by highly powerful mid-singularity tools, we can accomplish far more 'correct work' than we could just using our modern tools. (Again, this does require that we survive long enough for the singularity to happen, and that you don't have a strong discount for future vs. present utility.) Personally, I think creating a super-tool which can only be used correctly is the best approach.

by thunk 16y ago

apsec112 16y ago

"There are any number of threats to our species far more urgent and reality-based than xenocide by a weakly godlike post-singularity AI." We work on preventing those too. Some of the risks we work to prevent: - Deliberate misuse of nanotechnology - Nuclear holocaust - We’re living in a simulation and it gets shut down - Genetically engineered biological agent - Accidental misuse of nanotechnology (“gray goo”) - Physics disasters - Misguided world government or another static social equilibrium stops technological progress - “Dysgenic” pressures - Take-over by a transcending upload - Repressive totalitarian global regime - Our potential or even our core values are eroded by evolutionary development For more information, see http://www.nickbostrom.com/existential/risks.html http://www.nickbostrom.com/existential/risks.html.

seiji 16y ago
I've been working on getting our simulation terminated while attracting a transcedent to upload everybody (giving us a bridge to outside the soon-to-be-terminated simulation). Also, I'm waiting for SLAC to burn through their barrier and burrow through Palo Alto.
westbywest 16y ago
Industrial food production is quite fragile, and the world basically has yet to experience that production in any sort of "failure mode," e.g. global blight due to excessive monocultures. The probability of such a global failure occurring within the next 100 years strikes me as near 1.0, and thus possibly more pressing than any of the events listed above.
- jessriedel 16y ago
  This isn't an existential risk.
  - indrax 16y ago
    If there is a significant die-off in industrial nations it could lead to a major delay in the singularity, perhaps making it impossible altogether. http://www.nickbostrom.com/existential/risks.html http://www.nickbostrom.com/existential/risks.html In these terms this could be a 'crunch'.
    - jessriedel 16y ago
      Importance-wise, delays in the singularity are trivial compared to a risk of it not happening at all. So suppose we are worried that a die-off in industrial nations will actually make it impossible. Isn't this risk (that there is a major catastrophic disaster for industrialized nations and this disaster prevents a singularity from ever taking place) much smaller than the risk that the singularity goes badly when it happens? If so, then this supports the spirit of my original comment: worrying about the singularity going well is more pressing than worrying about major shocks to civilization which do not completely wipe out humanity.
    - jessriedel 16y ago
      The badness of a delayed singularity is insignificant to the badness of it never happening. So let's consider the latter. Isn't the risk that a large industrial collapse happens and it prevents a singularity from ever taking place much smaller than the risk that the singularity goes badly? If so, then this agrees with the spirit of my comment: worrying about the singularity going well is much more pressing than worrying about catastrophes that do not immediately wipe out humanity.
SkyMarshal 16y ago
Have you contacted Nassim Taleb yet? 'Robustifying' society against such unpredictables is his speciality. http://fooledbyrandomness.com/ http://fooledbyrandomness.com/
coffeemug 16y ago
Before I state my criticism of your mission, let me say that I respect what you are doing, I think it's incredibly interesting, and I think it adds spice to the world that an organization like the Singularity Institute exists. I think the world is a better place for it, and I respect you very much for spending your time exploring these issues. That being said, I think the way you state your mission appears a little immature to me and many other people, and I think that very seriously detracts from your credibility. This is very important, because with more credibility you could accomplish many more things that make the world a better place. One of the most important things about scientific research is having a concrete, well defined way to attack a problem you're setting out to solve. People are working on quantum mechanics because there are lots of concrete things they can do and test, and people aren't working on antigravity because they don't know where to begin. Your research is from the "antigravity" category of problems, i.e. problems where we don't really have an attack. This doesn't mean working on these problems isn't useful, or that you should stop, or that you should spend your time doing something else. It does mean that your research is highly philosophical - it is very interesting intellectually, it has potential to attract great young minds to work on interesting things, and it will likely produce results that will get people thinking in new directions. This is always a good thing. But it also means that you are extremely unlikely to find practical solutions to any of these problems, and it seems that everyone is recognizing this other than you. When you present yourself as an organization that is working on practical problems, instead of intellectually interesting philosophical problems, it becomes very difficult for many intelligent people to take you seriously, and I think it significantly detracts from your goal. Once again, I think what you're doing is great, I wish more people were doing it, and I sincerely wish you luck and success. I am merely trying to show you how many intelligent people that would otherwise be interested in what you're doing view your organization. Please treat this comment as a constructive argument for changing your marketing. At the very least, think seriously about slightly adjusting the wording of your mission statement. It might get more of the right people interested, which will likely accelerate your work, and make all of us better off for it.

arundelo 16y ago

Here are some arguments for why AI is different that I find convincing: http://lesswrong.com/lw/qk/that_alien_message/ http://lesswrong.com/lw/qk/that_alien_message/ http://lesswrong.com/lw/rk/optimization_and_the_singularity/ http://lesswrong.com/lw/rk/optimization_and_the_singularity/

crystalis 16y ago

The nice thing about a post-singularity AI is that it also handles all of those other threats.

deleted 16y ago

[deleted]

Retric 16y ago
The core problems with singularity hypothesis makes the whole issue moot. A) An infinitely intelligent machine is not capable of breaking the laws of physics so they don't get god like powers. B) Intelligence and creativity are always bound by insanity. (You can always over fit any data set.) C) Sensors limit the acquisition of new information. (You can deduce the existence of quarks from looking at a web cam pointed at a brick.) D) While processing power has grown exponentially the cost of fabricating that technology has continued to increase faster than world GDP.
- apsec112 16y ago
  http://yudkowsky.net/singularity/power http://yudkowsky.net/singularity/power http://lesswrong.com/lw/qk/that_alien_message/ http://lesswrong.com/lw/qk/that_alien_message/ http://som.csudh.edu/fac/lpress/netapps/hout/techprogress/kurzweil.gif http://som.csudh.edu/fac/lpress/netapps/hout/techprogress/ku...
- obfuscate 16y ago
  > While processing power has grown exponentially the cost of fabricating that technology has continued to increase faster than world GDP. Note that "Singularity" doesn't always refer to exponential growth (http://yudkowsky.net/singularity/schools http://yudkowsky.net/singularity/schools); SIAI in particular focuses on the "intelligence explosion" definition.
- smanek 16y ago
  A) But it could discover new laws of physics. i.e., A nuclear bomb would have looked like 'god like powers' before Einstein/Relativity. B) This is nonsensical ... C) You can deduce a ludicrous amount of information from relatively little sensor input. Humans don't extract a fraction of a percent of the maximum (in the information theoretic sense) information from our environment. See http://lesswrong.com/lw/qk/that_alien_message/ http://lesswrong.com/lw/qk/that_alien_message/ D) This is trivially, factually incorrect. A 3GHz P4 is cheaper today (in absolute or relative terms) than it was when it was brand new. A new Core 2 Duo will give you better performance/dollar than a P4 when it was new (which was better than a P3, which was better than a PII, etc).

obfuscate 16y ago

> There are any number of threats to our species far more urgent and reality-based than xenocide by a weakly godlike post-singularity AI. Is this belief based on your familiarity with and assessment of the best(1) arguments(2) for and against Singularity/AI risk? If not, then what? Do you think the idea is absurd enough to confidently dismiss without engaging, and if so, why? (1) http://lesswrong.com/lw/lw/reversed_stupidity_is_not_intelligence/ http://lesswrong.com/lw/lw/reversed_stupidity_is_not_intelli... (2) e.g. http://singinst.org/AIRisk.pdf http://singinst.org/AIRisk.pdf, http://wiki.lesswrong.com/wiki/The_Hanson-Yudkowsky_AI-Foom_Debate http://wiki.lesswrong.com/wiki/The_Hanson-Yudkowsky_AI-Foom_...

thunk 16y ago
I took issue with the claim that shaping the Singularity is "the world's most important task" when there are so many extinction-level threats that already exist. It seems arrogant and presumptuous to claim the number one spot for a hypothetical threat that one happens to be deeply involved with. Debating the likelihood of recursively self-improving AI is tedious, usually pointless, and often goes the way of political and religious debate, which could hint at its inclusion in the participants' self definition [1], and I don't have the time or energy to get involved right now. I apologize if I came off as dismissive -- I was frusterated over my initial point. If it matters to you, I certainly haven't completely written off recursively self-improving AI. I just find it extremely unlikely in the near-to-mid future, and think there are other more pressing issues that have the distinction of currently existing. [1] http://www.paulgraham.com/identity.html http://www.paulgraham.com/identity.html
- indrax 16y ago
  Note that Shaping the Singularity was taken as the same task as averting human extinction. The talk of AI is secondary, it comes up because many of us think that AI can offer good solutions. In looking at AI, it also seems apparent that AI is itself an existential threat. Is shaping the singularity the same as averting human extinction? If the singularity itself poses existential risks (this seems obvious,) and if extinction is inevitable without a singularity (this seems likely,) then we need a singularity that is 'safe and soon'. (though the initial wording does seem to gloss over the need to not go extinct in modern ways.) Put another way: We can do the most work with the most powerful tools. There is a great need to use very powerful tools correctly. If we can even maybe influence the work done by highly powerful mid-singularity tools, we can accomplish far more 'correct work' than we could just using our modern tools. (Again, this does require that we survive long enough for the singularity to happen, and that you don't have a strong discount for future vs. present utility.) Personally, I think creating a super-tool which can only be used correctly is the best approach.