4 ms·

> 90% confidence [0] that creating a self-sustaining nuclear reaction with Uranium was impossible But the fact that Fermi was doing such a calculation in the f

> 90% confidence [0] that creating a self-sustaining nuclear reaction with Uranium was impossible But the fact that Fermi was doing such a calculation in the first place proves that we knew in principle how a fission weapon could work, even if we didn't know "how far off [they] were". As soon as we figured out the moon was just a rock 240,000 miles away, we knew in principle we could go there, even if we didn't know how far off that would be. By contrast, we don't know what consciousness or intelligence even is. A child could define what walking on the moon is, and Fermi was able to define a self-sustaining nuclear reaction as soon as he learned what nuclear reactions were. What even is the definition of consciousness?

DuskStar 7y ago
None of this really counters the core point - that we don't know how long it will be before we have AGI. Is there some way to define consciousness that will be discovered in the future that makes the problem possible?
- laughinghan 7y ago
  Your core point (and that of the MIRI article you linked to) is not just "we don't know". It's that the chance of being imminent and catastrophic is worth taking seriously. I am of course not saying you're wrong that "we don't know". We obviously don't know. It's possible, just like it's possible that we could discover cheap free energy (fusion?) tomorrow and then be in a post-scarcity utopia. But that's worth taking about as seriously as the possibility that we'll discover AGI tomorrow and be in a Terminator dystopia, or also a post-scarcity utopia. More importantly, it's a distraction from the very real, well-past-imminent problems that existing dumb AI has, such as the surveillance economy and misinformation. OpenAI, to their credit, does a good job of taking these existing problems quite seriously. They draw a strong contrast to MIRI's AI alarmism. Have you ever read idlewords? Best writing I know of on this subject: https://idlewords.com/talks/superintelligence.htm https://idlewords.com/talks/superintelligence.htm
acqq 7y ago
> as soon as we figured out the moon was just a rock 240,000 miles away, we knew in principle we could go there, even if we didn't know how far off that would be I have problems agreeing with that specific claim, knowing that both "the rock" and the distance were known to some ancient Greeks around 2200 years ago. https://en.wikipedia.org/wiki/Hipparchus https://en.wikipedia.org/wiki/Hipparchus Hipparchus estimated the distance to the Moon in the Earth radii to between 62 and 80 (depending on the method he used, as he intentionally used two different). Today's measurements are between 55 and 64.
- laughinghan 7y ago
  Holy shit, that is so impressive. They didn't even have Newton's law of gravity yet. Once we had Newton's law of gravity though, we knew the distance, radius, mass, and even surface gravity of the moon. Would you say it's fair to say that by then we knew in principle we could go there and walk there? (P.S. I assume you know this but the way you wrote your comment makes it seem like our measurements of lunar distance are nearly as inaccurate as Hipparchus's, when we actually know it down to the millimeter (thanks to retroreflectors placed by Apollo, actually). The wide variation from 55x to 64x Earth's radius is because it changes over the course of the moon's orbit, due to [edit: primarily its elliptical orbit, and only secondarily] the Sun and Jupiter's gravity.)
  - acqq 7y ago
    > The wide variation from 55x to 64x Earth's radius is because it changes over the course of the moon's orbit, due to the Sun and Jupiter's gravity I think you’re not only wrong but even Kepler and Newton already knew that better than you: https://en.m.wikipedia.org/wiki/Elliptic_orbit https://en.m.wikipedia.org/wiki/Elliptic_orbit “Strictly speaking, both bodies revolve around the same focus of the ellipse, the one closer to the more massive body, but when one body is significantly more massive, such as the sun in relation to the earth, the focus may be contained within the larger massing body, and thus the smaller is said to revolve around it.” But maybe you have some better information?
    - 7y ago

DuskStar 7y ago

None of this really counters the core point - that we don't know how long it will be before we have AGI. Is there some way to define consciousness that will be discovered in the future that makes the problem possible?

laughinghan 7y ago
Your core point (and that of the MIRI article you linked to) is not just "we don't know". It's that the chance of being imminent and catastrophic is worth taking seriously. I am of course not saying you're wrong that "we don't know". We obviously don't know. It's possible, just like it's possible that we could discover cheap free energy (fusion?) tomorrow and then be in a post-scarcity utopia. But that's worth taking about as seriously as the possibility that we'll discover AGI tomorrow and be in a Terminator dystopia, or also a post-scarcity utopia. More importantly, it's a distraction from the very real, well-past-imminent problems that existing dumb AI has, such as the surveillance economy and misinformation. OpenAI, to their credit, does a good job of taking these existing problems quite seriously. They draw a strong contrast to MIRI's AI alarmism. Have you ever read idlewords? Best writing I know of on this subject: https://idlewords.com/talks/superintelligence.htm https://idlewords.com/talks/superintelligence.htm

acqq 7y ago

> as soon as we figured out the moon was just a rock 240,000 miles away, we knew in principle we could go there, even if we didn't know how far off that would be I have problems agreeing with that specific claim, knowing that both "the rock" and the distance were known to some ancient Greeks around 2200 years ago. https://en.wikipedia.org/wiki/Hipparchus https://en.wikipedia.org/wiki/Hipparchus Hipparchus estimated the distance to the Moon in the Earth radii to between 62 and 80 (depending on the method he used, as he intentionally used two different). Today's measurements are between 55 and 64.

laughinghan 7y ago
Holy shit, that is so impressive. They didn't even have Newton's law of gravity yet. Once we had Newton's law of gravity though, we knew the distance, radius, mass, and even surface gravity of the moon. Would you say it's fair to say that by then we knew in principle we could go there and walk there? (P.S. I assume you know this but the way you wrote your comment makes it seem like our measurements of lunar distance are nearly as inaccurate as Hipparchus's, when we actually know it down to the millimeter (thanks to retroreflectors placed by Apollo, actually). The wide variation from 55x to 64x Earth's radius is because it changes over the course of the moon's orbit, due to [edit: primarily its elliptical orbit, and only secondarily] the Sun and Jupiter's gravity.)
- acqq 7y ago
  > The wide variation from 55x to 64x Earth's radius is because it changes over the course of the moon's orbit, due to the Sun and Jupiter's gravity I think you’re not only wrong but even Kepler and Newton already knew that better than you: https://en.m.wikipedia.org/wiki/Elliptic_orbit https://en.m.wikipedia.org/wiki/Elliptic_orbit “Strictly speaking, both bodies revolve around the same focus of the ellipse, the one closer to the more massive body, but when one body is significantly more massive, such as the sun in relation to the earth, the focus may be contained within the larger massing body, and thus the smaller is said to revolve around it.” But maybe you have some better information?
  - 7y ago