4 ms·

>What if someone builds a supermassive teapot in Earth's orbit and it breaks free and comes hurtling towards us? Ok, that seems ridiculous and infeasible and n

>What if someone builds a supermassive teapot in Earth's orbit and it breaks free and comes hurtling towards us? Ok, that seems ridiculous and infeasible and nothing that has happened so far indicates that this is a real threat. >Yes, but it would be so catastrophic to humanity, that we must take it seriously, regardless of how improbable it is! This is the AGI debate, in my view. If we're picking different extremely improbable events to get worked up about, why stop at AI?

circuit10 2y ago
It’s arguable whether it’s actually improbable. Many people, including (and seemingly especially) experts think it’s likely or even inevitable. This video might be interesting to see some arguments for why people think it’s important: https://youtu.be/9i1WlcCudpU https://youtu.be/9i1WlcCudpU
- JKCalhoun 2y ago
  I feel like we'll know right away though. We can then act right away. I'm not sure what you would do now that is different because you are anticipating such a thing.
  - circuit10 2y ago
    It would be far too late by that point, as it can already take measures to stop us. This is a difficult problem and we really should have started putting a lot more effort into this a lot earlier. Of course we can’t test a superintelligence directly yet, but we should do as much theoretical work as we can Actually the video addresses this point, the comparison it makes is waiting until we’re already on Mars before we start thinking about spacesuits and airlocks
- Aloisius 2y ago
  Inevitable? Sure assuming we don't extinct first. In the near future? No. This is just another in a long line of technology panics. Unfortunately, there always seems to be some "concerned" experts that are both overly optimistic about the speed of technological progress and overly pessimistic about where that progress will lead adding fuel to the fire. Eventually some other new technology will incite a new panic and this one will become another footnote in history like the fears over grey goo or genetically engineered superhumans.
  - circuit10 2y ago
    What about when people pointed out problems like leaded fuel and climate change and we didn’t take action for far too long? We shouldn’t dismiss it just because of things it sounds similar to. In fact you could always dismiss any concern that could lead to human extinction like that, because if there was an existing example of human extinction then we wouldn’t be around to talk about it; that doesn’t mean it can’t happen.
- throwaway115 2y ago
  I know someone who fancies himself an expert, and indeed, by comparison to the average tech worker, he is an expert in LLMs and AI. He vehemently believes AGI is coming this year. At least, that's what he said last year. I suppose he will probably feel sheepish if I bring it up now that we're at month 6. My point is that the only thing that seems to convince the pro-AGI crowd is having them make specific predictions, waiting until the deadline, then asking them why they didn't happen yet.
richardw 2y ago
Because an increasing amount of capital and effort is going towards improving the chances we eventually achieve AGI. We might hit another winter where the party stops, but right now a lot of stock prices are encouraging more research. That increases the odds. Assuming we never get there is a bet against human ingenuity.
agucova 2y ago
Agreeing with circuit10's comments, I don't think many proponents of AI Safety are doing so through Pascal wagers. People differ a lot in their assessment of how likely certain risks are, but people I know working on AI Safety tend to place numbers between 1% to 30% of a global catastrophe associated with AI in the next 20 years [1]. I don't think this is negligible, nor analogous to your supermassive teapot example. [1]: You might disagree with this assessment, but my point is that this is not where most people are arguing from.
hehdhdjehehegwv 2y ago
It’s not all or nothing. I’d be surprised if we don’t see autonomous quad copters in eastern Ukraine dropping bombs on soldiers by end of ‘25. The parts and models are off the shelf now, it just needs the first person desperate enough to release a swarm of killcopters with cheap ML classifiers to make it a reality. Curious how people will feel after the first autonomous swarm becomes widely known, because it will be replicated widely.
Angostura 2y ago
Is there a company out there building large orbital teapots? If so, you analogy is a good one. But I haven't heard of said company

circuit10 2y ago

It’s arguable whether it’s actually improbable. Many people, including (and seemingly especially) experts think it’s likely or even inevitable. This video might be interesting to see some arguments for why people think it’s important: https://youtu.be/9i1WlcCudpU https://youtu.be/9i1WlcCudpU

JKCalhoun 2y ago
I feel like we'll know right away though. We can then act right away. I'm not sure what you would do now that is different because you are anticipating such a thing.
- circuit10 2y ago
  It would be far too late by that point, as it can already take measures to stop us. This is a difficult problem and we really should have started putting a lot more effort into this a lot earlier. Of course we can’t test a superintelligence directly yet, but we should do as much theoretical work as we can Actually the video addresses this point, the comparison it makes is waiting until we’re already on Mars before we start thinking about spacesuits and airlocks
Aloisius 2y ago
Inevitable? Sure assuming we don't extinct first. In the near future? No. This is just another in a long line of technology panics. Unfortunately, there always seems to be some "concerned" experts that are both overly optimistic about the speed of technological progress and overly pessimistic about where that progress will lead adding fuel to the fire. Eventually some other new technology will incite a new panic and this one will become another footnote in history like the fears over grey goo or genetically engineered superhumans.
- circuit10 2y ago
  What about when people pointed out problems like leaded fuel and climate change and we didn’t take action for far too long? We shouldn’t dismiss it just because of things it sounds similar to. In fact you could always dismiss any concern that could lead to human extinction like that, because if there was an existing example of human extinction then we wouldn’t be around to talk about it; that doesn’t mean it can’t happen.
throwaway115 2y ago
I know someone who fancies himself an expert, and indeed, by comparison to the average tech worker, he is an expert in LLMs and AI. He vehemently believes AGI is coming this year. At least, that's what he said last year. I suppose he will probably feel sheepish if I bring it up now that we're at month 6. My point is that the only thing that seems to convince the pro-AGI crowd is having them make specific predictions, waiting until the deadline, then asking them why they didn't happen yet.

richardw 2y ago

Because an increasing amount of capital and effort is going towards improving the chances we eventually achieve AGI. We might hit another winter where the party stops, but right now a lot of stock prices are encouraging more research. That increases the odds. Assuming we never get there is a bet against human ingenuity.

agucova 2y ago

Agreeing with circuit10's comments, I don't think many proponents of AI Safety are doing so through Pascal wagers. People differ a lot in their assessment of how likely certain risks are, but people I know working on AI Safety tend to place numbers between 1% to 30% of a global catastrophe associated with AI in the next 20 years [1]. I don't think this is negligible, nor analogous to your supermassive teapot example. [1]: You might disagree with this assessment, but my point is that this is not where most people are arguing from.

hehdhdjehehegwv 2y ago

It’s not all or nothing. I’d be surprised if we don’t see autonomous quad copters in eastern Ukraine dropping bombs on soldiers by end of ‘25. The parts and models are off the shelf now, it just needs the first person desperate enough to release a swarm of killcopters with cheap ML classifiers to make it a reality. Curious how people will feel after the first autonomous swarm becomes widely known, because it will be replicated widely.

Angostura 2y ago

Is there a company out there building large orbital teapots? If so, you analogy is a good one. But I haven't heard of said company