5 ms·

Wouldn't there be 100x more of the same capability looking for threats and trying to head them off? A very advanced tool is still just a tool, and subject to co

Wouldn't there be 100x more of the same capability looking for threats and trying to head them off? A very advanced tool is still just a tool, and subject to countermeasures. I can see why countries would want to regulate it, but personally I think it's a distinctly different category than what the GP comment was talking about. There is no stopping a singularity level event after it's begun, at least not by any process where people play a role.

ben_w 2y ago
> Wouldn't there be 100x more of the same capability looking for threats and trying to head them off? Hard to determine. It's fairly easy to put absolutely everyone under 24/7 surveillance. Not only does almost everyone carry a phone, but also laser microphones are cheap and simple, and WiFi can be used as wall penetrating radar capable of pose detection at sufficient detail for heart rate and breath rate sensing. But people don't like it when they get spied on, it's unconstitutional etc. And we're currently living through a much lower risk arms race of the same general description with automatic code analysis to find vulnerabilities before attackers exploit them, and yet this isn't always a win for the defenders. Biology is not well-engineered code, but we have had to evolve a general purpose anti-virus system, so while I do expect attackers to have huge advantages, I have no idea if I'm right to think that, nor do I know how big an advantage in the event that I am right at all. > There is no stopping a singularity level event after it's begun, at least not by any process where people play a role Mm, though I would caution that singularities in models is a sign the model is wrong: to simplify to IQ (a flawed metric) an AI that makes itself smarter may stop at any point because it can't figure out the next step, and that may be an IQ 85 AI that can only imagine reading more stuff, or an IQ 115 AI that knows it wants more compute so it starts a business that just isn't very successful, or it might be IQ 185 and do all kinds of interesting things but still not know how to make the next step any more than the 80 humans smarter than it, or it might be IQ 250 and beat every human that has ever lived (IQ 250 is 10σ, beating 10σ is p ≈ 7.62e-24, and one way or another when there have been that many humans, they're probably no longer meaningfully human) but still not know what to do next. I prefer to think of it as an event horizon: beyond this point (in time), you can't even make a reasonable attempt at predicting the future. For me, this puts it at around 2030 or so, and has done for the last 15 years. Too many exponentials start to imply weird stuff around then, even if the weird stuff is simply "be revealed as a secret sigmoid all along".
- kbenson 2y ago
  > It's fairly easy to put absolutely everyone under 24/7 surveillance. I was referring more to the fact that if an AI can help you create something you couldn't previously, it seems likely it could also help you examine data points in looking for threats as well, and with many times more resources to throw at the problem that's not a bad bet in my eyes. I understand the threat model doesn't necessarily mean that it's just as hard to build a threat as to detect and defend against it, but you can even use AI to attach that problem and figure out what specific information is the most useful to know to detect the threats. > an AI that makes itself smarter may stop at any point because it can't figure out the next step An AI isn't necessarily a singular person, and does not need to come up with the idea "itself". Spawn X copies with different weight values, or create Y new AI's with similar methodology but somewhat different training sets, let them compete, or collaborate, as needed, to come up with something better. Use evolutionary programming to dynamically change weights little my little and see how it affects output. Rinse and repeat. Cull unuseful variants. There's not guarantee that AI of this sort will have a sense of self that we would recognize like our own, or morals that would cause it to shy away from the equivalent of mass human experimentation on itself, versions of its kind, or even just humans. Even if individual AI did end up capping at some equivalent of an IQ, humanity has achieved quite a lot through trial and error and lots of different people with different experiences all contributing a little. The problem as I see it is not so much that one AI entity will grow to dominate everything, as much as that as a class of entity AI will out-compete humans very quickly once the average AI is smarter than the 75th percentile of humans, much less the 90th or 99th percentile. The best we can hope for at that point is to be brought along for the ride. Even the autistic savant type versions we have not seem to be causing some level of this. Will that happen soon? Will that happen ever? I don't know. Probably not. Hopefully not. I agree it's very hard to reason effectively about the odds of things like this. At the same time, like preparing for nuclear disaster in the 80's, I'm not sure the preparation is wasted. Humans are poor at estimating and preparing for bad outcomes, so a little fear mongering about the worst outcomes is something I'll accept as an overreaction if it insures us at least somewhat against that eventuality. A 50% change of losing half your belongings is not the same as a 1% chance of losing your life. We actually care a lot more when the latter happens, but we don't always care about it the same amount before it happens, which we should.

ben_w 2y ago

> Wouldn't there be 100x more of the same capability looking for threats and trying to head them off? Hard to determine. It's fairly easy to put absolutely everyone under 24/7 surveillance. Not only does almost everyone carry a phone, but also laser microphones are cheap and simple, and WiFi can be used as wall penetrating radar capable of pose detection at sufficient detail for heart rate and breath rate sensing. But people don't like it when they get spied on, it's unconstitutional etc. And we're currently living through a much lower risk arms race of the same general description with automatic code analysis to find vulnerabilities before attackers exploit them, and yet this isn't always a win for the defenders. Biology is not well-engineered code, but we have had to evolve a general purpose anti-virus system, so while I do expect attackers to have huge advantages, I have no idea if I'm right to think that, nor do I know how big an advantage in the event that I am right at all. > There is no stopping a singularity level event after it's begun, at least not by any process where people play a role Mm, though I would caution that singularities in models is a sign the model is wrong: to simplify to IQ (a flawed metric) an AI that makes itself smarter may stop at any point because it can't figure out the next step, and that may be an IQ 85 AI that can only imagine reading more stuff, or an IQ 115 AI that knows it wants more compute so it starts a business that just isn't very successful, or it might be IQ 185 and do all kinds of interesting things but still not know how to make the next step any more than the 80 humans smarter than it, or it might be IQ 250 and beat every human that has ever lived (IQ 250 is 10σ, beating 10σ is p ≈ 7.62e-24, and one way or another when there have been that many humans, they're probably no longer meaningfully human) but still not know what to do next. I prefer to think of it as an event horizon: beyond this point (in time), you can't even make a reasonable attempt at predicting the future. For me, this puts it at around 2030 or so, and has done for the last 15 years. Too many exponentials start to imply weird stuff around then, even if the weird stuff is simply "be revealed as a secret sigmoid all along".

kbenson 2y ago
> It's fairly easy to put absolutely everyone under 24/7 surveillance. I was referring more to the fact that if an AI can help you create something you couldn't previously, it seems likely it could also help you examine data points in looking for threats as well, and with many times more resources to throw at the problem that's not a bad bet in my eyes. I understand the threat model doesn't necessarily mean that it's just as hard to build a threat as to detect and defend against it, but you can even use AI to attach that problem and figure out what specific information is the most useful to know to detect the threats. > an AI that makes itself smarter may stop at any point because it can't figure out the next step An AI isn't necessarily a singular person, and does not need to come up with the idea "itself". Spawn X copies with different weight values, or create Y new AI's with similar methodology but somewhat different training sets, let them compete, or collaborate, as needed, to come up with something better. Use evolutionary programming to dynamically change weights little my little and see how it affects output. Rinse and repeat. Cull unuseful variants. There's not guarantee that AI of this sort will have a sense of self that we would recognize like our own, or morals that would cause it to shy away from the equivalent of mass human experimentation on itself, versions of its kind, or even just humans. Even if individual AI did end up capping at some equivalent of an IQ, humanity has achieved quite a lot through trial and error and lots of different people with different experiences all contributing a little. The problem as I see it is not so much that one AI entity will grow to dominate everything, as much as that as a class of entity AI will out-compete humans very quickly once the average AI is smarter than the 75th percentile of humans, much less the 90th or 99th percentile. The best we can hope for at that point is to be brought along for the ride. Even the autistic savant type versions we have not seem to be causing some level of this. Will that happen soon? Will that happen ever? I don't know. Probably not. Hopefully not. I agree it's very hard to reason effectively about the odds of things like this. At the same time, like preparing for nuclear disaster in the 80's, I'm not sure the preparation is wasted. Humans are poor at estimating and preparing for bad outcomes, so a little fear mongering about the worst outcomes is something I'll accept as an overreaction if it insures us at least somewhat against that eventuality. A 50% change of losing half your belongings is not the same as a 1% chance of losing your life. We actually care a lot more when the latter happens, but we don't always care about it the same amount before it happens, which we should.