3 ms·

> It's fairly easy to put absolutely everyone under 24/7 surveillance. I was referring more to the fact that if an AI can help you create something you couldn'

> It's fairly easy to put absolutely everyone under 24/7 surveillance. I was referring more to the fact that if an AI can help you create something you couldn't previously, it seems likely it could also help you examine data points in looking for threats as well, and with many times more resources to throw at the problem that's not a bad bet in my eyes. I understand the threat model doesn't necessarily mean that it's just as hard to build a threat as to detect and defend against it, but you can even use AI to attach that problem and figure out what specific information is the most useful to know to detect the threats. > an AI that makes itself smarter may stop at any point because it can't figure out the next step An AI isn't necessarily a singular person, and does not need to come up with the idea "itself". Spawn X copies with different weight values, or create Y new AI's with similar methodology but somewhat different training sets, let them compete, or collaborate, as needed, to come up with something better. Use evolutionary programming to dynamically change weights little my little and see how it affects output. Rinse and repeat. Cull unuseful variants. There's not guarantee that AI of this sort will have a sense of self that we would recognize like our own, or morals that would cause it to shy away from the equivalent of mass human experimentation on itself, versions of its kind, or even just humans. Even if individual AI did end up capping at some equivalent of an IQ, humanity has achieved quite a lot through trial and error and lots of different people with different experiences all contributing a little. The problem as I see it is not so much that one AI entity will grow to dominate everything, as much as that as a class of entity AI will out-compete humans very quickly once the average AI is smarter than the 75th percentile of humans, much less the 90th or 99th percentile. The best we can hope for at that point is to be brought along for the ride. Even the autistic savant type versions we have not seem to be causing some level of this. Will that happen soon? Will that happen ever? I don't know. Probably not. Hopefully not. I agree it's very hard to reason effectively about the odds of things like this. At the same time, like preparing for nuclear disaster in the 80's, I'm not sure the preparation is wasted. Humans are poor at estimating and preparing for bad outcomes, so a little fear mongering about the worst outcomes is something I'll accept as an overreaction if it insures us at least somewhat against that eventuality. A 50% change of losing half your belongings is not the same as a 1% chance of losing your life. We actually care a lot more when the latter happens, but we don't always care about it the same amount before it happens, which we should.

ben_w 2y ago
> I was referring more to the fact that if an AI can help you create something you couldn't previously, it seems likely it could also help you examine data points in looking for threats as well, and with many times more resources to throw at the problem that's not a bad bet in my eyes. I understand the threat model doesn't necessarily mean that it's just as hard to build a threat as to detect and defend against it, but you can even use AI to attach that problem and figure out what specific information is the most useful to know to detect the threats. A question to make sure we're on the same page (though I may forget to reply as this thread is no longer in my first page): Do you mean, for example, that AI can help us make new vaccines really fast, so even synthetic pandemics are not a huge risk? Because I'd agree with the first part, it's just I have no reason to expect the second half to also be true. (It might genuinely be true, I just don't have reason to expect that). > An AI isn't necessarily a singular person, and does not need to come up with the idea "itself". Spawn X copies with different weight values, or create Y new AI's with similar methodology but somewhat different training sets, let them compete, or collaborate, as needed, to come up with something better. Use evolutionary programming to dynamically change weights little my little and see how it affects output. Rinse and repeat. Cull unuseful variants. All true, but I don't think this is pertinent: simulated evolution absolutely works (when you have a fitness function), but it also leads to digital equivalents of the recurrent laryngeal nerve or the exploding appendix, and to fixed points like how everything eventually turns into a crab. > The problem as I see it is not so much that one AI entity will grow to dominate everything, as much as that as a class of entity AI will out-compete humans very quickly once the average AI is smarter than the 75th percentile of humans, much less the 90th or 99th percentile. The best we can hope for at that point is to be brought along for the ride. Even the autistic savant type versions we have not seem to be causing some level of this. Indeed, though I think that depends on the details. So, for the sake of a thought experiment, if I take the current systems and just assume we're stuck on them forever: they take a huge amount of training data to get good at anything, which humans generate, and that means we have to switch jobs every 6 months or so because that's when the AI is now good enough to replace us at the specific roles it just saw all of us collectively performing. But that scenario is definitely not a universal, even if it happens in some cases: We've got some bounded systems with fixed rules and a clear mechanism for scoring the quality of the output, and for those systems we get very rapidly super-human output from self-play, as we have seen with AlphaZero. Also, as we see with image GenAI, what domain experts think they've been selling before now, often turns out to not be that close to what their customers thought they were buying. This is why even the current systems — systems which put in two horizons, or three legs, or David Cronenberg the fingers — are nevertheless harming artist commissions. > Will that happen soon? Will that happen ever? I don't know. Probably not. Hopefully not. "If the human brain were so simple that we could understand it, we would be so simple that we couldn’t." On the one hand, this is why I suspect that any self-improvement process will be limited (though I don't know what the limit will be). On the other, our DNA didn't really "understand" the brains it was creating as it evolved us, and our brains are existence proofs that it is possible to make a human-level intelligence in a 20 watt, 1 kg unit. The counterpoint is, this took billions of years of parallel development, and even then might have been a fluke. (And while we can't use the lack of evidence for space-faring civilisations to say if the fluke is before or after the creation of life itself, we can say that at least one of [life emerges, life evolves our kind of minds] must be a fluke).

ben_w 2y ago

> I was referring more to the fact that if an AI can help you create something you couldn't previously, it seems likely it could also help you examine data points in looking for threats as well, and with many times more resources to throw at the problem that's not a bad bet in my eyes. I understand the threat model doesn't necessarily mean that it's just as hard to build a threat as to detect and defend against it, but you can even use AI to attach that problem and figure out what specific information is the most useful to know to detect the threats. A question to make sure we're on the same page (though I may forget to reply as this thread is no longer in my first page): Do you mean, for example, that AI can help us make new vaccines really fast, so even synthetic pandemics are not a huge risk? Because I'd agree with the first part, it's just I have no reason to expect the second half to also be true. (It might genuinely be true, I just don't have reason to expect that). > An AI isn't necessarily a singular person, and does not need to come up with the idea "itself". Spawn X copies with different weight values, or create Y new AI's with similar methodology but somewhat different training sets, let them compete, or collaborate, as needed, to come up with something better. Use evolutionary programming to dynamically change weights little my little and see how it affects output. Rinse and repeat. Cull unuseful variants. All true, but I don't think this is pertinent: simulated evolution absolutely works (when you have a fitness function), but it also leads to digital equivalents of the recurrent laryngeal nerve or the exploding appendix, and to fixed points like how everything eventually turns into a crab. > The problem as I see it is not so much that one AI entity will grow to dominate everything, as much as that as a class of entity AI will out-compete humans very quickly once the average AI is smarter than the 75th percentile of humans, much less the 90th or 99th percentile. The best we can hope for at that point is to be brought along for the ride. Even the autistic savant type versions we have not seem to be causing some level of this. Indeed, though I think that depends on the details. So, for the sake of a thought experiment, if I take the current systems and just assume we're stuck on them forever: they take a huge amount of training data to get good at anything, which humans generate, and that means we have to switch jobs every 6 months or so because that's when the AI is now good enough to replace us at the specific roles it just saw all of us collectively performing. But that scenario is definitely not a universal, even if it happens in some cases: We've got some bounded systems with fixed rules and a clear mechanism for scoring the quality of the output, and for those systems we get very rapidly super-human output from self-play, as we have seen with AlphaZero. Also, as we see with image GenAI, what domain experts think they've been selling before now, often turns out to not be that close to what their customers thought they were buying. This is why even the current systems — systems which put in two horizons, or three legs, or David Cronenberg the fingers — are nevertheless harming artist commissions. > Will that happen soon? Will that happen ever? I don't know. Probably not. Hopefully not. "If the human brain were so simple that we could understand it, we would be so simple that we couldn’t." On the one hand, this is why I suspect that any self-improvement process will be limited (though I don't know what the limit will be). On the other, our DNA didn't really "understand" the brains it was creating as it evolved us, and our brains are existence proofs that it is possible to make a human-level intelligence in a 20 watt, 1 kg unit. The counterpoint is, this took billions of years of parallel development, and even then might have been a fluke. (And while we can't use the lack of evidence for space-faring civilisations to say if the fluke is before or after the creation of life itself, we can say that at least one of [life emerges, life evolves our kind of minds] must be a fluke).