5 ms·
>If AGI has even a 1% chance of happening this century and those donations reduce the risk of it going wrong and killing everyone by 1% that's still money very
by coldtea 4y ago
>If AGI has even a 1% chance of happening this century and those donations reduce the risk of it going wrong and killing everyone by 1% that's still money very well spent.
This is Pascal's wager level absurdity combined with Drake equation level accuracy...
- marginalia_nu 4y agoIt's honestly kinda unsettling how many bizarre sci-fi takes on AGI have gained popular plausibility since the new AI marketing boom. Potent kool-aid.
- llamaimperative 4y agoNot weird at all. When (probably when, not if) we create it, we will quickly be interacting with an entity with a similar power imbalance as we have to ants. No one knows exactly what the outcome will be, even less so what to do about it, but surely the kool-aid is watching accelerating progress toward this state and believing either it won’t happen or it won’t be interesting/surprising. Disclaimer: I don’t think this is coming soon per se, but 1) I don’t know for sure that it’s not and 2) it’s definitely coming at some point, if humans manage our other existential risks long enough.
- coldtea 4y ago>Not weird at all. When (probably when, not if) we create it, we will quickly be interacting with an entity with a similar power imbalance as we have to ants. That's the sci-fi idea, which handwaves away several very big assumptions and improbabilities, such as that intelligence can be grown to absurd levels (or even just majorly) just by adding hardware to allow for it, that intelligence can just be expanded by itself (thinking itself into being more intelligent), and more importantly intelligence is some kind of superpower that transends very real limitations (like, physical access, or convincing people to do its bidding). The idea is that if you're super-smart you can convince anybody to do anything, but I'm not sure how this came about, because most super-smart people usually can't convince others to do shit, it's charismatic people (which is orthogonal to high-IQ) that do that. In fact many MENSA geniuses just rub people the wrong way. I mean one of the most serious (I use the term loosely) proponents of this charade, thinks the "singularity" will create nanobots to control the world...
- ethanbond 4y ago1) Why do we have a reason to believe there’s a cap to intelligence? 2) No, you don’t need to assume it happens via adding hardware 3) No, you don’t need to assume it can expand by itself 4) We’ve already seen people connecting AI systems to open internet and open internet to command and control of physical systems (cars, utilities, soon weapons systems) - turns out there was no convincing necessary 5) Nanobots also not a necessary assumption Also, incentives, not charisma, are what steer human behavior. Consider for example if the Bitcoin whitepaper were written by an AI? No escape necessary. No convincing necessary. Yet the idea has managed to marshal enormous amounts of capital, computational power, social power, and even some political clout. IMO there’s no reason to believe the AI looks like a robot terminator rather than an incentive-generating piece of information.
- notahacker 4y ago> Also, incentives, not charisma, are what steer human behavior. The number of people with incentives to work towards destroying the whole of humanity is rather small, and I'd rather we deal with them first since they need not wait for unfriendly AI to be dangerous...
- ethanbond 4y agoAnd yet here we all are consuming and pumping oil because… incentives.
- coldtea 4y agoThat's assuming (1) "consuming and pumping oil" is a bad thing, and not a complex win-some/lose-some thing, which gave us huge progress and benefits in the 200 years we've used fossil fuels to pump the industrial revolution, (2) and that we can just absurtly stop consuming and pumping it right now, and there won't be huge disastrous 2nd order effects (in economy, society, ability for people to heat themselves etc) until better replacement technologies are not just available, but have similar EROI AND (even more important) capacity to scale up production, building, and deployment fast enough to cover our needs (which we're nowhere near with any "sustainable" technology, including nuclear).
- olalonde 4y ago> I don’t think this is coming soon Personally, I feel it might become a risk one day, but how soon is an important detail. If the threat is still far away, its nature is very blurry and there is no use to worry too much about it today (or advocate for a ban on AI research). There are still many safe intermediary steps we have to go through before the threat concretizes and becomes worth worrying about.
- ethanbond 4y agoThese are definitely important considerations, but it’s not clear 1) we’ll see those intermediary steps, or 2) we’ll be able to stop at them. In reality we’ve already blown through many red lines e.g. not connecting AI to the open internet, not connecting it to physical systems. We didn’t even blink! Agreed a ban on AI research is unworkable, which is part of why this threat is so insidious.
- cactusplant7374 4y agoThe question is why do programmers believe they know more than Chomsky? The AI we have now is engineering and not science. Don’t you need science to create AI? Don’t you need to understand something fundamental about the brain?
- coldtea 4y agoI don't believe that much in any realistic timeframe from AGI, but I don't think you "need to understand something fundamental" to create it. You can create almost anything empirically, without knowing the science behind it at any fundamental level. Evolution, for example, didn't understand anything (as it's just a process). It wasn't someone with an understanding of brains at the scientific level that built our brains. Just an iterative trial and error process with evolutionary feedback. To make a simplistic argument, kind of like we can create fire, without knowing about oxidation and chemistry.
- cactusplant7374 4y agoIf I understand you correctly, your sentiment is that humans can stumble upon and solve the most complex problem on the planet? OpenAI claims to be working on AGI. Nothing they have built is even tangentially related.
- olalonde 4y agoNot necessarily. Volta invented the battery almost a 100 years before the electron was discovered. Also, planes fly very differently from birds.
- cactusplant7374 4y agoThose things are a billion times less complex than human cognition. It’s also interesting your remark on birds and planes. Chomsky uses a similar analogy to say they are dissimilar: can a submarine swim? Well, depends on what you mean by swim.
- simonh 4y agoSaid in one sentence what took me 4 paragraphs.
- bioemerl 4y agoI also hate I absurd this argument is, we already have a GI, it's you, it's me, it's the exponentially growing humanity that we all live around and contribute to every single day. Why are we so scared of an artificial brain when we have millions of living brains doing exactly what we are scared the artificial brains are going to do. We are growing exponentially, and it almost all certainty these AGI, if they even become a worthwhile thing compared to non-general AI, are probably going to be human, but incredibly similar and it probably won't go exponentially out of control. This idea that a computer could infinitely improve itself while sitting inside of a boxes absurd, do you know what would happen if you put a human thinker inside of a box for say 2 billion years for them to make theories and think about stuff? They eventually go mad and come out with some of the craziest theories in the world because they don't know what they're talking about because in order to learn you have to experiment and work with the world, not just sit in a box and make up theories. Artificial intelligence is going to have the exact same access to resources and information as humanity does, if not less because there's only going to be a couple of AGI and was going to be billions of humans, and humans are out there reproducing like mad still, well a artificial intelligence needs to build itself for relatively rare and hard to source materials. And if it was easy to self-improve and artificial intelligence, we would have figured it out, we'd be artificially improving ourselves by now, but we haven't, because it's not that easy, and for the most part we are probably limited by the laws of physics People worry so much about HI because they assume Moore's law is just going to continue forever, and soon we're going to have the universe on a chip, but anyone saying that I don't think realizes what exponential growth typically ends in, and that is a plateau. I'll tell you what's going to happen with this. We're going to create AGI, and they're going to be different from us, they're going to be different from us in radical ways, but they're still going to have their own form of emotions, goals, desires, influences, and so on. All of this fear is only going to cause us to delay and obstruct the eventual future where we adjust our own ideals of what is moral, what is good, to create a society that can incorporate these radically new beings. These people are shooting ourselves in the foot, delaying the creation of a better society, on the basis of a fear that has no real basis.
- concordDance 4y ago> And if it was easy to self-improve and artificial intelligence, we would have figured it out, we'd be artificially improving ourselves by now, but we haven't, because it's not that easy, and for the most part we are probably limited by the laws of physics There are three big things that stand in the way of human improvement that do not for AI. The first is reproducibility, you can't make a few copies of a starting state of a human and run them with different parameters to observe the results. The second is ethics, you can't try out a bunch of genetic tweaks on babies because one might have bad side effects. The third is observability. People complain about how much of a black box modern AIs are but you can still see it working and figure out what layers do what. > We're going to create AGI, and they're going to be different from us, they're going to be different from us in radical ways, but they're still going to have their own form of emotions, goals, desires, influences, and so on. I am surprised this reassures you. A race of humans that have perfect memory and can copy themselves would be a terrifying thing.