6 ms·
>Not weird at all. When (probably when, not if) we create it, we will quickly be interacting with an entity with a similar power imbalance as we have to ants.
by coldtea 4y ago
>Not weird at all. When (probably when, not if) we create it, we will quickly be interacting with an entity with a similar power imbalance as we have to ants.
That's the sci-fi idea, which handwaves away several very big assumptions and improbabilities, such as that intelligence can be grown to absurd levels (or even just majorly) just by adding hardware to allow for it, that intelligence can just be expanded by itself (thinking itself into being more intelligent), and more importantly intelligence is some kind of superpower that transends very real limitations (like, physical access, or convincing people to do its bidding). The idea is that if you're super-smart you can convince anybody to do anything, but I'm not sure how this came about, because most super-smart people usually can't convince others to do shit, it's charismatic people (which is orthogonal to high-IQ) that do that. In fact many MENSA geniuses just rub people the wrong way.
I mean one of the most serious (I use the term loosely) proponents of this charade, thinks the "singularity" will create nanobots to control the world...
- ethanbond 4y ago1) Why do we have a reason to believe there’s a cap to intelligence? 2) No, you don’t need to assume it happens via adding hardware 3) No, you don’t need to assume it can expand by itself 4) We’ve already seen people connecting AI systems to open internet and open internet to command and control of physical systems (cars, utilities, soon weapons systems) - turns out there was no convincing necessary 5) Nanobots also not a necessary assumption Also, incentives, not charisma, are what steer human behavior. Consider for example if the Bitcoin whitepaper were written by an AI? No escape necessary. No convincing necessary. Yet the idea has managed to marshal enormous amounts of capital, computational power, social power, and even some political clout. IMO there’s no reason to believe the AI looks like a robot terminator rather than an incentive-generating piece of information.
- notahacker 4y ago> Also, incentives, not charisma, are what steer human behavior. The number of people with incentives to work towards destroying the whole of humanity is rather small, and I'd rather we deal with them first since they need not wait for unfriendly AI to be dangerous...
- ethanbond 4y agoAnd yet here we all are consuming and pumping oil because… incentives.
- coldtea 4y agoThat's assuming (1) "consuming and pumping oil" is a bad thing, and not a complex win-some/lose-some thing, which gave us huge progress and benefits in the 200 years we've used fossil fuels to pump the industrial revolution, (2) and that we can just absurtly stop consuming and pumping it right now, and there won't be huge disastrous 2nd order effects (in economy, society, ability for people to heat themselves etc) until better replacement technologies are not just available, but have similar EROI AND (even more important) capacity to scale up production, building, and deployment fast enough to cover our needs (which we're nowhere near with any "sustainable" technology, including nuclear).
- ethanbond 4y agoIt doesn’t assume either of those things. In fact it accounts for exactly those points under the heading “incentives.” We are engaging in clearly self-destructive behavior because, incentive structure accounted for (your points), it very well appears to be the best option. It wouldn’t be a hard problem if it didn’t. It should be obvious that it’s entirely possible to follow your (real) incentives all the way into a very bad outcome.
- coldtea 4y ago>We are engaging in clearly self-destructive behavior because, incentive structure accounted for (your points), it very well appears to be the best option Well, if it's the "best option" then it clearly isn't self-destructive... It's just something with side effects, but which also yielded huge benefits. Kind of how alcoholism is self-destructing. But a side-effect from taking a drug that helps us perform betetr or treats some condition we have, is not self-destructing. It's more of an unforseen issue, that we can't immediately correct (e.g. because of the condition worsening if we do it abruptly).
- coldtea 4y ago>1) Why do we have a reason to believe there’s a cap to intelligence? There's a cap to all things. Diminishing returns is a thing. Both physical substrates have limits, and there's a utility limit to thinking or having more data. Limitless intelligence is fairy-tale magical thinking - especially in the way it's portrayed as some magical ability to build and control everything. Like, invent working cold-fusion - as if you don't need the experiments, iterative process, random breakthroughs, and such, but just "smarter" thinking. > 2) No, you don’t need to assume it happens via adding hardware Yes, you do, else it's just the same capped-capability hardware, thinking itself into being smarter. > 3) No, you don’t need to assume it can expand by itself Well, I, personally, don't need to assume anything. But people talking about AGI, do assume it. > 4) We’ve already seen people connecting AI systems to open internet and open internet to command and control of physical systems (cars, utilities, soon weapons systems) - turns out there was no convincing necessary That's neither here nor there, though, because I didn't say that connecting AGI to the open internet is impossible or can't happen. Just that it doesn't mean much. We still control the kind of control we give an AI, and any physical systems any AGI would be connected will still have limitations. It's not some magical portal into conquering the world. >5) Nanobots also not a necessary assumption Didn't say it's necessary, just pointed out that it has been made - along with many other non necessary, and far-fetched assumptions and hand waving... >Also, incentives, not charisma, are what steer human behavior. Both do. With charisma you can do it without incentives - and even instill incentives into people. Otherwise you're stuck with innate incentives like people wanting to be rich - which you don't need to be a AGI to exploit, just rich or a con-man.
- ethanbond 4y ago1) Obviously intelligence plays a role in how fast science develops. The quality of experiments and the quality of inference from those experiments are (very obviously) modulated by intelligence. No reason to believe the absolute pinnacle of intelligence happens to sit inside a species of mostly hairless apes here on earth. Lots of reasons to believe fundamental understanding of the universe is more accessible to intelligent beings than to unintelligent ones. 2) Nope. Could be different hardware. The brain consumes far less power than any human-designed computer that could hope to match its computational power. 3) Sure some do, but it’s not a necessary assumption to take AGI risk seriously. 4) Meh, this seems to me just the next concession in the line of “we can’t do that, we wouldn’t do that, we do it but it’s not that bad” 5) Okay so it’s unrelated to this conversation except I suppose some sort of ad hominem? 6) So we agree charisma is at best not necessary to manipulate the world
- bioemerl 4y agoWe are already made of nanobots, we are the nanobots, we are in intelligence, we are a general intelligence, and we are growing exponentially. AGI is here. AGI is you. You are an incredibly efficient piece of carbon thinkware able to sell for replicate into more carbon thinkware and that that extra carbon think where can go out and create its own life and make its own observations and spread information through the society of you both inherit, that's society is growing exponentially and growing in its ability to modify the world and manipulate and improve itself exponentially. If you're scared of AGI, don't be. You are the superior AGI, because these machines made of metal and circuits are never going to compare to machines made of carbon using chemical reactions that are far more efficient than transistors to do the thinking. The future is in improving ourselves and reaching a point at which our biology is no longer a mysterious black box so that we can modify and improve ourselves and those around us. We can already improve machines, but that's because the machines are far simpler and far more primitive than our minds are.
- afiori 4y agoMachines being simpler is the problem. The problems with AGI are 1) runaway growth and the singularity and 2) the scale and reach of its actions. Humans are deeply limited in their senses, their actions, and their ability to communicate. An AGI botnet installed on every computer/control system in the world would have none of those limitations.
- bioemerl 4y agoComplexity breeds ability. Our computers will have to continue to get more complex in order for AI to do the same. What humans have is scale. There are 10 billion devices connected to the internet. Cool, right? More than humans? No. Being human has really big perks. We are mobile. We have hands and feet. We have eyes and touch and smell and we use tools to expand what we can see. We each possess the ability to process data and filter data at scales many magnitudes greater than a computer. Note that data in this case isn't "many miles of spreadsheet data" - data in this case is worldly observation. You notice weird things, you report it to your friends. You try weird things, you report it to your friends. Communication between humans is limited, as is our memory, but it's limited for a reason. 99 percent of information is worthless junk human communication gets most of the important parts through. Then we need to parse mass amounts of data? Computers do that. Comparing your information to all other known information? It's an O n squared problem. You can throw hardware at it but as informations scope grows it will outpace the hardware eventually. Humans are crazy advanced compared to machines in basically every sense. I think the viewpoint here that machines will have superior senses is extrapolating how computers work today and is thoroughly impractical. We will find these logical and reproducible systems look more and more like we do as we push the line. I expect eventual marginal improvements that can't outpace human "plentifulness", and biological modifications and creations to be our real future.
- concordDance 4y agoIntelligence lets a guy with a moustache convince millions of people to invade Poland and kill all the Jews. Its not his liver or his kidneys, its his brain. Feel free to call this type of thing "charisma" but for the purposes of thinking about AI it still comes from the but doing the thinking. And there's actually a pretty good correlation between being good at the various types of mental things, including charisma and IQ (e.g. presidents tend to be over iq 120), because the brain is a very complex organ with many ways it can get screwed up. Would an AI necessarily be super charismatic initially? Maybe not, but don't assume greater intelligence won't lead to greater charisma. (Yes, I know it wasn't just Hitler doing the convincing, but again, that distinction isn't important here as making a message come from many different people is trivial marketing 201/propaganda 101 in the 21st century)
- coldtea 4y ago>Intelligence lets a guy with a moustache convince millions of people to invade Poland and kill all the Jews. Its not his liver or his kidneys, its his brain. Feel free to call this type of thing "charisma" but for the purposes of thinking about AI it still comes from the but doing the thinking. Try to get people to do what you want as an intelligent aspie without charisma, and see how far this goes... even if you have 60+ IQ points over Hitler. Also, Hitler was mostly redundant. He convinced people because they were ready to convinced - Germans would have gone to that one way or another... (they wouldn't have had killed the Jews - that was his obsession and "contribution", but the revenge for the WWI defeat and expansionism and WWII were brewing and inevitable in Germany, and the brand of fascism was also on the rise in places that felt they got the short end of the stick during colonialism, which is why similar movements rose in Japan and Italy and Spain and elsewhere, even before Hitler became a thing. Hitler just rode the currents to sell his particular brand of snake-oil).
- naasking 4y ago> such as that intelligence can be grown to absurd levels (or even just majorly) just by adding hardware to allow for it This has been the empirically validated trend. The more hardware, the more parallel lines of thought can be explored at once (like in chess). > that intelligence can just be expanded by itself (thinking itself into being more intelligent) Of course it can, because humans have already done this. What do you think language, writing, books, school, science and computers are? They are humans augmenting their own effective intelligence by various means. An AI will likely have further means at its disposal, such as access to its own source code (see Godel machines and AIXI for formal models of how this works). > more importantly intelligence is some kind of superpower that transends very real limitations (like, physical access, or convincing people to do its bidding) Intelligence is a power, not a superpower. However, money is a power too, and all of our money is digitized these days, and a digital superintelligence might arguably have an advantage in this domain. A superintelligence with lots of money can arguably convince a whole lot of people to do what it wants and get the physical access it needs.
- coldtea 4y ago>Of course it can, because humans have already done this. What do you think language, writing, books, school, science and computers are? Language, writing, books, school, science and computers never raised anybody's IQ. Genetics and nutrition, on the other hand, did. >However, money is a power too, and all of our money is digitized these days, and a digital superintelligence might arguably have an advantage in this domain. A superintelligence with lots of money can arguably convince a whole lot of people to do what it wants and get the physical access it needs. Who would let superintelligence have a wallet and budget? Or would we just sit and have it steal money? And why would it be any more effective, and any less visible or trackable and able to be shut down than a rich person using his money? Handwavy recursive argument: "but because its superintelligent"
- naasking 4y ago> Language, writing, books, school, science and computers never raised anybody's IQ. Whether IQ took us to the moon is still a topic of debate, as is whether IQ is equal to or only part of intelligence. But it's clear that language, books, school, science and computers absolutely did take us to the moon. Clearly all of these things improved our intellectual effectiveness, and that's what matters, regardless of whether it fits into your personal definition of "intelligence". > Who would let superintelligence have a wallet and budget? Or would we just sit and have it steal money? Cryptocurrency makes this trivial. All computer systems likely also have vulnerabilities, including financial systems. An AI can probe these systems around the clock, or can subtly introduce vulnerabilities into our vulnerable software supply chains. These are only two among many options that are entirely within the realm of plausibility. > And why would it be any more effective, and any less visible or trackable and able to be shut down than a rich person using his money? An AI wouldn't necessarily have a physical location that can be tracked as easily as humans. Where is the AI jail? Honestly, it's like you're not even trying to consider how vulnerable we might be to manipulation by a non-human intelligence. All of our protections are fairly effective against human actors. Some of them might provide partial protection from non-human actors, but you should doubt your confidence that they would be effective against such threats.