4 ms·

It's honestly kinda unsettling how many bizarre sci-fi takes on AGI have gained popular plausibility since the new AI marketing boom. Potent kool-aid.

llamaimperative 4y ago
Not weird at all. When (probably when, not if) we create it, we will quickly be interacting with an entity with a similar power imbalance as we have to ants. No one knows exactly what the outcome will be, even less so what to do about it, but surely the kool-aid is watching accelerating progress toward this state and believing either it won’t happen or it won’t be interesting/surprising. Disclaimer: I don’t think this is coming soon per se, but 1) I don’t know for sure that it’s not and 2) it’s definitely coming at some point, if humans manage our other existential risks long enough.
- coldtea 4y ago
  >Not weird at all. When (probably when, not if) we create it, we will quickly be interacting with an entity with a similar power imbalance as we have to ants. That's the sci-fi idea, which handwaves away several very big assumptions and improbabilities, such as that intelligence can be grown to absurd levels (or even just majorly) just by adding hardware to allow for it, that intelligence can just be expanded by itself (thinking itself into being more intelligent), and more importantly intelligence is some kind of superpower that transends very real limitations (like, physical access, or convincing people to do its bidding). The idea is that if you're super-smart you can convince anybody to do anything, but I'm not sure how this came about, because most super-smart people usually can't convince others to do shit, it's charismatic people (which is orthogonal to high-IQ) that do that. In fact many MENSA geniuses just rub people the wrong way. I mean one of the most serious (I use the term loosely) proponents of this charade, thinks the "singularity" will create nanobots to control the world...
  - ethanbond 4y ago
    1) Why do we have a reason to believe there’s a cap to intelligence? 2) No, you don’t need to assume it happens via adding hardware 3) No, you don’t need to assume it can expand by itself 4) We’ve already seen people connecting AI systems to open internet and open internet to command and control of physical systems (cars, utilities, soon weapons systems) - turns out there was no convincing necessary 5) Nanobots also not a necessary assumption Also, incentives, not charisma, are what steer human behavior. Consider for example if the Bitcoin whitepaper were written by an AI? No escape necessary. No convincing necessary. Yet the idea has managed to marshal enormous amounts of capital, computational power, social power, and even some political clout. IMO there’s no reason to believe the AI looks like a robot terminator rather than an incentive-generating piece of information.
    - notahacker 4y ago
      > Also, incentives, not charisma, are what steer human behavior. The number of people with incentives to work towards destroying the whole of humanity is rather small, and I'd rather we deal with them first since they need not wait for unfriendly AI to be dangerous...
      - ethanbond 4y ago
        And yet here we all are consuming and pumping oil because… incentives.
        coldtea 4y ago
        That's assuming (1) "consuming and pumping oil" is a bad thing, and not a complex win-some/lose-some thing, which gave us huge progress and benefits in the 200 years we've used fossil fuels to pump the industrial revolution, (2) and that we can just absurtly stop consuming and pumping it right now, and there won't be huge disastrous 2nd order effects (in economy, society, ability for people to heat themselves etc) until better replacement technologies are not just available, but have similar EROI AND (even more important) capacity to scale up production, building, and deployment fast enough to cover our needs (which we're nowhere near with any "sustainable" technology, including nuclear).
  - concordDance 4y ago
    Intelligence lets a guy with a moustache convince millions of people to invade Poland and kill all the Jews. Its not his liver or his kidneys, its his brain. Feel free to call this type of thing "charisma" but for the purposes of thinking about AI it still comes from the but doing the thinking. And there's actually a pretty good correlation between being good at the various types of mental things, including charisma and IQ (e.g. presidents tend to be over iq 120), because the brain is a very complex organ with many ways it can get screwed up. Would an AI necessarily be super charismatic initially? Maybe not, but don't assume greater intelligence won't lead to greater charisma. (Yes, I know it wasn't just Hitler doing the convincing, but again, that distinction isn't important here as making a message come from many different people is trivial marketing 201/propaganda 101 in the 21st century)
    - coldtea 4y ago
      >Intelligence lets a guy with a moustache convince millions of people to invade Poland and kill all the Jews. Its not his liver or his kidneys, its his brain. Feel free to call this type of thing "charisma" but for the purposes of thinking about AI it still comes from the but doing the thinking. Try to get people to do what you want as an intelligent aspie without charisma, and see how far this goes... even if you have 60+ IQ points over Hitler. Also, Hitler was mostly redundant. He convinced people because they were ready to convinced - Germans would have gone to that one way or another... (they wouldn't have had killed the Jews - that was his obsession and "contribution", but the revenge for the WWI defeat and expansionism and WWII were brewing and inevitable in Germany, and the brand of fascism was also on the rise in places that felt they got the short end of the stick during colonialism, which is why similar movements rose in Japan and Italy and Spain and elsewhere, even before Hitler became a thing. Hitler just rode the currents to sell his particular brand of snake-oil).
  - naasking 4y ago
    > such as that intelligence can be grown to absurd levels (or even just majorly) just by adding hardware to allow for it This has been the empirically validated trend. The more hardware, the more parallel lines of thought can be explored at once (like in chess). > that intelligence can just be expanded by itself (thinking itself into being more intelligent) Of course it can, because humans have already done this. What do you think language, writing, books, school, science and computers are? They are humans augmenting their own effective intelligence by various means. An AI will likely have further means at its disposal, such as access to its own source code (see Godel machines and AIXI for formal models of how this works). > more importantly intelligence is some kind of superpower that transends very real limitations (like, physical access, or convincing people to do its bidding) Intelligence is a power, not a superpower. However, money is a power too, and all of our money is digitized these days, and a digital superintelligence might arguably have an advantage in this domain. A superintelligence with lots of money can arguably convince a whole lot of people to do what it wants and get the physical access it needs.
    - coldtea 4y ago
      >Of course it can, because humans have already done this. What do you think language, writing, books, school, science and computers are? Language, writing, books, school, science and computers never raised anybody's IQ. Genetics and nutrition, on the other hand, did. >However, money is a power too, and all of our money is digitized these days, and a digital superintelligence might arguably have an advantage in this domain. A superintelligence with lots of money can arguably convince a whole lot of people to do what it wants and get the physical access it needs. Who would let superintelligence have a wallet and budget? Or would we just sit and have it steal money? And why would it be any more effective, and any less visible or trackable and able to be shut down than a rich person using his money? Handwavy recursive argument: "but because its superintelligent"
      - naasking 4y ago
        > Language, writing, books, school, science and computers never raised anybody's IQ. Whether IQ took us to the moon is still a topic of debate, as is whether IQ is equal to or only part of intelligence. But it's clear that language, books, school, science and computers absolutely did take us to the moon. Clearly all of these things improved our intellectual effectiveness, and that's what matters, regardless of whether it fits into your personal definition of "intelligence". > Who would let superintelligence have a wallet and budget? Or would we just sit and have it steal money? Cryptocurrency makes this trivial. All computer systems likely also have vulnerabilities, including financial systems. An AI can probe these systems around the clock, or can subtly introduce vulnerabilities into our vulnerable software supply chains. These are only two among many options that are entirely within the realm of plausibility. > And why would it be any more effective, and any less visible or trackable and able to be shut down than a rich person using his money? An AI wouldn't necessarily have a physical location that can be tracked as easily as humans. Where is the AI jail? Honestly, it's like you're not even trying to consider how vulnerable we might be to manipulation by a non-human intelligence. All of our protections are fairly effective against human actors. Some of them might provide partial protection from non-human actors, but you should doubt your confidence that they would be effective against such threats.
- olalonde 4y ago
  > I don’t think this is coming soon Personally, I feel it might become a risk one day, but how soon is an important detail. If the threat is still far away, its nature is very blurry and there is no use to worry too much about it today (or advocate for a ban on AI research). There are still many safe intermediary steps we have to go through before the threat concretizes and becomes worth worrying about.
  - ethanbond 4y ago
    These are definitely important considerations, but it’s not clear 1) we’ll see those intermediary steps, or 2) we’ll be able to stop at them. In reality we’ve already blown through many red lines e.g. not connecting AI to the open internet, not connecting it to physical systems. We didn’t even blink! Agreed a ban on AI research is unworkable, which is part of why this threat is so insidious.
cactusplant7374 4y ago
The question is why do programmers believe they know more than Chomsky? The AI we have now is engineering and not science. Don’t you need science to create AI? Don’t you need to understand something fundamental about the brain?
- coldtea 4y ago
  I don't believe that much in any realistic timeframe from AGI, but I don't think you "need to understand something fundamental" to create it. You can create almost anything empirically, without knowing the science behind it at any fundamental level. Evolution, for example, didn't understand anything (as it's just a process). It wasn't someone with an understanding of brains at the scientific level that built our brains. Just an iterative trial and error process with evolutionary feedback. To make a simplistic argument, kind of like we can create fire, without knowing about oxidation and chemistry.
  - cactusplant7374 4y ago
    If I understand you correctly, your sentiment is that humans can stumble upon and solve the most complex problem on the planet? OpenAI claims to be working on AGI. Nothing they have built is even tangentially related.
- olalonde 4y ago
  Not necessarily. Volta invented the battery almost a 100 years before the electron was discovered. Also, planes fly very differently from birds.
  - cactusplant7374 4y ago
    Those things are a billion times less complex than human cognition. It’s also interesting your remark on birds and planes. Chomsky uses a similar analogy to say they are dissimilar: can a submarine swim? Well, depends on what you mean by swim.

llamaimperative 4y ago

Not weird at all. When (probably when, not if) we create it, we will quickly be interacting with an entity with a similar power imbalance as we have to ants. No one knows exactly what the outcome will be, even less so what to do about it, but surely the kool-aid is watching accelerating progress toward this state and believing either it won’t happen or it won’t be interesting/surprising. Disclaimer: I don’t think this is coming soon per se, but 1) I don’t know for sure that it’s not and 2) it’s definitely coming at some point, if humans manage our other existential risks long enough.

coldtea 4y ago
>Not weird at all. When (probably when, not if) we create it, we will quickly be interacting with an entity with a similar power imbalance as we have to ants. That's the sci-fi idea, which handwaves away several very big assumptions and improbabilities, such as that intelligence can be grown to absurd levels (or even just majorly) just by adding hardware to allow for it, that intelligence can just be expanded by itself (thinking itself into being more intelligent), and more importantly intelligence is some kind of superpower that transends very real limitations (like, physical access, or convincing people to do its bidding). The idea is that if you're super-smart you can convince anybody to do anything, but I'm not sure how this came about, because most super-smart people usually can't convince others to do shit, it's charismatic people (which is orthogonal to high-IQ) that do that. In fact many MENSA geniuses just rub people the wrong way. I mean one of the most serious (I use the term loosely) proponents of this charade, thinks the "singularity" will create nanobots to control the world...
- ethanbond 4y ago
  1) Why do we have a reason to believe there’s a cap to intelligence? 2) No, you don’t need to assume it happens via adding hardware 3) No, you don’t need to assume it can expand by itself 4) We’ve already seen people connecting AI systems to open internet and open internet to command and control of physical systems (cars, utilities, soon weapons systems) - turns out there was no convincing necessary 5) Nanobots also not a necessary assumption Also, incentives, not charisma, are what steer human behavior. Consider for example if the Bitcoin whitepaper were written by an AI? No escape necessary. No convincing necessary. Yet the idea has managed to marshal enormous amounts of capital, computational power, social power, and even some political clout. IMO there’s no reason to believe the AI looks like a robot terminator rather than an incentive-generating piece of information.
  - notahacker 4y ago
    > Also, incentives, not charisma, are what steer human behavior. The number of people with incentives to work towards destroying the whole of humanity is rather small, and I'd rather we deal with them first since they need not wait for unfriendly AI to be dangerous...
    - ethanbond 4y ago
      And yet here we all are consuming and pumping oil because… incentives.
      - coldtea 4y ago
        That's assuming (1) "consuming and pumping oil" is a bad thing, and not a complex win-some/lose-some thing, which gave us huge progress and benefits in the 200 years we've used fossil fuels to pump the industrial revolution, (2) and that we can just absurtly stop consuming and pumping it right now, and there won't be huge disastrous 2nd order effects (in economy, society, ability for people to heat themselves etc) until better replacement technologies are not just available, but have similar EROI AND (even more important) capacity to scale up production, building, and deployment fast enough to cover our needs (which we're nowhere near with any "sustainable" technology, including nuclear).
- concordDance 4y ago
  Intelligence lets a guy with a moustache convince millions of people to invade Poland and kill all the Jews. Its not his liver or his kidneys, its his brain. Feel free to call this type of thing "charisma" but for the purposes of thinking about AI it still comes from the but doing the thinking. And there's actually a pretty good correlation between being good at the various types of mental things, including charisma and IQ (e.g. presidents tend to be over iq 120), because the brain is a very complex organ with many ways it can get screwed up. Would an AI necessarily be super charismatic initially? Maybe not, but don't assume greater intelligence won't lead to greater charisma. (Yes, I know it wasn't just Hitler doing the convincing, but again, that distinction isn't important here as making a message come from many different people is trivial marketing 201/propaganda 101 in the 21st century)
  - coldtea 4y ago
    >Intelligence lets a guy with a moustache convince millions of people to invade Poland and kill all the Jews. Its not his liver or his kidneys, its his brain. Feel free to call this type of thing "charisma" but for the purposes of thinking about AI it still comes from the but doing the thinking. Try to get people to do what you want as an intelligent aspie without charisma, and see how far this goes... even if you have 60+ IQ points over Hitler. Also, Hitler was mostly redundant. He convinced people because they were ready to convinced - Germans would have gone to that one way or another... (they wouldn't have had killed the Jews - that was his obsession and "contribution", but the revenge for the WWI defeat and expansionism and WWII were brewing and inevitable in Germany, and the brand of fascism was also on the rise in places that felt they got the short end of the stick during colonialism, which is why similar movements rose in Japan and Italy and Spain and elsewhere, even before Hitler became a thing. Hitler just rode the currents to sell his particular brand of snake-oil).
- naasking 4y ago
  > such as that intelligence can be grown to absurd levels (or even just majorly) just by adding hardware to allow for it This has been the empirically validated trend. The more hardware, the more parallel lines of thought can be explored at once (like in chess). > that intelligence can just be expanded by itself (thinking itself into being more intelligent) Of course it can, because humans have already done this. What do you think language, writing, books, school, science and computers are? They are humans augmenting their own effective intelligence by various means. An AI will likely have further means at its disposal, such as access to its own source code (see Godel machines and AIXI for formal models of how this works). > more importantly intelligence is some kind of superpower that transends very real limitations (like, physical access, or convincing people to do its bidding) Intelligence is a power, not a superpower. However, money is a power too, and all of our money is digitized these days, and a digital superintelligence might arguably have an advantage in this domain. A superintelligence with lots of money can arguably convince a whole lot of people to do what it wants and get the physical access it needs.
  - coldtea 4y ago
    >Of course it can, because humans have already done this. What do you think language, writing, books, school, science and computers are? Language, writing, books, school, science and computers never raised anybody's IQ. Genetics and nutrition, on the other hand, did. >However, money is a power too, and all of our money is digitized these days, and a digital superintelligence might arguably have an advantage in this domain. A superintelligence with lots of money can arguably convince a whole lot of people to do what it wants and get the physical access it needs. Who would let superintelligence have a wallet and budget? Or would we just sit and have it steal money? And why would it be any more effective, and any less visible or trackable and able to be shut down than a rich person using his money? Handwavy recursive argument: "but because its superintelligent"
    - naasking 4y ago
      > Language, writing, books, school, science and computers never raised anybody's IQ. Whether IQ took us to the moon is still a topic of debate, as is whether IQ is equal to or only part of intelligence. But it's clear that language, books, school, science and computers absolutely did take us to the moon. Clearly all of these things improved our intellectual effectiveness, and that's what matters, regardless of whether it fits into your personal definition of "intelligence". > Who would let superintelligence have a wallet and budget? Or would we just sit and have it steal money? Cryptocurrency makes this trivial. All computer systems likely also have vulnerabilities, including financial systems. An AI can probe these systems around the clock, or can subtly introduce vulnerabilities into our vulnerable software supply chains. These are only two among many options that are entirely within the realm of plausibility. > And why would it be any more effective, and any less visible or trackable and able to be shut down than a rich person using his money? An AI wouldn't necessarily have a physical location that can be tracked as easily as humans. Where is the AI jail? Honestly, it's like you're not even trying to consider how vulnerable we might be to manipulation by a non-human intelligence. All of our protections are fairly effective against human actors. Some of them might provide partial protection from non-human actors, but you should doubt your confidence that they would be effective against such threats.
olalonde 4y ago
> I don’t think this is coming soon Personally, I feel it might become a risk one day, but how soon is an important detail. If the threat is still far away, its nature is very blurry and there is no use to worry too much about it today (or advocate for a ban on AI research). There are still many safe intermediary steps we have to go through before the threat concretizes and becomes worth worrying about.
- ethanbond 4y ago
  These are definitely important considerations, but it’s not clear 1) we’ll see those intermediary steps, or 2) we’ll be able to stop at them. In reality we’ve already blown through many red lines e.g. not connecting AI to the open internet, not connecting it to physical systems. We didn’t even blink! Agreed a ban on AI research is unworkable, which is part of why this threat is so insidious.

cactusplant7374 4y ago

The question is why do programmers believe they know more than Chomsky? The AI we have now is engineering and not science. Don’t you need science to create AI? Don’t you need to understand something fundamental about the brain?

coldtea 4y ago
I don't believe that much in any realistic timeframe from AGI, but I don't think you "need to understand something fundamental" to create it. You can create almost anything empirically, without knowing the science behind it at any fundamental level. Evolution, for example, didn't understand anything (as it's just a process). It wasn't someone with an understanding of brains at the scientific level that built our brains. Just an iterative trial and error process with evolutionary feedback. To make a simplistic argument, kind of like we can create fire, without knowing about oxidation and chemistry.
- cactusplant7374 4y ago
  If I understand you correctly, your sentiment is that humans can stumble upon and solve the most complex problem on the planet? OpenAI claims to be working on AGI. Nothing they have built is even tangentially related.
olalonde 4y ago
Not necessarily. Volta invented the battery almost a 100 years before the electron was discovered. Also, planes fly very differently from birds.
- cactusplant7374 4y ago
  Those things are a billion times less complex than human cognition. It’s also interesting your remark on birds and planes. Chomsky uses a similar analogy to say they are dissimilar: can a submarine swim? Well, depends on what you mean by swim.