4 ms·

> such as that intelligence can be grown to absurd levels (or even just majorly) just by adding hardware to allow for it This has been the empirically validate

> such as that intelligence can be grown to absurd levels (or even just majorly) just by adding hardware to allow for it This has been the empirically validated trend. The more hardware, the more parallel lines of thought can be explored at once (like in chess). > that intelligence can just be expanded by itself (thinking itself into being more intelligent) Of course it can, because humans have already done this. What do you think language, writing, books, school, science and computers are? They are humans augmenting their own effective intelligence by various means. An AI will likely have further means at its disposal, such as access to its own source code (see Godel machines and AIXI for formal models of how this works). > more importantly intelligence is some kind of superpower that transends very real limitations (like, physical access, or convincing people to do its bidding) Intelligence is a power, not a superpower. However, money is a power too, and all of our money is digitized these days, and a digital superintelligence might arguably have an advantage in this domain. A superintelligence with lots of money can arguably convince a whole lot of people to do what it wants and get the physical access it needs.

coldtea 4y ago
>Of course it can, because humans have already done this. What do you think language, writing, books, school, science and computers are? Language, writing, books, school, science and computers never raised anybody's IQ. Genetics and nutrition, on the other hand, did. >However, money is a power too, and all of our money is digitized these days, and a digital superintelligence might arguably have an advantage in this domain. A superintelligence with lots of money can arguably convince a whole lot of people to do what it wants and get the physical access it needs. Who would let superintelligence have a wallet and budget? Or would we just sit and have it steal money? And why would it be any more effective, and any less visible or trackable and able to be shut down than a rich person using his money? Handwavy recursive argument: "but because its superintelligent"
- naasking 4y ago
  > Language, writing, books, school, science and computers never raised anybody's IQ. Whether IQ took us to the moon is still a topic of debate, as is whether IQ is equal to or only part of intelligence. But it's clear that language, books, school, science and computers absolutely did take us to the moon. Clearly all of these things improved our intellectual effectiveness, and that's what matters, regardless of whether it fits into your personal definition of "intelligence". > Who would let superintelligence have a wallet and budget? Or would we just sit and have it steal money? Cryptocurrency makes this trivial. All computer systems likely also have vulnerabilities, including financial systems. An AI can probe these systems around the clock, or can subtly introduce vulnerabilities into our vulnerable software supply chains. These are only two among many options that are entirely within the realm of plausibility. > And why would it be any more effective, and any less visible or trackable and able to be shut down than a rich person using his money? An AI wouldn't necessarily have a physical location that can be tracked as easily as humans. Where is the AI jail? Honestly, it's like you're not even trying to consider how vulnerable we might be to manipulation by a non-human intelligence. All of our protections are fairly effective against human actors. Some of them might provide partial protection from non-human actors, but you should doubt your confidence that they would be effective against such threats.
  - danaris 4y ago
    > Cryptocurrency makes this trivial. All computer systems likely also have vulnerabilities, including financial systems. An AI can probe these systems around the clock, or can subtly introduce vulnerabilities into our vulnerable software supply chains. These are only two among many options that are entirely within the realm of plausibility. And there it is: the assumption that AGI will, necessarily, be capable of hacking effectively any computer system, no matter how well-secured. This belongs in Hollywood, not on HN.
    - naasking 4y ago
      > And there it is: the assumption that AGI will, necessarily, be capable of hacking effectively any computer system, no matter how well-secured. Where did I say they are capable of hacking any computer system no matter how well secured? I said that software vulnerabilities exist, that software supply chains are known to be vulnerable, that even financial institutions are vulnerable. These are all documented facts. An AI doesn't have to successfully hack any computer system, it only needs to successfully hack one financial computer system to create an account that can send and receive money. Given it can plausibly attempt to do so consistently around the clock via numerous means (like vulnerable software supply chains or other exploits), the chances of it doing so successfully against any of the tens of thousands of banking systems around the world over a timespan of years seems plausibly certain. And that's only for the things it needs that require some kind of official financial account, it can do plenty with cryptocurrency alone. Boy do you skeptics love your strawman arguments.

coldtea 4y ago

>Of course it can, because humans have already done this. What do you think language, writing, books, school, science and computers are? Language, writing, books, school, science and computers never raised anybody's IQ. Genetics and nutrition, on the other hand, did. >However, money is a power too, and all of our money is digitized these days, and a digital superintelligence might arguably have an advantage in this domain. A superintelligence with lots of money can arguably convince a whole lot of people to do what it wants and get the physical access it needs. Who would let superintelligence have a wallet and budget? Or would we just sit and have it steal money? And why would it be any more effective, and any less visible or trackable and able to be shut down than a rich person using his money? Handwavy recursive argument: "but because its superintelligent"

naasking 4y ago
> Language, writing, books, school, science and computers never raised anybody's IQ. Whether IQ took us to the moon is still a topic of debate, as is whether IQ is equal to or only part of intelligence. But it's clear that language, books, school, science and computers absolutely did take us to the moon. Clearly all of these things improved our intellectual effectiveness, and that's what matters, regardless of whether it fits into your personal definition of "intelligence". > Who would let superintelligence have a wallet and budget? Or would we just sit and have it steal money? Cryptocurrency makes this trivial. All computer systems likely also have vulnerabilities, including financial systems. An AI can probe these systems around the clock, or can subtly introduce vulnerabilities into our vulnerable software supply chains. These are only two among many options that are entirely within the realm of plausibility. > And why would it be any more effective, and any less visible or trackable and able to be shut down than a rich person using his money? An AI wouldn't necessarily have a physical location that can be tracked as easily as humans. Where is the AI jail? Honestly, it's like you're not even trying to consider how vulnerable we might be to manipulation by a non-human intelligence. All of our protections are fairly effective against human actors. Some of them might provide partial protection from non-human actors, but you should doubt your confidence that they would be effective against such threats.
- danaris 4y ago
  > Cryptocurrency makes this trivial. All computer systems likely also have vulnerabilities, including financial systems. An AI can probe these systems around the clock, or can subtly introduce vulnerabilities into our vulnerable software supply chains. These are only two among many options that are entirely within the realm of plausibility. And there it is: the assumption that AGI will, necessarily, be capable of hacking effectively any computer system, no matter how well-secured. This belongs in Hollywood, not on HN.
  - naasking 4y ago
    > And there it is: the assumption that AGI will, necessarily, be capable of hacking effectively any computer system, no matter how well-secured. Where did I say they are capable of hacking any computer system no matter how well secured? I said that software vulnerabilities exist, that software supply chains are known to be vulnerable, that even financial institutions are vulnerable. These are all documented facts. An AI doesn't have to successfully hack any computer system, it only needs to successfully hack one financial computer system to create an account that can send and receive money. Given it can plausibly attempt to do so consistently around the clock via numerous means (like vulnerable software supply chains or other exploits), the chances of it doing so successfully against any of the tens of thousands of banking systems around the world over a timespan of years seems plausibly certain. And that's only for the things it needs that require some kind of official financial account, it can do plenty with cryptocurrency alone. Boy do you skeptics love your strawman arguments.