6 ms·

1) Why do we have a reason to believe there’s a cap to intelligence? 2) No, you don’t need to assume it happens via adding hardware 3) No, you don’t need to a

1) Why do we have a reason to believe there’s a cap to intelligence? 2) No, you don’t need to assume it happens via adding hardware 3) No, you don’t need to assume it can expand by itself 4) We’ve already seen people connecting AI systems to open internet and open internet to command and control of physical systems (cars, utilities, soon weapons systems) - turns out there was no convincing necessary 5) Nanobots also not a necessary assumption Also, incentives, not charisma, are what steer human behavior. Consider for example if the Bitcoin whitepaper were written by an AI? No escape necessary. No convincing necessary. Yet the idea has managed to marshal enormous amounts of capital, computational power, social power, and even some political clout. IMO there’s no reason to believe the AI looks like a robot terminator rather than an incentive-generating piece of information.

notahacker 4y ago
> Also, incentives, not charisma, are what steer human behavior. The number of people with incentives to work towards destroying the whole of humanity is rather small, and I'd rather we deal with them first since they need not wait for unfriendly AI to be dangerous...
- ethanbond 4y ago
  And yet here we all are consuming and pumping oil because… incentives.
  - coldtea 4y ago
    That's assuming (1) "consuming and pumping oil" is a bad thing, and not a complex win-some/lose-some thing, which gave us huge progress and benefits in the 200 years we've used fossil fuels to pump the industrial revolution, (2) and that we can just absurtly stop consuming and pumping it right now, and there won't be huge disastrous 2nd order effects (in economy, society, ability for people to heat themselves etc) until better replacement technologies are not just available, but have similar EROI AND (even more important) capacity to scale up production, building, and deployment fast enough to cover our needs (which we're nowhere near with any "sustainable" technology, including nuclear).
    - ethanbond 4y ago
      It doesn’t assume either of those things. In fact it accounts for exactly those points under the heading “incentives.” We are engaging in clearly self-destructive behavior because, incentive structure accounted for (your points), it very well appears to be the best option. It wouldn’t be a hard problem if it didn’t. It should be obvious that it’s entirely possible to follow your (real) incentives all the way into a very bad outcome.
      - coldtea 4y ago
        >We are engaging in clearly self-destructive behavior because, incentive structure accounted for (your points), it very well appears to be the best option Well, if it's the "best option" then it clearly isn't self-destructive... It's just something with side effects, but which also yielded huge benefits. Kind of how alcoholism is self-destructing. But a side-effect from taking a drug that helps us perform betetr or treats some condition we have, is not self-destructing. It's more of an unforseen issue, that we can't immediately correct (e.g. because of the condition worsening if we do it abruptly).
coldtea 4y ago
>1) Why do we have a reason to believe there’s a cap to intelligence? There's a cap to all things. Diminishing returns is a thing. Both physical substrates have limits, and there's a utility limit to thinking or having more data. Limitless intelligence is fairy-tale magical thinking - especially in the way it's portrayed as some magical ability to build and control everything. Like, invent working cold-fusion - as if you don't need the experiments, iterative process, random breakthroughs, and such, but just "smarter" thinking. > 2) No, you don’t need to assume it happens via adding hardware Yes, you do, else it's just the same capped-capability hardware, thinking itself into being smarter. > 3) No, you don’t need to assume it can expand by itself Well, I, personally, don't need to assume anything. But people talking about AGI, do assume it. > 4) We’ve already seen people connecting AI systems to open internet and open internet to command and control of physical systems (cars, utilities, soon weapons systems) - turns out there was no convincing necessary That's neither here nor there, though, because I didn't say that connecting AGI to the open internet is impossible or can't happen. Just that it doesn't mean much. We still control the kind of control we give an AI, and any physical systems any AGI would be connected will still have limitations. It's not some magical portal into conquering the world. >5) Nanobots also not a necessary assumption Didn't say it's necessary, just pointed out that it has been made - along with many other non necessary, and far-fetched assumptions and hand waving... >Also, incentives, not charisma, are what steer human behavior. Both do. With charisma you can do it without incentives - and even instill incentives into people. Otherwise you're stuck with innate incentives like people wanting to be rich - which you don't need to be a AGI to exploit, just rich or a con-man.
- ethanbond 4y ago
  1) Obviously intelligence plays a role in how fast science develops. The quality of experiments and the quality of inference from those experiments are (very obviously) modulated by intelligence. No reason to believe the absolute pinnacle of intelligence happens to sit inside a species of mostly hairless apes here on earth. Lots of reasons to believe fundamental understanding of the universe is more accessible to intelligent beings than to unintelligent ones. 2) Nope. Could be different hardware. The brain consumes far less power than any human-designed computer that could hope to match its computational power. 3) Sure some do, but it’s not a necessary assumption to take AGI risk seriously. 4) Meh, this seems to me just the next concession in the line of “we can’t do that, we wouldn’t do that, we do it but it’s not that bad” 5) Okay so it’s unrelated to this conversation except I suppose some sort of ad hominem? 6) So we agree charisma is at best not necessary to manipulate the world
bioemerl 4y ago
We are already made of nanobots, we are the nanobots, we are in intelligence, we are a general intelligence, and we are growing exponentially. AGI is here. AGI is you. You are an incredibly efficient piece of carbon thinkware able to sell for replicate into more carbon thinkware and that that extra carbon think where can go out and create its own life and make its own observations and spread information through the society of you both inherit, that's society is growing exponentially and growing in its ability to modify the world and manipulate and improve itself exponentially. If you're scared of AGI, don't be. You are the superior AGI, because these machines made of metal and circuits are never going to compare to machines made of carbon using chemical reactions that are far more efficient than transistors to do the thinking. The future is in improving ourselves and reaching a point at which our biology is no longer a mysterious black box so that we can modify and improve ourselves and those around us. We can already improve machines, but that's because the machines are far simpler and far more primitive than our minds are.
- afiori 4y ago
  Machines being simpler is the problem. The problems with AGI are 1) runaway growth and the singularity and 2) the scale and reach of its actions. Humans are deeply limited in their senses, their actions, and their ability to communicate. An AGI botnet installed on every computer/control system in the world would have none of those limitations.
  - bioemerl 4y ago
    Complexity breeds ability. Our computers will have to continue to get more complex in order for AI to do the same. What humans have is scale. There are 10 billion devices connected to the internet. Cool, right? More than humans? No. Being human has really big perks. We are mobile. We have hands and feet. We have eyes and touch and smell and we use tools to expand what we can see. We each possess the ability to process data and filter data at scales many magnitudes greater than a computer. Note that data in this case isn't "many miles of spreadsheet data" - data in this case is worldly observation. You notice weird things, you report it to your friends. You try weird things, you report it to your friends. Communication between humans is limited, as is our memory, but it's limited for a reason. 99 percent of information is worthless junk human communication gets most of the important parts through. Then we need to parse mass amounts of data? Computers do that. Comparing your information to all other known information? It's an O n squared problem. You can throw hardware at it but as informations scope grows it will outpace the hardware eventually. Humans are crazy advanced compared to machines in basically every sense. I think the viewpoint here that machines will have superior senses is extrapolating how computers work today and is thoroughly impractical. We will find these logical and reproducible systems look more and more like we do as we push the line. I expect eventual marginal improvements that can't outpace human "plentifulness", and biological modifications and creations to be our real future.

notahacker 4y ago

> Also, incentives, not charisma, are what steer human behavior. The number of people with incentives to work towards destroying the whole of humanity is rather small, and I'd rather we deal with them first since they need not wait for unfriendly AI to be dangerous...

ethanbond 4y ago
And yet here we all are consuming and pumping oil because… incentives.
- coldtea 4y ago
  That's assuming (1) "consuming and pumping oil" is a bad thing, and not a complex win-some/lose-some thing, which gave us huge progress and benefits in the 200 years we've used fossil fuels to pump the industrial revolution, (2) and that we can just absurtly stop consuming and pumping it right now, and there won't be huge disastrous 2nd order effects (in economy, society, ability for people to heat themselves etc) until better replacement technologies are not just available, but have similar EROI AND (even more important) capacity to scale up production, building, and deployment fast enough to cover our needs (which we're nowhere near with any "sustainable" technology, including nuclear).
  - ethanbond 4y ago
    It doesn’t assume either of those things. In fact it accounts for exactly those points under the heading “incentives.” We are engaging in clearly self-destructive behavior because, incentive structure accounted for (your points), it very well appears to be the best option. It wouldn’t be a hard problem if it didn’t. It should be obvious that it’s entirely possible to follow your (real) incentives all the way into a very bad outcome.
    - coldtea 4y ago
      >We are engaging in clearly self-destructive behavior because, incentive structure accounted for (your points), it very well appears to be the best option Well, if it's the "best option" then it clearly isn't self-destructive... It's just something with side effects, but which also yielded huge benefits. Kind of how alcoholism is self-destructing. But a side-effect from taking a drug that helps us perform betetr or treats some condition we have, is not self-destructing. It's more of an unforseen issue, that we can't immediately correct (e.g. because of the condition worsening if we do it abruptly).

coldtea 4y ago

>1) Why do we have a reason to believe there’s a cap to intelligence? There's a cap to all things. Diminishing returns is a thing. Both physical substrates have limits, and there's a utility limit to thinking or having more data. Limitless intelligence is fairy-tale magical thinking - especially in the way it's portrayed as some magical ability to build and control everything. Like, invent working cold-fusion - as if you don't need the experiments, iterative process, random breakthroughs, and such, but just "smarter" thinking. > 2) No, you don’t need to assume it happens via adding hardware Yes, you do, else it's just the same capped-capability hardware, thinking itself into being smarter. > 3) No, you don’t need to assume it can expand by itself Well, I, personally, don't need to assume anything. But people talking about AGI, do assume it. > 4) We’ve already seen people connecting AI systems to open internet and open internet to command and control of physical systems (cars, utilities, soon weapons systems) - turns out there was no convincing necessary That's neither here nor there, though, because I didn't say that connecting AGI to the open internet is impossible or can't happen. Just that it doesn't mean much. We still control the kind of control we give an AI, and any physical systems any AGI would be connected will still have limitations. It's not some magical portal into conquering the world. >5) Nanobots also not a necessary assumption Didn't say it's necessary, just pointed out that it has been made - along with many other non necessary, and far-fetched assumptions and hand waving... >Also, incentives, not charisma, are what steer human behavior. Both do. With charisma you can do it without incentives - and even instill incentives into people. Otherwise you're stuck with innate incentives like people wanting to be rich - which you don't need to be a AGI to exploit, just rich or a con-man.

ethanbond 4y ago
1) Obviously intelligence plays a role in how fast science develops. The quality of experiments and the quality of inference from those experiments are (very obviously) modulated by intelligence. No reason to believe the absolute pinnacle of intelligence happens to sit inside a species of mostly hairless apes here on earth. Lots of reasons to believe fundamental understanding of the universe is more accessible to intelligent beings than to unintelligent ones. 2) Nope. Could be different hardware. The brain consumes far less power than any human-designed computer that could hope to match its computational power. 3) Sure some do, but it’s not a necessary assumption to take AGI risk seriously. 4) Meh, this seems to me just the next concession in the line of “we can’t do that, we wouldn’t do that, we do it but it’s not that bad” 5) Okay so it’s unrelated to this conversation except I suppose some sort of ad hominem? 6) So we agree charisma is at best not necessary to manipulate the world

bioemerl 4y ago

We are already made of nanobots, we are the nanobots, we are in intelligence, we are a general intelligence, and we are growing exponentially. AGI is here. AGI is you. You are an incredibly efficient piece of carbon thinkware able to sell for replicate into more carbon thinkware and that that extra carbon think where can go out and create its own life and make its own observations and spread information through the society of you both inherit, that's society is growing exponentially and growing in its ability to modify the world and manipulate and improve itself exponentially. If you're scared of AGI, don't be. You are the superior AGI, because these machines made of metal and circuits are never going to compare to machines made of carbon using chemical reactions that are far more efficient than transistors to do the thinking. The future is in improving ourselves and reaching a point at which our biology is no longer a mysterious black box so that we can modify and improve ourselves and those around us. We can already improve machines, but that's because the machines are far simpler and far more primitive than our minds are.

afiori 4y ago
Machines being simpler is the problem. The problems with AGI are 1) runaway growth and the singularity and 2) the scale and reach of its actions. Humans are deeply limited in their senses, their actions, and their ability to communicate. An AGI botnet installed on every computer/control system in the world would have none of those limitations.
- bioemerl 4y ago
  Complexity breeds ability. Our computers will have to continue to get more complex in order for AI to do the same. What humans have is scale. There are 10 billion devices connected to the internet. Cool, right? More than humans? No. Being human has really big perks. We are mobile. We have hands and feet. We have eyes and touch and smell and we use tools to expand what we can see. We each possess the ability to process data and filter data at scales many magnitudes greater than a computer. Note that data in this case isn't "many miles of spreadsheet data" - data in this case is worldly observation. You notice weird things, you report it to your friends. You try weird things, you report it to your friends. Communication between humans is limited, as is our memory, but it's limited for a reason. 99 percent of information is worthless junk human communication gets most of the important parts through. Then we need to parse mass amounts of data? Computers do that. Comparing your information to all other known information? It's an O n squared problem. You can throw hardware at it but as informations scope grows it will outpace the hardware eventually. Humans are crazy advanced compared to machines in basically every sense. I think the viewpoint here that machines will have superior senses is extrapolating how computers work today and is thoroughly impractical. We will find these logical and reproducible systems look more and more like we do as we push the line. I expect eventual marginal improvements that can't outpace human "plentifulness", and biological modifications and creations to be our real future.