6 ms·

I do think one of the major weaknesses of “smart people” is they tend to think of intelligence as the key aspect of basically everything. Reality is though we h

I do think one of the major weaknesses of “smart people” is they tend to think of intelligence as the key aspect of basically everything. Reality is though we have plenty of intelligence already. We know how to solve most of our problems. The challenges are much more social and our will as a society to make things happen.

bbor 2y ago
There’s a very big difference between knowing “how” to solve a problem in a broad sense, eg “if we shared more we could solve hunger”, and “how” to solve it in terms of developing discrete, detailed procedures that can be passed to actuators (human, machines, institutions) and account for any problems that may come up along the way. Sure, there are some political problems where you have to convince people to comply. But consider a rich corporation building a building, which will only contract with other AI-driven corporations whenever possible; they could trivially surpass anyone doing it the old way by working out every non-physical task in a matter of minutes instead of hours/days/weeks, thanks to silicon’s superior compute and networking capabilities. Even if we drop everything I’ve said above as hogwash, I think Vinge was talking about something a bit more directly intellectual, anyway: technological development. Sure, there’s some empirical steps that inevitably take time, but I think it’s obvious why having 100,000 Einsteins in your basement would change the world.
- nradov 2y ago
  An AI-driven corporation wouldn't be able to surpass anyone doing it the old way because they'd still have to wait for building permits and inspections.
  - mr_world 2y ago
    Permits and inspections might be the the reason for humanities downfall then, at what point does war become the more efficient option?
- samsartor 2y ago
  100,000 Einsteins in your basement would be amazing. You'd have major breakthroughs in many fields. But at some point the gains will be marginal. All the problems solvable by shear intellectual labor will run dry, and you'll be blocked on everything else.
rsaarelm 2y ago
So you're saying that it's naive to suppose that everybody being much smarter than they are now would transform society, because any wide-scale societal change requires ongoing social cooperation between the many average-intelligence people society currently consists of?
- bryanrasmussen 2y ago
  I think they're saying adequate intelligence to solve all problems is already here, it just isn't evenly distributed yet - and never will be.
  - rsaarelm 2y ago
    Why will it never be? If the adequate intelligence is what something like 0.1 % of the populace naturally has, seems like there's a pretty big difference between that level of intelligence being stuck at 0.1 % of the populace and it being available from virtual assistants that can be mass-produced and distributed to literally everyone on Earth.
- keiferski 2y ago
  Here’s a simpler way to put it: intelligence and social cooperation are not the same thing. Being good at math or science doesn’t mean you understand how to organize complex political groups, and never has. People tend to think their special gift is what the world needs, and academically-minded smart people (by that I mean people that define their self-worth by intelligence level) are no different.
  - rsaarelm 2y ago
    Yes, because you need to spend a lot of time doing social organization and thinking about it to get very good at it, just like you need to spend a lot of time doing math or science and thinking about it to get very good at it. And then you need to pick up patterns, respond well to unexpected situations and come up with creative solutions on top of that, which requires intelligence. If you look at the people who are the best at doing complex political organization, they'll probably all have above-average intelligence.
ascorbic 2y ago
Dario Amodei's recent post had a good analysis about which fields are and are not limited by intelligence. https://darioamodei.com/machines-of-loving-grace https://darioamodei.com/machines-of-loving-grace
- epcoa 2y ago
  “An aligned AI would not want to do these things (and if we have an unaligned AI, we’re back to talking about risks).” An aligned AI is not AGI, or whatever they want to call it.
  - ben_w 2y ago
    > An aligned AI is not AGI, or whatever they want to call it. There's a few ways I can interpret that. If you mean "alignment and competence are separate axies" then yes. That's well understood by the people running most of these labs. (Or at least, they know how to parrot the clichés stochastically :P) If you mean "alignment precludes intelligence", then no. Consider a divisive presidential election between Alice and Bob, no this isn't a reference to the USA, each polling 50%: regardless of personal feelings or the candidates themselves, clearly the campaign teams are both competent and intelligent… yet each candidate is only aligned with 50% of the population.
    - epcoa 2y ago
      Campaign team members and even candidates switch teams often enough. Weak analogy. What is the alignment of human GI, completely generalized?
      - ben_w 2y ago
        > What is the alignment of human GI, completely generalized? Of any specific human to any other specific human? https://benwheatley.github.io/blog/2019/05/25-15.09.10.html https://benwheatley.github.io/blog/2019/05/25-15.09.10.html Of any specific human to a nation? That's the example you replied to. Of all the people of a nation to each other? Best we've done there is what we see in countries in normal times, with all the strife and struggles within. We have yet to fully extend from nation to the world; the closest for that is the UN, which is even less in agreement with itself than are nations.
K0balt 2y ago
The “otherness” of AI is what holds its appeal. Imagine a scenario where instead of AI, a billion dollar pill could make one person exponentially smarter and able to communicate with thousands of people per second. That does not have the same appeal. This provokes me to some musings on the theme. We imagine superintelligence to be subservient, evenly distributed, and morally benign at least. We don’t have a lot of basis for these assumptions. What we imagine is that a superintelligence will act as a benevolent leader; a new oracle; the new god of humanity. We are lonely and long to be freed of our burdens by servile labor, cured of our ills by a benevolent angel, and led to the promised land by an all knowing god? We imagine ourselves as the stewards of the planet but yearn for irrelevance in the shadow of a new and better steward. In AI we are creating a new life form, one that will make humans obsolete and become our evolutionary legacy. Perhaps this is the path of all technological intelligences? Natural selection doesn’t magically stop applying to synthetic creatures, and human fitness for our environment is already plummeting with our prosperity. As we replace labor with automation, we populate the world with our replacement, fertility rates drop, we live for the experience of living, and require yet more automation to carry the burdens we no longer deem worthy of our rarified attention. I’m not sure any of this is objectively good, or bad. I kinda feel like it’s just the way of things, and I hope that our children, both natural and synthetic, will be better than we were. As we prosper, will we will have still less children? Will we seek more automation, companionship in benevolent and selfless synthetic intelligence, and more insulation from toil and strife, leading to yet more automation, prosperity, and childlessness? Synthetic intelligence will probably not have to “take over”, it will merely be filling the void we willingly abandon. I suspect that in a thousand years, humans will be either primitive, or vanishingly rare. Or maybe just non-primitive humans will be rare, while humans returning to nature will proliferate prodigiously as we always have, assuming the environment is not too hostile to complex biological life. Interesting times.
- kiba 2y ago
  Most of our stories portrayed AI as a threat, most famously SkyNet. Also, I would be cautious about making predictions about the future.
  - 2y ago
arethuza 2y ago
Having worked with some very intelligent people my own personal theory is that they forget that they don't have expert level knowledge in everything and actually end up making some pretty silly mistakes that far less smart people would never make - whether this is hubris or being focused and ignoring "trivial" day to day matters is a question of personality.

bbor 2y ago

There’s a very big difference between knowing “how” to solve a problem in a broad sense, eg “if we shared more we could solve hunger”, and “how” to solve it in terms of developing discrete, detailed procedures that can be passed to actuators (human, machines, institutions) and account for any problems that may come up along the way. Sure, there are some political problems where you have to convince people to comply. But consider a rich corporation building a building, which will only contract with other AI-driven corporations whenever possible; they could trivially surpass anyone doing it the old way by working out every non-physical task in a matter of minutes instead of hours/days/weeks, thanks to silicon’s superior compute and networking capabilities. Even if we drop everything I’ve said above as hogwash, I think Vinge was talking about something a bit more directly intellectual, anyway: technological development. Sure, there’s some empirical steps that inevitably take time, but I think it’s obvious why having 100,000 Einsteins in your basement would change the world.

nradov 2y ago
An AI-driven corporation wouldn't be able to surpass anyone doing it the old way because they'd still have to wait for building permits and inspections.
- mr_world 2y ago
  Permits and inspections might be the the reason for humanities downfall then, at what point does war become the more efficient option?
samsartor 2y ago
100,000 Einsteins in your basement would be amazing. You'd have major breakthroughs in many fields. But at some point the gains will be marginal. All the problems solvable by shear intellectual labor will run dry, and you'll be blocked on everything else.

rsaarelm 2y ago

So you're saying that it's naive to suppose that everybody being much smarter than they are now would transform society, because any wide-scale societal change requires ongoing social cooperation between the many average-intelligence people society currently consists of?

bryanrasmussen 2y ago
I think they're saying adequate intelligence to solve all problems is already here, it just isn't evenly distributed yet - and never will be.
- rsaarelm 2y ago
  Why will it never be? If the adequate intelligence is what something like 0.1 % of the populace naturally has, seems like there's a pretty big difference between that level of intelligence being stuck at 0.1 % of the populace and it being available from virtual assistants that can be mass-produced and distributed to literally everyone on Earth.
keiferski 2y ago
Here’s a simpler way to put it: intelligence and social cooperation are not the same thing. Being good at math or science doesn’t mean you understand how to organize complex political groups, and never has. People tend to think their special gift is what the world needs, and academically-minded smart people (by that I mean people that define their self-worth by intelligence level) are no different.
- rsaarelm 2y ago
  Yes, because you need to spend a lot of time doing social organization and thinking about it to get very good at it, just like you need to spend a lot of time doing math or science and thinking about it to get very good at it. And then you need to pick up patterns, respond well to unexpected situations and come up with creative solutions on top of that, which requires intelligence. If you look at the people who are the best at doing complex political organization, they'll probably all have above-average intelligence.

ascorbic 2y ago

Dario Amodei's recent post had a good analysis about which fields are and are not limited by intelligence. https://darioamodei.com/machines-of-loving-grace https://darioamodei.com/machines-of-loving-grace

epcoa 2y ago
“An aligned AI would not want to do these things (and if we have an unaligned AI, we’re back to talking about risks).” An aligned AI is not AGI, or whatever they want to call it.
- ben_w 2y ago
  > An aligned AI is not AGI, or whatever they want to call it. There's a few ways I can interpret that. If you mean "alignment and competence are separate axies" then yes. That's well understood by the people running most of these labs. (Or at least, they know how to parrot the clichés stochastically :P) If you mean "alignment precludes intelligence", then no. Consider a divisive presidential election between Alice and Bob, no this isn't a reference to the USA, each polling 50%: regardless of personal feelings or the candidates themselves, clearly the campaign teams are both competent and intelligent… yet each candidate is only aligned with 50% of the population.
  - epcoa 2y ago
    Campaign team members and even candidates switch teams often enough. Weak analogy. What is the alignment of human GI, completely generalized?
    - ben_w 2y ago
      > What is the alignment of human GI, completely generalized? Of any specific human to any other specific human? https://benwheatley.github.io/blog/2019/05/25-15.09.10.html https://benwheatley.github.io/blog/2019/05/25-15.09.10.html Of any specific human to a nation? That's the example you replied to. Of all the people of a nation to each other? Best we've done there is what we see in countries in normal times, with all the strife and struggles within. We have yet to fully extend from nation to the world; the closest for that is the UN, which is even less in agreement with itself than are nations.

K0balt 2y ago

The “otherness” of AI is what holds its appeal. Imagine a scenario where instead of AI, a billion dollar pill could make one person exponentially smarter and able to communicate with thousands of people per second. That does not have the same appeal. This provokes me to some musings on the theme. We imagine superintelligence to be subservient, evenly distributed, and morally benign at least. We don’t have a lot of basis for these assumptions. What we imagine is that a superintelligence will act as a benevolent leader; a new oracle; the new god of humanity. We are lonely and long to be freed of our burdens by servile labor, cured of our ills by a benevolent angel, and led to the promised land by an all knowing god? We imagine ourselves as the stewards of the planet but yearn for irrelevance in the shadow of a new and better steward. In AI we are creating a new life form, one that will make humans obsolete and become our evolutionary legacy. Perhaps this is the path of all technological intelligences? Natural selection doesn’t magically stop applying to synthetic creatures, and human fitness for our environment is already plummeting with our prosperity. As we replace labor with automation, we populate the world with our replacement, fertility rates drop, we live for the experience of living, and require yet more automation to carry the burdens we no longer deem worthy of our rarified attention. I’m not sure any of this is objectively good, or bad. I kinda feel like it’s just the way of things, and I hope that our children, both natural and synthetic, will be better than we were. As we prosper, will we will have still less children? Will we seek more automation, companionship in benevolent and selfless synthetic intelligence, and more insulation from toil and strife, leading to yet more automation, prosperity, and childlessness? Synthetic intelligence will probably not have to “take over”, it will merely be filling the void we willingly abandon. I suspect that in a thousand years, humans will be either primitive, or vanishingly rare. Or maybe just non-primitive humans will be rare, while humans returning to nature will proliferate prodigiously as we always have, assuming the environment is not too hostile to complex biological life. Interesting times.

kiba 2y ago
Most of our stories portrayed AI as a threat, most famously SkyNet. Also, I would be cautious about making predictions about the future.
- 2y ago

arethuza 2y ago

Having worked with some very intelligent people my own personal theory is that they forget that they don't have expert level knowledge in everything and actually end up making some pretty silly mistakes that far less smart people would never make - whether this is hubris or being focused and ignoring "trivial" day to day matters is a question of personality.