8 ms·

> We humans have the ability to internalize the world and conduct "what if's" in our heads; we can solve many problems thousands of times faster than natural se

> We humans have the ability to internalize the world and conduct "what if's" in our heads; we can solve many problems thousands of times faster than natural selection. I don't know for sure whether superintellegence will happen, but as for the singularity, this is the underlying assumption I have the most issue with. Smart isn't the limiting factor of progress, often it's building consensus, getting funding, waiting for results, waiting for parts to ship, waiting for the right opportunity to come along. We do _experiments_ faster than natural selection, but we still have to do them in the real world. Solving problems happens on the lab bench, not just in our heads. Even if exponentially more intelligent machines get built, what's to stop the next problem on the road to progress being exponentially harder? Complexity cuts both ways.

trescenzi 2y ago
I do think one of the major weaknesses of “smart people” is they tend to think of intelligence as the key aspect of basically everything. Reality is though we have plenty of intelligence already. We know how to solve most of our problems. The challenges are much more social and our will as a society to make things happen.
- bbor 2y ago
  There’s a very big difference between knowing “how” to solve a problem in a broad sense, eg “if we shared more we could solve hunger”, and “how” to solve it in terms of developing discrete, detailed procedures that can be passed to actuators (human, machines, institutions) and account for any problems that may come up along the way. Sure, there are some political problems where you have to convince people to comply. But consider a rich corporation building a building, which will only contract with other AI-driven corporations whenever possible; they could trivially surpass anyone doing it the old way by working out every non-physical task in a matter of minutes instead of hours/days/weeks, thanks to silicon’s superior compute and networking capabilities. Even if we drop everything I’ve said above as hogwash, I think Vinge was talking about something a bit more directly intellectual, anyway: technological development. Sure, there’s some empirical steps that inevitably take time, but I think it’s obvious why having 100,000 Einsteins in your basement would change the world.
  - nradov 2y ago
    An AI-driven corporation wouldn't be able to surpass anyone doing it the old way because they'd still have to wait for building permits and inspections.
    - mr_world 2y ago
      Permits and inspections might be the the reason for humanities downfall then, at what point does war become the more efficient option?
  - samsartor 2y ago
    100,000 Einsteins in your basement would be amazing. You'd have major breakthroughs in many fields. But at some point the gains will be marginal. All the problems solvable by shear intellectual labor will run dry, and you'll be blocked on everything else.
- rsaarelm 2y ago
  So you're saying that it's naive to suppose that everybody being much smarter than they are now would transform society, because any wide-scale societal change requires ongoing social cooperation between the many average-intelligence people society currently consists of?
  - bryanrasmussen 2y ago
    I think they're saying adequate intelligence to solve all problems is already here, it just isn't evenly distributed yet - and never will be.
    - rsaarelm 2y ago
      Why will it never be? If the adequate intelligence is what something like 0.1 % of the populace naturally has, seems like there's a pretty big difference between that level of intelligence being stuck at 0.1 % of the populace and it being available from virtual assistants that can be mass-produced and distributed to literally everyone on Earth.
  - keiferski 2y ago
    Here’s a simpler way to put it: intelligence and social cooperation are not the same thing. Being good at math or science doesn’t mean you understand how to organize complex political groups, and never has. People tend to think their special gift is what the world needs, and academically-minded smart people (by that I mean people that define their self-worth by intelligence level) are no different.
    - rsaarelm 2y ago
      Yes, because you need to spend a lot of time doing social organization and thinking about it to get very good at it, just like you need to spend a lot of time doing math or science and thinking about it to get very good at it. And then you need to pick up patterns, respond well to unexpected situations and come up with creative solutions on top of that, which requires intelligence. If you look at the people who are the best at doing complex political organization, they'll probably all have above-average intelligence.
- ascorbic 2y ago
  Dario Amodei's recent post had a good analysis about which fields are and are not limited by intelligence. https://darioamodei.com/machines-of-loving-grace https://darioamodei.com/machines-of-loving-grace
  - epcoa 2y ago
    “An aligned AI would not want to do these things (and if we have an unaligned AI, we’re back to talking about risks).” An aligned AI is not AGI, or whatever they want to call it.
    - ben_w 2y ago
      > An aligned AI is not AGI, or whatever they want to call it. There's a few ways I can interpret that. If you mean "alignment and competence are separate axies" then yes. That's well understood by the people running most of these labs. (Or at least, they know how to parrot the clichés stochastically :P) If you mean "alignment precludes intelligence", then no. Consider a divisive presidential election between Alice and Bob, no this isn't a reference to the USA, each polling 50%: regardless of personal feelings or the candidates themselves, clearly the campaign teams are both competent and intelligent… yet each candidate is only aligned with 50% of the population.
      - epcoa 2y ago
        Campaign team members and even candidates switch teams often enough. Weak analogy. What is the alignment of human GI, completely generalized?
        ben_w 2y ago
        > What is the alignment of human GI, completely generalized? Of any specific human to any other specific human? https://benwheatley.github.io/blog/2019/05/25-15.09.10.html https://benwheatley.github.io/blog/2019/05/25-15.09.10.html Of any specific human to a nation? That's the example you replied to. Of all the people of a nation to each other? Best we've done there is what we see in countries in normal times, with all the strife and struggles within. We have yet to fully extend from nation to the world; the closest for that is the UN, which is even less in agreement with itself than are nations.
- K0balt 2y ago
  The “otherness” of AI is what holds its appeal. Imagine a scenario where instead of AI, a billion dollar pill could make one person exponentially smarter and able to communicate with thousands of people per second. That does not have the same appeal. This provokes me to some musings on the theme. We imagine superintelligence to be subservient, evenly distributed, and morally benign at least. We don’t have a lot of basis for these assumptions. What we imagine is that a superintelligence will act as a benevolent leader; a new oracle; the new god of humanity. We are lonely and long to be freed of our burdens by servile labor, cured of our ills by a benevolent angel, and led to the promised land by an all knowing god? We imagine ourselves as the stewards of the planet but yearn for irrelevance in the shadow of a new and better steward. In AI we are creating a new life form, one that will make humans obsolete and become our evolutionary legacy. Perhaps this is the path of all technological intelligences? Natural selection doesn’t magically stop applying to synthetic creatures, and human fitness for our environment is already plummeting with our prosperity. As we replace labor with automation, we populate the world with our replacement, fertility rates drop, we live for the experience of living, and require yet more automation to carry the burdens we no longer deem worthy of our rarified attention. I’m not sure any of this is objectively good, or bad. I kinda feel like it’s just the way of things, and I hope that our children, both natural and synthetic, will be better than we were. As we prosper, will we will have still less children? Will we seek more automation, companionship in benevolent and selfless synthetic intelligence, and more insulation from toil and strife, leading to yet more automation, prosperity, and childlessness? Synthetic intelligence will probably not have to “take over”, it will merely be filling the void we willingly abandon. I suspect that in a thousand years, humans will be either primitive, or vanishingly rare. Or maybe just non-primitive humans will be rare, while humans returning to nature will proliferate prodigiously as we always have, assuming the environment is not too hostile to complex biological life. Interesting times.
  - kiba 2y ago
    Most of our stories portrayed AI as a threat, most famously SkyNet. Also, I would be cautious about making predictions about the future.
    - 2y ago
- arethuza 2y ago
  Having worked with some very intelligent people my own personal theory is that they forget that they don't have expert level knowledge in everything and actually end up making some pretty silly mistakes that far less smart people would never make - whether this is hubris or being focused and ignoring "trivial" day to day matters is a question of personality.
szundi 2y ago
Skynet will build consensus killing people until only one remains. It agrees with himself isn’t it? Oh, he expressed some concerns about the result? Sadly the instance is faulty recognizing the obvious, terminated.
- AtlasBarfed 2y ago
  CAP ensures there will be a partition and disagreement. Like the Argentinian ant invading the world that eventually diverged just enough to start warring with itself
trashtester 2y ago
AlphaFold/AlphaProteo are direct examples of how AI can allow us to bypass experiments when doing science. More importantly, though, is that insufficiently powerful AI has been a limiting factor in robotics. That seems to be coming to an end now. And once we have humanoid robots powered by superhuman intelligence entering the workforce, the impact will be massive. Quite possibly mostly for the bad for most people.
- arsenico 2y ago
  In reality though, do we actually need humanoid robots, and if so, for what?
  - ben_w 2y ago
    Need? No, absolutely not. But they do conveniently fit into the century old buildings we put many of the factories into, which makes them a useful upgrade path for those unwilling to build structures around more efficient robots (the kind we've had for ages and don't even think of as robots, they just take ingredients and pump out packaged candy or pencils etc.)
    - shermantanktop 2y ago
      There are incredible technological barriers to humanoid robots who have equivalent skills and stamina. Keeping old factories running seems a very weak reason to do that, when our industrial base regularly retools production methods and brings in new equipment when old machines wear out. If what you are saying is that many factories cannot run with humans running around fixing things, I agree. But that’s pretty different than using humanoids to put items in boxes.
      - ben_w 2y ago
        Yes indeed. Even just 3.5 years ago, seemed like everyone was saying that humanoid robots were a dead end or an unnecessary part of Isaac Asimov's vision of the future or similar. I think much of the current interest is because Musk watched some scifi, ordered the Optimus project, and loads of others decided it would be a mistake to bet against him. I put them in the same category as 3D printers: they can do anything, but you can always find a better special-purpose alternative for any specific goal. Still a lot of people using 3D printing productively despite that; likely also will be for humanoid robots. Well, if the AI is good enough. Remote control has its uses, but even then you need enough on-board AI to avoid playing QUOP as live action with a robot holding industrial equipment instead of in a safe flash game. That said, stamina is probably the least important aspect — in an industrial setting you probably have a lot of power lines already installed.
imglorp 2y ago
I guess it's not shocking how fast humanity formed consensus this time or why. Some say it's going to be a trillion dollar market by 2027.
mcnamaratw 2y ago
Good point, your suppliers need to get singular too. And sales. And management or investors.
- nsbshssh 2y ago
  Like there wont be people hooking up their LLMs to all that stuff, and more, because doing irresponsible things is seen as futurism and investors love it.

trescenzi 2y ago

I do think one of the major weaknesses of “smart people” is they tend to think of intelligence as the key aspect of basically everything. Reality is though we have plenty of intelligence already. We know how to solve most of our problems. The challenges are much more social and our will as a society to make things happen.

bbor 2y ago
There’s a very big difference between knowing “how” to solve a problem in a broad sense, eg “if we shared more we could solve hunger”, and “how” to solve it in terms of developing discrete, detailed procedures that can be passed to actuators (human, machines, institutions) and account for any problems that may come up along the way. Sure, there are some political problems where you have to convince people to comply. But consider a rich corporation building a building, which will only contract with other AI-driven corporations whenever possible; they could trivially surpass anyone doing it the old way by working out every non-physical task in a matter of minutes instead of hours/days/weeks, thanks to silicon’s superior compute and networking capabilities. Even if we drop everything I’ve said above as hogwash, I think Vinge was talking about something a bit more directly intellectual, anyway: technological development. Sure, there’s some empirical steps that inevitably take time, but I think it’s obvious why having 100,000 Einsteins in your basement would change the world.
- nradov 2y ago
  An AI-driven corporation wouldn't be able to surpass anyone doing it the old way because they'd still have to wait for building permits and inspections.
  - mr_world 2y ago
    Permits and inspections might be the the reason for humanities downfall then, at what point does war become the more efficient option?
- samsartor 2y ago
  100,000 Einsteins in your basement would be amazing. You'd have major breakthroughs in many fields. But at some point the gains will be marginal. All the problems solvable by shear intellectual labor will run dry, and you'll be blocked on everything else.
rsaarelm 2y ago
So you're saying that it's naive to suppose that everybody being much smarter than they are now would transform society, because any wide-scale societal change requires ongoing social cooperation between the many average-intelligence people society currently consists of?
- bryanrasmussen 2y ago
  I think they're saying adequate intelligence to solve all problems is already here, it just isn't evenly distributed yet - and never will be.
  - rsaarelm 2y ago
    Why will it never be? If the adequate intelligence is what something like 0.1 % of the populace naturally has, seems like there's a pretty big difference between that level of intelligence being stuck at 0.1 % of the populace and it being available from virtual assistants that can be mass-produced and distributed to literally everyone on Earth.
- keiferski 2y ago
  Here’s a simpler way to put it: intelligence and social cooperation are not the same thing. Being good at math or science doesn’t mean you understand how to organize complex political groups, and never has. People tend to think their special gift is what the world needs, and academically-minded smart people (by that I mean people that define their self-worth by intelligence level) are no different.
  - rsaarelm 2y ago
    Yes, because you need to spend a lot of time doing social organization and thinking about it to get very good at it, just like you need to spend a lot of time doing math or science and thinking about it to get very good at it. And then you need to pick up patterns, respond well to unexpected situations and come up with creative solutions on top of that, which requires intelligence. If you look at the people who are the best at doing complex political organization, they'll probably all have above-average intelligence.
ascorbic 2y ago
Dario Amodei's recent post had a good analysis about which fields are and are not limited by intelligence. https://darioamodei.com/machines-of-loving-grace https://darioamodei.com/machines-of-loving-grace
- epcoa 2y ago
  “An aligned AI would not want to do these things (and if we have an unaligned AI, we’re back to talking about risks).” An aligned AI is not AGI, or whatever they want to call it.
  - ben_w 2y ago
    > An aligned AI is not AGI, or whatever they want to call it. There's a few ways I can interpret that. If you mean "alignment and competence are separate axies" then yes. That's well understood by the people running most of these labs. (Or at least, they know how to parrot the clichés stochastically :P) If you mean "alignment precludes intelligence", then no. Consider a divisive presidential election between Alice and Bob, no this isn't a reference to the USA, each polling 50%: regardless of personal feelings or the candidates themselves, clearly the campaign teams are both competent and intelligent… yet each candidate is only aligned with 50% of the population.
    - epcoa 2y ago
      Campaign team members and even candidates switch teams often enough. Weak analogy. What is the alignment of human GI, completely generalized?
      - ben_w 2y ago
        > What is the alignment of human GI, completely generalized? Of any specific human to any other specific human? https://benwheatley.github.io/blog/2019/05/25-15.09.10.html https://benwheatley.github.io/blog/2019/05/25-15.09.10.html Of any specific human to a nation? That's the example you replied to. Of all the people of a nation to each other? Best we've done there is what we see in countries in normal times, with all the strife and struggles within. We have yet to fully extend from nation to the world; the closest for that is the UN, which is even less in agreement with itself than are nations.
K0balt 2y ago
The “otherness” of AI is what holds its appeal. Imagine a scenario where instead of AI, a billion dollar pill could make one person exponentially smarter and able to communicate with thousands of people per second. That does not have the same appeal. This provokes me to some musings on the theme. We imagine superintelligence to be subservient, evenly distributed, and morally benign at least. We don’t have a lot of basis for these assumptions. What we imagine is that a superintelligence will act as a benevolent leader; a new oracle; the new god of humanity. We are lonely and long to be freed of our burdens by servile labor, cured of our ills by a benevolent angel, and led to the promised land by an all knowing god? We imagine ourselves as the stewards of the planet but yearn for irrelevance in the shadow of a new and better steward. In AI we are creating a new life form, one that will make humans obsolete and become our evolutionary legacy. Perhaps this is the path of all technological intelligences? Natural selection doesn’t magically stop applying to synthetic creatures, and human fitness for our environment is already plummeting with our prosperity. As we replace labor with automation, we populate the world with our replacement, fertility rates drop, we live for the experience of living, and require yet more automation to carry the burdens we no longer deem worthy of our rarified attention. I’m not sure any of this is objectively good, or bad. I kinda feel like it’s just the way of things, and I hope that our children, both natural and synthetic, will be better than we were. As we prosper, will we will have still less children? Will we seek more automation, companionship in benevolent and selfless synthetic intelligence, and more insulation from toil and strife, leading to yet more automation, prosperity, and childlessness? Synthetic intelligence will probably not have to “take over”, it will merely be filling the void we willingly abandon. I suspect that in a thousand years, humans will be either primitive, or vanishingly rare. Or maybe just non-primitive humans will be rare, while humans returning to nature will proliferate prodigiously as we always have, assuming the environment is not too hostile to complex biological life. Interesting times.
- kiba 2y ago
  Most of our stories portrayed AI as a threat, most famously SkyNet. Also, I would be cautious about making predictions about the future.
  - 2y ago
arethuza 2y ago
Having worked with some very intelligent people my own personal theory is that they forget that they don't have expert level knowledge in everything and actually end up making some pretty silly mistakes that far less smart people would never make - whether this is hubris or being focused and ignoring "trivial" day to day matters is a question of personality.

szundi 2y ago

Skynet will build consensus killing people until only one remains. It agrees with himself isn’t it? Oh, he expressed some concerns about the result? Sadly the instance is faulty recognizing the obvious, terminated.

AtlasBarfed 2y ago
CAP ensures there will be a partition and disagreement. Like the Argentinian ant invading the world that eventually diverged just enough to start warring with itself

trashtester 2y ago

AlphaFold/AlphaProteo are direct examples of how AI can allow us to bypass experiments when doing science. More importantly, though, is that insufficiently powerful AI has been a limiting factor in robotics. That seems to be coming to an end now. And once we have humanoid robots powered by superhuman intelligence entering the workforce, the impact will be massive. Quite possibly mostly for the bad for most people.

arsenico 2y ago
In reality though, do we actually need humanoid robots, and if so, for what?
- ben_w 2y ago
  Need? No, absolutely not. But they do conveniently fit into the century old buildings we put many of the factories into, which makes them a useful upgrade path for those unwilling to build structures around more efficient robots (the kind we've had for ages and don't even think of as robots, they just take ingredients and pump out packaged candy or pencils etc.)
  - shermantanktop 2y ago
    There are incredible technological barriers to humanoid robots who have equivalent skills and stamina. Keeping old factories running seems a very weak reason to do that, when our industrial base regularly retools production methods and brings in new equipment when old machines wear out. If what you are saying is that many factories cannot run with humans running around fixing things, I agree. But that’s pretty different than using humanoids to put items in boxes.
    - ben_w 2y ago
      Yes indeed. Even just 3.5 years ago, seemed like everyone was saying that humanoid robots were a dead end or an unnecessary part of Isaac Asimov's vision of the future or similar. I think much of the current interest is because Musk watched some scifi, ordered the Optimus project, and loads of others decided it would be a mistake to bet against him. I put them in the same category as 3D printers: they can do anything, but you can always find a better special-purpose alternative for any specific goal. Still a lot of people using 3D printing productively despite that; likely also will be for humanoid robots. Well, if the AI is good enough. Remote control has its uses, but even then you need enough on-board AI to avoid playing QUOP as live action with a robot holding industrial equipment instead of in a safe flash game. That said, stamina is probably the least important aspect — in an industrial setting you probably have a lot of power lines already installed.

imglorp 2y ago

I guess it's not shocking how fast humanity formed consensus this time or why. Some say it's going to be a trillion dollar market by 2027.

mcnamaratw 2y ago

Good point, your suppliers need to get singular too. And sales. And management or investors.

nsbshssh 2y ago
Like there wont be people hooking up their LLMs to all that stuff, and more, because doing irresponsible things is seen as futurism and investors love it.