5 ms·

I couldn't read the whole article, but just from the part I could read: > Strikingly, this concern is being openly voiced by the very people who have the stron

I couldn't read the whole article, but just from the part I could read: > Strikingly, this concern is being openly voiced by the very people who have the strongest incentives to project confidence rather than alarm: the founders of the largest ai laboratories. I don't know, they also have an incentive to make their technology seem transformative and powerful, and saying that your technology has the power to cause a massive catastrophe is a way to promote that idea.

deleted 4mo ago
[deleted]
holmesworcester 4mo ago
https://archive.ph/2OWwO https://archive.ph/2OWwO
holmesworcester 4mo ago
If AI is now ascending an economic learning curve from: 1. Extremely useful (Claude Code & Waymo now) 2. Doing ~everything we do (AGI & Optimus in a few years? 10?) 3. RSI (?) 4. Being smarter than any living person at every intellectual task (?) 5. Being smarter than the best-organized aggregate of all humans (10-100 years?) ...And all of the scientific and resource-allocation institutions that brought us the computer and the second half of the 20th century are now fixated on this learning curve, what universe can we possibly imagine where this is not transformative and powerful? Honestly the only one I can think of is one in which we kill almost everyone in some other way first, and contrary to what you read in the news, almost everyone dying is not what the trend line has been from existing problems like war, disease, or even climate change. Also, just to pre-empt a common quibble: when I say "AI" I mean the set of all AI and their combined decision vector, not any one AI, so conflicting interests within the set of AI's will not save anyone any more than the conflicting interests of colonizers saved indigenous Americans.
- notarobot123 4mo ago
  > Being smarter than the best-organized aggregate of all humans Isn't there pretty much a consensus that committees and institutions are not all that smart? I think you're confusing the categories of "intelligence" and power. Institutions are powerful. The smartest AI is still just a tool without the infrastructure to turn that into real world effects and someone to direct it. It seems you have faith that this is inevitable and unavoidable. I get it, even rationalists succumb to religious thinking eventually. We're only human after all.
  - wartywhoa23 4mo ago
    I'm genuinely baffled each time I encounter this notion that the power is dumb. It's as if power was a kind of demigod Heracles who paves its way with sheer brute force alone. But in reality it consists of very mortal and feeble creatures of meat and bone. How in the world can people assume dumb those who possess all the means and rights for violence by controlling and manipulating myriads of other creatures of meat and bone, is totally beyond me.
    - notarobot123 4mo ago
      Power and intelligence are not completely unrelated. Are government officials, politicians, CEOs, judges, etc, usually above average intelligence? Probably. Do the institutions they work within make intelligent decisions? Sometimes. The fallacy is assuming intelligence automatically equates to power and influence. University professors and doctors have very little power in their institutions even though they may often be more intelligent than their bosses. A worse fallacy is assuming intelligence creates better political outcomes (in the broad sense of politics as contested collective decision-making). I don't know many people who would willingly trade their freedom to make a bad decision for the enforcement of the "right" decision by a superior intelligence.
- consumer451 4mo ago
  Here is an issue I think about often, but I am not quite sure how to put it into words. We have many extremely smart people in various fields. Executives, politicians, and society generally ignore them and do whatever they want. I don't believe that lack of access to intelligence is our problem. How is "free" intelligence going to improve this? I don't just mean climate, but business planning, health, risk assessment, everything.
  - zzgo 4mo ago
    There are lots of incentives for politicians and executives (and anyone else holding the levers of power) to ignore information, intelligence, and advice. I think you're right to be skeptical that "free" intelligence is going to improve anything without first addressing the incentives of the people holding power.
  - fsflover 4mo ago
    The free intelligence may help to steer the society and politicians to one or the other side. The only question is who decides which.
    - wartywhoa23 4mo ago
      No invention was able to steer the psychopaths away from waging wars. But this time it's absolutely different, right? Riiight?..
      - fsflover 4mo ago
        Don't we have now much less wars than before, in general?
        wartywhoa23 4mo ago
        No, and we have wars no one in their sane mind would even imagine 20 years ago, like that of Russia vs Ukraine. See, it's easy to speculate how there are less wars when you live in a place which haven't seen war for decades or cenuries, but it's a complete game changer when it's 150 km from where you live, and it's not just some regular war, but a long play intentional meatgrinding AI drone debugging polygon.
- protocolture 4mo ago
  What makes you think you can assume that it will continue geometrically forever?
petre 4mo ago
> and saying that your technology has the power to cause a massive catastrophe Possibly, but by using up resources, not by its output. Its draw on resources will definitely accelerate climate change to create slop.
Nevermark 4mo ago
> an incentive to make their technology seem transformative and powerful I have no frame of reference to process this. Humans species perhaps 300,000 years, we are essentially the same. Transistor, 79 years, explosive growth in numbers and power. Integrated circuit, 68 years, explosive growth in numbers and power. "Attention is all you need", 9 years. ChatGPT, 4 years, explosive growth in instances and power. Humans species, not getting smarter. [0,1,2,3] There is a syndrome where many people seem unable to perceive or reason about rates of change in technology. We are going to spend the vast majority of our future lives without the intelligence crown. In terms of verbally expressible knowledge, models are passing many people completely, and passing all of our individually respectively weak reasoning areas. Other modalities are progressing very quickly. There will be short periods where progress happens quickly, but the impact feels slow. There will be radical changes that feel slow too, because if something anticipated or important isn't instant, we tend to perceive it as slow. But it won't be slow. And it won't be long. We are smart in a kind of pick the-best-of-us at the-best-of-times way. We are rarely consistently or broadly smart individually. We are not in the same galaxy as "ready". What would that look like? [0] https://en.wikipedia.org/wiki/Human https://en.wikipedia.org/wiki/Human [1] https://en.wikipedia.org/wiki/Transistor https://en.wikipedia.org/wiki/Transistor [2] https://en.wikipedia.org/wiki/Integrated_circuit https://en.wikipedia.org/wiki/Integrated_circuit [3] https://en.wikipedia.org/wiki/Large_language_model https://en.wikipedia.org/wiki/Large_language_model
- dbspin 4mo ago
  Agree with most of this, it's well articulated and captures how we react to change. However - 'Humans species perhaps 300,000 years, we are essentially the same' is an enormous misconception. That fact that we lack gross anatomical changes during this period, ignores everything we now know about punctuated equilibrium and rapid evolution. It's highly probably we've had an enormous number of evolved psychological changes during the last few hundred, and even tens of thousands of years. Changes that relate to our capacity to live in large groups, adapt to urban environments, resist disease and so on. We know that's the case simply because acute pandemics become epidemics through herd immunity, and through the acquisition of lactose tolerance etc. It seems highly unlikely that adaptations stop there. Altering the environment (in the last 10K years that means the built environment) alters the environment of evolutionary adaptiveness. It seems likely that we've essentially domesticated out much of our propensity for violence and increased our capacity for mood regulation. Obviously it's incredibly tricky to pair these specific behavioural changes to genetic changes -> protein synthesis -> behaviour. Bearing in mind though we're only 20 years out from the first study to link allele variant to behaviour (the COMT Val/Met polymorphism), and the potential controversy around such research, this shouldn't be surprising.

deleted 4mo ago

[deleted]

holmesworcester 4mo ago

https://archive.ph/2OWwO https://archive.ph/2OWwO

holmesworcester 4mo ago

If AI is now ascending an economic learning curve from: 1. Extremely useful (Claude Code & Waymo now) 2. Doing ~everything we do (AGI & Optimus in a few years? 10?) 3. RSI (?) 4. Being smarter than any living person at every intellectual task (?) 5. Being smarter than the best-organized aggregate of all humans (10-100 years?) ...And all of the scientific and resource-allocation institutions that brought us the computer and the second half of the 20th century are now fixated on this learning curve, what universe can we possibly imagine where this is not transformative and powerful? Honestly the only one I can think of is one in which we kill almost everyone in some other way first, and contrary to what you read in the news, almost everyone dying is not what the trend line has been from existing problems like war, disease, or even climate change. Also, just to pre-empt a common quibble: when I say "AI" I mean the set of all AI and their combined decision vector, not any one AI, so conflicting interests within the set of AI's will not save anyone any more than the conflicting interests of colonizers saved indigenous Americans.

notarobot123 4mo ago
> Being smarter than the best-organized aggregate of all humans Isn't there pretty much a consensus that committees and institutions are not all that smart? I think you're confusing the categories of "intelligence" and power. Institutions are powerful. The smartest AI is still just a tool without the infrastructure to turn that into real world effects and someone to direct it. It seems you have faith that this is inevitable and unavoidable. I get it, even rationalists succumb to religious thinking eventually. We're only human after all.
- wartywhoa23 4mo ago
  I'm genuinely baffled each time I encounter this notion that the power is dumb. It's as if power was a kind of demigod Heracles who paves its way with sheer brute force alone. But in reality it consists of very mortal and feeble creatures of meat and bone. How in the world can people assume dumb those who possess all the means and rights for violence by controlling and manipulating myriads of other creatures of meat and bone, is totally beyond me.
  - notarobot123 4mo ago
    Power and intelligence are not completely unrelated. Are government officials, politicians, CEOs, judges, etc, usually above average intelligence? Probably. Do the institutions they work within make intelligent decisions? Sometimes. The fallacy is assuming intelligence automatically equates to power and influence. University professors and doctors have very little power in their institutions even though they may often be more intelligent than their bosses. A worse fallacy is assuming intelligence creates better political outcomes (in the broad sense of politics as contested collective decision-making). I don't know many people who would willingly trade their freedom to make a bad decision for the enforcement of the "right" decision by a superior intelligence.
consumer451 4mo ago
Here is an issue I think about often, but I am not quite sure how to put it into words. We have many extremely smart people in various fields. Executives, politicians, and society generally ignore them and do whatever they want. I don't believe that lack of access to intelligence is our problem. How is "free" intelligence going to improve this? I don't just mean climate, but business planning, health, risk assessment, everything.
- zzgo 4mo ago
  There are lots of incentives for politicians and executives (and anyone else holding the levers of power) to ignore information, intelligence, and advice. I think you're right to be skeptical that "free" intelligence is going to improve anything without first addressing the incentives of the people holding power.
- fsflover 4mo ago
  The free intelligence may help to steer the society and politicians to one or the other side. The only question is who decides which.
  - wartywhoa23 4mo ago
    No invention was able to steer the psychopaths away from waging wars. But this time it's absolutely different, right? Riiight?..
    - fsflover 4mo ago
      Don't we have now much less wars than before, in general?
      - wartywhoa23 4mo ago
        No, and we have wars no one in their sane mind would even imagine 20 years ago, like that of Russia vs Ukraine. See, it's easy to speculate how there are less wars when you live in a place which haven't seen war for decades or cenuries, but it's a complete game changer when it's 150 km from where you live, and it's not just some regular war, but a long play intentional meatgrinding AI drone debugging polygon.
protocolture 4mo ago
What makes you think you can assume that it will continue geometrically forever?

petre 4mo ago

> and saying that your technology has the power to cause a massive catastrophe Possibly, but by using up resources, not by its output. Its draw on resources will definitely accelerate climate change to create slop.

Nevermark 4mo ago

> an incentive to make their technology seem transformative and powerful I have no frame of reference to process this. Humans species perhaps 300,000 years, we are essentially the same. Transistor, 79 years, explosive growth in numbers and power. Integrated circuit, 68 years, explosive growth in numbers and power. "Attention is all you need", 9 years. ChatGPT, 4 years, explosive growth in instances and power. Humans species, not getting smarter. [0,1,2,3] There is a syndrome where many people seem unable to perceive or reason about rates of change in technology. We are going to spend the vast majority of our future lives without the intelligence crown. In terms of verbally expressible knowledge, models are passing many people completely, and passing all of our individually respectively weak reasoning areas. Other modalities are progressing very quickly. There will be short periods where progress happens quickly, but the impact feels slow. There will be radical changes that feel slow too, because if something anticipated or important isn't instant, we tend to perceive it as slow. But it won't be slow. And it won't be long. We are smart in a kind of pick the-best-of-us at the-best-of-times way. We are rarely consistently or broadly smart individually. We are not in the same galaxy as "ready". What would that look like? [0] https://en.wikipedia.org/wiki/Human https://en.wikipedia.org/wiki/Human [1] https://en.wikipedia.org/wiki/Transistor https://en.wikipedia.org/wiki/Transistor [2] https://en.wikipedia.org/wiki/Integrated_circuit https://en.wikipedia.org/wiki/Integrated_circuit [3] https://en.wikipedia.org/wiki/Large_language_model https://en.wikipedia.org/wiki/Large_language_model

dbspin 4mo ago
Agree with most of this, it's well articulated and captures how we react to change. However - 'Humans species perhaps 300,000 years, we are essentially the same' is an enormous misconception. That fact that we lack gross anatomical changes during this period, ignores everything we now know about punctuated equilibrium and rapid evolution. It's highly probably we've had an enormous number of evolved psychological changes during the last few hundred, and even tens of thousands of years. Changes that relate to our capacity to live in large groups, adapt to urban environments, resist disease and so on. We know that's the case simply because acute pandemics become epidemics through herd immunity, and through the acquisition of lactose tolerance etc. It seems highly unlikely that adaptations stop there. Altering the environment (in the last 10K years that means the built environment) alters the environment of evolutionary adaptiveness. It seems likely that we've essentially domesticated out much of our propensity for violence and increased our capacity for mood regulation. Obviously it's incredibly tricky to pair these specific behavioural changes to genetic changes -> protein synthesis -> behaviour. Bearing in mind though we're only 20 years out from the first study to link allele variant to behaviour (the COMT Val/Met polymorphism), and the potential controversy around such research, this shouldn't be surprising.