4 ms·

IMO, this is a highly bimodal probability distribution. I model this as two very strong attractor states (radical abundance and totalitarian power concentratio

by theptip 1mo ago

IMO, this is a highly bimodal probability distribution. I model this as two very strong attractor states (radical abundance and totalitarian power concentration). I believe the latter is way more likely without a concerted effort that we currently seem incapable of. The “somewhere in the middle” outcomes in my opinion are not AGI-pilled, and only come about if the exponential dies hard in the next year or maybe two - as in, it’s a very specific timing prediction that needs to come true for this scenario to occur.

fluoridation 1mo ago
What exponential?
- theptip 1mo ago
  LLM capability improvement, eg as measured by METR task time.
  - fluoridation 1mo ago
    So it's an exponential with a negative exponent? The thing about those kinds of curves is that since they have asymptotes to zero, it's hard to tell when you've plateaued.
    - theptip 1mo ago
      No, the metric is “p50 task duration”. 7-month doubling time, recently 4-month: https://metr.org/ https://metr.org/ Plenty of other exponential metrics too, compute built, AI revenue, etc.
- KellyCriterion 1mo ago
  monthly AI bills :-)
newsomix9xl 1mo ago
Why is totalitarianism always the "easy likely" outcome? These are often hard fought bloody and some times brilliantly organized strategic and tactical victories. That took years. Years. Its portrayed like "oh, good guys stayed home, I guess the dictatorship wins". Really?
- adrianN 1mo ago
  Without concerted action against it, people with power have better chances of increasing their power than people without power. Power naturally accumulates into small groups. Empirically it seems less likely that people with power are particularly altruistic, so some form of totalitarian control appears the likely outcome.
  - newsomix9xl 1mo ago
    Some might say there are altruistic totalitarians. Altruism is less likely in all populations. It's just rare.
    - jubilee33 1mo ago
      Perhaps we should design a leadership qualification system that selects for it as a primary trait.
- theptip 1mo ago
  It looks to me like AI will enable extreme power concentration by default. For example, suppose we have a world with centrally planned economy where humanoid robots outnumber humans. And also suppose the government has taken ultimate control of these. Seems like it would be essentially impossible for any civil uprising to overthrow such a system. Or if you prefer less sci-fi, look at the trajectory of power consolidation in corporate America over the last five decades. In the short term, the “good guys” we need are antitrust regulators. Yeah, they got sent home.
  - newsomix9xl 1mo ago
spidersoverjoye 1mo ago
Why only scenarios where humans remain in control? I think the kind of AGI needed for radical abundance would be difficult to achieve in a way that also does not lead to ASI
- theptip 1mo ago
  Agreed, I think AGI->ASI. My happy radical abundance stories are happy ASI stories mostly. And therefore much less likely. BDFL / Benevolent Machine God is definitely one of the ways you can get a happy attractor. Seems scary to roll the dice on. And yep, I include non-human totalitarian outcomes in the unhappy bucket.
  - ChrisMarshallNY 1mo ago
    [dead]
a2ff6eeb0 1mo ago
If we get effective AI, humans aren't going to remain in charge -- that much is inevitable. If you had a governance machine that made your country perform 10% better than other countries, so long as you deferred all decisions to it, you'd defer those decisions or get left behind. And AI probably has the potential to do far better than 10%. Either everyone, globally, gives up on AI development, or AI is going to be in charge of all human development, and we'll just be along for the ride. Right now, the work ahead is to shape AI so that the decision making it does as it carries us along is something we like.
- radicaldreamer 1mo ago
  No, because if this were the case, the UK wouldn’t have royals, North Korean wouldn’t be a dictatorship, China wouldn’t have the CCP, and America wouldn’t be a two party state. Hardly any existing polity would willingly hand over control to an AI. More likely an AI would overwhelm human polities through endless hacking and warfare.
  - jbstack 1mo ago
    You're assuming a handover happens in a single definable moment. More realistically it will be one small decision making power at a time, so that at any given moment it never really feels like control is being given up. Boiled frog.
  - a2ff6eeb0 1mo ago
    The UK royals are in charge of the country, then?
- theptip 1mo ago
  I mostly agree, though there are versions of ASI that I believe leave humans “in charge”. Most of my probability mass for good outcomes is around some variant of “guardian angels” or “benign machine god”. Banks’s Culture novels being on the realistic end of that range. “Keep humans around because the world is more interesting with them in it”. But it’s possible that we might manage to construct some entity that really loves humans and wants to shepherd them while leaving them in control. (Perhaps declining to indulge the most self-destructive tendencies.) This feels like the very hardest alignment path to engineer. The other branches of course are post-humanism; human uploads and “merging” with AI, whatever that means. Or post-humans augmented to have intelligence sufficient to eschew AI and take back their destiny.
Loic 1mo ago
I agree with you from a mathematical point of view. You have this "dead economy theory"[0]. I spent a lot of time trying to model this theory, enhance the theory (lot of good HN comments) and find a "middle ground" path. My saying is that you end-up on an optimization saddle point where it is really hard to not fall into concentration of power. I was able to get radical abundance only at the cost of pretty hard regulations. [0]: https://news.ycombinator.com/item?id=48324712 https://news.ycombinator.com/item?id=48324712
- pbronez 1mo ago
  Did you publish your results anywhere? I’d be interested to read about them in detail.
- agileAlligator 1mo ago
  https://daviddfriedman.substack.com/p/shrinking-hierarchy https://daviddfriedman.substack.com/p/shrinking-hierarchy This is almost a direct counter to your essay, albeit written before you
- theptip 1mo ago
  I agree there are some extremely bad outcomes from fast labor replacement. I don’t think it’s inevitable that the economy collapses though. The problem is that this model assumes that capital needs labor, but soon it may not. The easiest model is - assume we get robot laborers in the next few years. The AI firms that are displacing humans still need services, most of these will be purchased from other AI/robot firms, and so while prices will swing around dramatically, there is no reason the engine has to stop. (Depending on what the AIs want; you can assume that more compute and robots will be part of it.) I do agree that unbridled capitalism is mostly incompatible with human flourishing in this world and in any event you’ll see a lot more regulation; made feasible by how good AIs are at navigating huge Byzantine rulebooks. I view the techno-feudalist capitalist endgame as fairly unstable due to consolidation pressures, though I guess once we have nuclear-armed megacorps they might use MAD to prevent further mergers.
k__ 1mo ago
And with exponential, you mean sigmoid.

by theptip 1mo ago

fluoridation 1mo ago

What exponential?

theptip 1mo ago
LLM capability improvement, eg as measured by METR task time.
- fluoridation 1mo ago
  So it's an exponential with a negative exponent? The thing about those kinds of curves is that since they have asymptotes to zero, it's hard to tell when you've plateaued.
  - theptip 1mo ago
    No, the metric is “p50 task duration”. 7-month doubling time, recently 4-month: https://metr.org/ https://metr.org/ Plenty of other exponential metrics too, compute built, AI revenue, etc.
KellyCriterion 1mo ago
monthly AI bills :-)

newsomix9xl 1mo ago

Why is totalitarianism always the "easy likely" outcome? These are often hard fought bloody and some times brilliantly organized strategic and tactical victories. That took years. Years. Its portrayed like "oh, good guys stayed home, I guess the dictatorship wins". Really?

adrianN 1mo ago
Without concerted action against it, people with power have better chances of increasing their power than people without power. Power naturally accumulates into small groups. Empirically it seems less likely that people with power are particularly altruistic, so some form of totalitarian control appears the likely outcome.
- newsomix9xl 1mo ago
  Some might say there are altruistic totalitarians. Altruism is less likely in all populations. It's just rare.
  - jubilee33 1mo ago
    Perhaps we should design a leadership qualification system that selects for it as a primary trait.
theptip 1mo ago
It looks to me like AI will enable extreme power concentration by default. For example, suppose we have a world with centrally planned economy where humanoid robots outnumber humans. And also suppose the government has taken ultimate control of these. Seems like it would be essentially impossible for any civil uprising to overthrow such a system. Or if you prefer less sci-fi, look at the trajectory of power consolidation in corporate America over the last five decades. In the short term, the “good guys” we need are antitrust regulators. Yeah, they got sent home.
- newsomix9xl 1mo ago

spidersoverjoye 1mo ago

Why only scenarios where humans remain in control? I think the kind of AGI needed for radical abundance would be difficult to achieve in a way that also does not lead to ASI

theptip 1mo ago
Agreed, I think AGI->ASI. My happy radical abundance stories are happy ASI stories mostly. And therefore much less likely. BDFL / Benevolent Machine God is definitely one of the ways you can get a happy attractor. Seems scary to roll the dice on. And yep, I include non-human totalitarian outcomes in the unhappy bucket.
- ChrisMarshallNY 1mo ago
  [dead]

a2ff6eeb0 1mo ago

If we get effective AI, humans aren't going to remain in charge -- that much is inevitable. If you had a governance machine that made your country perform 10% better than other countries, so long as you deferred all decisions to it, you'd defer those decisions or get left behind. And AI probably has the potential to do far better than 10%. Either everyone, globally, gives up on AI development, or AI is going to be in charge of all human development, and we'll just be along for the ride. Right now, the work ahead is to shape AI so that the decision making it does as it carries us along is something we like.

radicaldreamer 1mo ago
No, because if this were the case, the UK wouldn’t have royals, North Korean wouldn’t be a dictatorship, China wouldn’t have the CCP, and America wouldn’t be a two party state. Hardly any existing polity would willingly hand over control to an AI. More likely an AI would overwhelm human polities through endless hacking and warfare.
- jbstack 1mo ago
  You're assuming a handover happens in a single definable moment. More realistically it will be one small decision making power at a time, so that at any given moment it never really feels like control is being given up. Boiled frog.
- a2ff6eeb0 1mo ago
  The UK royals are in charge of the country, then?
theptip 1mo ago
I mostly agree, though there are versions of ASI that I believe leave humans “in charge”. Most of my probability mass for good outcomes is around some variant of “guardian angels” or “benign machine god”. Banks’s Culture novels being on the realistic end of that range. “Keep humans around because the world is more interesting with them in it”. But it’s possible that we might manage to construct some entity that really loves humans and wants to shepherd them while leaving them in control. (Perhaps declining to indulge the most self-destructive tendencies.) This feels like the very hardest alignment path to engineer. The other branches of course are post-humanism; human uploads and “merging” with AI, whatever that means. Or post-humans augmented to have intelligence sufficient to eschew AI and take back their destiny.

Loic 1mo ago

I agree with you from a mathematical point of view. You have this "dead economy theory"[0]. I spent a lot of time trying to model this theory, enhance the theory (lot of good HN comments) and find a "middle ground" path. My saying is that you end-up on an optimization saddle point where it is really hard to not fall into concentration of power. I was able to get radical abundance only at the cost of pretty hard regulations. [0]: https://news.ycombinator.com/item?id=48324712 https://news.ycombinator.com/item?id=48324712

pbronez 1mo ago
Did you publish your results anywhere? I’d be interested to read about them in detail.
agileAlligator 1mo ago
https://daviddfriedman.substack.com/p/shrinking-hierarchy https://daviddfriedman.substack.com/p/shrinking-hierarchy This is almost a direct counter to your essay, albeit written before you
theptip 1mo ago
I agree there are some extremely bad outcomes from fast labor replacement. I don’t think it’s inevitable that the economy collapses though. The problem is that this model assumes that capital needs labor, but soon it may not. The easiest model is - assume we get robot laborers in the next few years. The AI firms that are displacing humans still need services, most of these will be purchased from other AI/robot firms, and so while prices will swing around dramatically, there is no reason the engine has to stop. (Depending on what the AIs want; you can assume that more compute and robots will be part of it.) I do agree that unbridled capitalism is mostly incompatible with human flourishing in this world and in any event you’ll see a lot more regulation; made feasible by how good AIs are at navigating huge Byzantine rulebooks. I view the techno-feudalist capitalist endgame as fairly unstable due to consolidation pressures, though I guess once we have nuclear-armed megacorps they might use MAD to prevent further mergers.

k__ 1mo ago

And with exponential, you mean sigmoid.