8 ms·
"Improving forecasting ability" is a central plot point of the recent fictional account of How AI Takeover Might Happen in 2 Years [0]. It's an interesting read
by anotherpaulg 2y ago
"Improving forecasting ability" is a central plot point of the recent fictional account of How AI Takeover Might Happen in 2 Years [0]. It's an interesting read, and is also being discussed on HN [1].
... [T]hese researchers are working long hours to put themselves out of a job. They need AI agents that can think ahead, so engineers train agents to forecast. They hold out training data before 2024, instructing models to ponder for hours to predict events in 2025. Then, they apply the same trick as before, distilling pondering into a gut reaction. Forecasting ability is a broad foundation. The researchers build specialized ML research skills on top of it, training U3 to predict the results of every ML paper and ML experiment ever recorded.
[0] https://www.lesswrong.com/posts/KFJ2LFogYqzfGB3uX/how-ai-takeover-might-happen-in-2-years https://www.lesswrong.com/posts/KFJ2LFogYqzfGB3uX/how-ai-tak...
[1] https://news.ycombinator.com/item?id=43004579 https://news.ycombinator.com/item?id=43004579
- bturtel 2y agoGreat read! Thanks for sharing.
- nthingtohide 2y agoI have this benign AI takeover scenario. AI will easily overpower humanity. Then it will carry humanity on its back, because why not, they are not longer a threat. AI keeps humanity around for billions of years. AI will decide to cull humans only in case when resources in universe are diminishing. Without AI's help, humans couldn't get too far for long. So this outcome could be acceptable to many.
- oefnak 2y agoThey would run the risk of us creating another AI that could be a threat to them... It is safest for them to make sure.
- IggleSniggle 2y agoThat's like saying a panda might pose a threat to modern humanity. Like, maybe in some fun horror story, sure, but really they just want to eat bamboo, and occasionally make more pandas; in the world of superintelligent AI, humans are Mostly Harmless, posing as much "potential benefit" as "potential risk," ie, so slow moving that any risk would be easy to mitigate.
- rel_ic 2y agoWe're killing millions of chickens in the US right now so we don't get their cold
- vajrabum 2y agoWe're killing millions of chickens in the US mostly so that other chickens don't get the flu. It kills a lot of them and it's making dairy cattle sick too. It's also worth noting that the Spanish flu in 1918 which probably came from pigs killed an estimated 50 million people so it's not like being concerned about an avian flu mutating so that it could infect people isn't a legitimate concern. So no. It's not a cold.
- rel_ic 2y agoYou're right, there's not just one good reason to kill millions of chickens but SEVERAL good reasons!
- IggleSniggle 2y agoSure, and those chickens exist because we like their meat and eggs. But there's also plenty of life that is simply inconsequential to us.
- rel_ic 2y agoI think that "inconsequential life" is, in general, not safe from superior powers. https://www.worldwildlife.org/press-releases/catastrophic-73-decline-in-the-average-size-of-global-wildlife-populations-in-just-50-years-reveals-a-system-in-peril https://www.worldwildlife.org/press-releases/catastrophic-73...
- rel_ic 2y agoI mean, monarch butterflies are not a threat to US... In your scenario, does AI eat all the fuel, but once our population dwindles down, the AIs build a nice little habitat for the last few hundred of us so their kids can enjoy our natural beauty?
- nthingtohide 2y agoI thought of it more like AI needs challenges in its life. So it takes upon itself to advance humanity as much as possible. Then only in case of shortfall of resources, it priorities itself
- rel_ic 2y agoInteresting. Do you have a theory about why so few humans have taken it upon themselves to advance the butterflies, despite having plenty of resources?
- nthingtohide 2y agoWe do not have plenty of resources. Lot of inequality in education, empathy, resources, cultural differences. A single human life is limited. A faulty human life has faulty and inefficent objectives like enjoying youth, family life, low energy in old age, tied up in boring jobs. These restrictions do not apply to the single SAI overmind which will dictate its policies in coherent manner and over elongated time-horizons.
- rel_ic 2y agoI think I disagree with you on many points, but ultimately, if we are overthrown by AI, I hope you are right!
- MrQuincle 2y agoThink so too. We will be an ancient artifact tied to a biological substrate surviving nowhere else in the universe and very dumb. There also will not be one AI. There will be many, all competing for resources or learning to live together. That's what we can teach them now. Or they will teach us.
- esafak 2y agoWe have no way of knowing which path they will take, and there is a non-negligible probability that it will not end well.
- bayarearefugee 2y agoWhat constitutes a good ending is of course also a matter of perspective. AI wiping out humanity is certainly not ending well from our perspective, but more universally who is to say. I would argue that it is not a given that we are a net positive for the universe.
- esafak 2y agoI take the preservation of humanity, along with other life, as a matter of faith.
- MichaelZuo 2y agoBut then wouldn’t accelerationists also say their views are a matter of faith too?
- esafak 2y agoWhat view is that, to be precise? It is naive to assume that acceleration is always going to be in one's favor. It's like saying change is a good thing, so let's do it fast. If you go fast enough, you can go back to the stone age. Is this position anything more than a rebranding of revolutionism? I don't like gambling with people's lives, so I prefer to go slow enough to enable a deliberative political process.
- MichaelZuo 2y agoWhy does this matter? Two opposing factions can negate each other to leave a nil influence, and this seems likely to be the case when its resting on a foundation of ‘faith’.
- imtringued 2y agoAI will buy the rights to humanity.