31 ms·

AI 2027

• deleted 2y ago
  [deleted]
• deleted 2y ago
  [deleted]
• ikerino 2y ago
  Feels reasonable in the first few paragraphs, then quickly starts reading like science fiction. Would love to read a perspective examining "what is the slowest reasonable pace of development we could expect." This feels to me like the fastest (unreasonable) trajectory we could expect.
  • admiralrohan 2y ago
    No one knows what will happen. But these thought experiments can be useful as a critical thinking practice.
  • layer8 2y ago
    The slowest is a sudden and permanent plateau, where all attempts at progress turn out to result in serious downsides that make them unworkable.
    • 9dev 2y ago
      Like an exponentially growing compute requirement for negligible performance gains, on the scale of the energy consumption of small countries? Because that is where we are, right now.
    • photonthug 2y ago
      Even if this were true, it's not quite the end of the story is it? The hype itself creates lots of compute and to some extent the power needed to feed that compute, even if approximately zero of the hype pans out. So an interesting question becomes.. what happens with all the excess? Sure it probably gets gobbled up in crypto ponzi schemes, but I guess we can try to be optimistic. IDK, maybe we get to solve cancer and climate change anyway, not with fancy new AGI, but merely with some new ability to cheaply crunch numbers for boring old school ODEs.
  • ddp26 2y ago
    The forecasts under "Research" are distributions, so you can compare the 10th percentile vs 90th percentile. Their research is consistent with a similar story unfolding over 8-10 years instead of 2.
  • zmj 2y ago
    If you described today's AI capabilities to someone from 3 years ago, that would also sound like science fiction. Extrapolate.
  • FeepingCreature 2y ago
    > Feels reasonable in the first few paragraphs, then quickly starts reading like science fiction. That's kind of unavoidably what accelerating progress feels like.
• ahofmann 2y ago
  Ok, I'll bite. I predict that everything in this article is horse manure. AGI will not happen. LLMs will be tools, that can automate away stuff, like today and they will get slightly, or quite a bit better at it. That will be all. See you in two years, I'm excited what will be the truth.
  • Tenoke 2y ago
    That seems naive in a status quo bias way to me. Why and where do you expect AI progress to stop? It sounds like somewhere very close to where we are at in your eyes. Why do you think there won't be many further improvements?
    • ahofmann 2y ago
      I write bog-standard PHP software. When GPT-4 came out, I was very frightened that my job could be automated away soon, because for PHP/Laravel/MySQL there must exist a lot of training data. The reality now is, that the current LLMs still often create stuff, that costs me more time to fix, than to do it myself. So I still write a lot of code myself. It is very impressive, that I can think about stopping writing code myself. But my job as a software developer is, very, very secure. LLMs are very unable to build maintainable software. They are unable to understand what humans want and what the codebase need. The stuff they build is good-looking garbage. One example I've seen yesterday: one dev committed code, where the LLM created 50 lines of React code, complete with all those useless comments and for good measure a setTimeout() for something that should be one HTML DIV with two tailwind classes. They can't write idiomatic code, because they write code, that they were prompted for. Almost daily I get code, commit messages, and even issue discussions that are clearly AI-generated. And it costs me time to deal with good-looking but useless content. To be honest, I hope that LLMs get better soon. Because right now, we are in an annoying phase, where software developers bog me down with AI-generated stuff. It just looks good but doesn't help writing usable software, that can be deployed in production. To get to this point, LLMs need to get maybe a hundred times faster, maybe a thousand or ten thousand times. They need a much bigger context window. Then they can have an inner dialogue, where they really "understand" how some feature should be built in a given codebase. That would be very useful. But it will also use so much energy that I doubt that it will be cheaper to let a LLM do those "thinking" parts over, and over again instead of paying a human to build the software. Perhaps this will be feasible in five or eight years. But not two. And this won't be AGI. This will still be a very, very fast stochastic parrot.
• WhatsName 2y ago
  This is absurd, like taking any trend and drawing a straight line to interpolate the future. If I would do this with my tech stock portfolio, we would probably cross the zero line somewhere late 2025... If this article were a AI model, it would be catastrophically overfit.
  • AnimalMuppet 2y ago
    It's worse. It's not drawing a straight line, it's drawing one that curves up, on a log graph.
• Lionga 2y ago
  AI now even got it's own fan fiction porn. It is so stupid not sure whether it is worse if it is written by AI or by a human.
  • deleted 2y ago
    [deleted]
• the_cat_kittles 2y ago
  "we demand to be taken seriously!"
• beklein 2y ago
  Older and related article from one of the authors titled "What 2026 looks like", that is holding up very well against time. Written in mid 2021 (pre ChatGPT) https://www.alignmentforum.org/posts/6Xgy6CAf2jqHhynHL/what-2026-looks-like https://www.alignmentforum.org/posts/6Xgy6CAf2jqHhynHL/what-... //edit: remove the referral tags from URL
  • dkdcwashere 2y ago
    > The alignment community now starts another research agenda, to interrogate AIs about AI-safety-related topics. For example, they literally ask the models “so, are you aligned? If we made bigger versions of you, would they kill us? Why or why not?” (In Diplomacy, you can actually collect data on the analogue of this question, i.e. “will you betray me?” Alas, the models often lie about that. But it’s Diplomacy, they are literally trained to lie, so no one cares.) …yeah?
  • motoxpro 2y ago
    That's incredible how much it broadly aligns with what has happened. Especially because it was before ChatGPT.
    • reducesuffering 2y ago
      Will people finally wake up that the AGI X-Risk people have been right and we’re rapidly approaching a really fucking big deal? This forum has been so behind for too long. Sama has been saying this a decade now: “Development of Superhuman machine intelligence is probably the greatest threat to the continued existence of humanity” 2015 https://blog.samaltman.com/machine-intelligence-part-1 https://blog.samaltman.com/machine-intelligence-part-1 Hinton, Ilya, Dario Amodei, RLHF inventor, Deepmind founders. They all get it, which is why they’re the smart cookies in those positions. First stage is denial, I get it, not easy to swallow the gravity of what’s coming.
      • ffsm8 2y ago
        People have been predicting the singularity to occur sometimes around 2030 and 2045 waaaay further back then 2015. And not just by enthusiasts, I dimly remember an interview with Richard Darkins from back in the day... Though that doesn't mean that the current version of language models will ever achieve AGI, and I sincerely doubt they will. They'll likely be a component in the AI, but likely not the thing that "drives"
• amarcheschi 2y ago
  I just spent some time trying to make claude and gemini make a violin plot of some polar dataframe. I've never used it and it's just for prototyping so i just went "apply a log to the values and make a violin plot of this polars dataframe". ANd had to iterate with them for 4/5 times each. Gemini got it right but then used deprecated methods I might be doing llm wrong, but i just can't get how people might actually do something not trivial just by vibe coding. And it's not like i'm an old fart either, i'm a university student
  • VOIPThrowaway 2y ago
    You're asking it to think and it can't. It's spicy auto complete. Ask it to create a program that can create a violin plot from a CVS file. Because this has been "done before", it will do a decent job.
    • suddenlybananas 2y ago
      But this blog post said that it's going to be God in like 5 years?!
  • pydry 2y ago
    all tech hype cycles are a bit like this. when you were born people were predicting the end of offline shops. The trough of disillusionment will set in for everybody else in due time.
  • dinfinity 2y ago
    Yes, you're most likely doing it wrong. I would like to add that "vibe coding" is a dreadful term thought up by someone who is arguably not very good at software engineering, as talented as he may be in other respects. The term has become a misleading and frankly pejorative term. A better, more neutral one is AI assisted software engineering. This is an article that describes a pretty good approach for that: https://getstream.io/blog/cursor-ai-large-projects/ https://getstream.io/blog/cursor-ai-large-projects/ But do skip (or at least significantly postpone) enabling the 'yolo mode' (sigh).
• moab 2y ago
  > "OpenBrain (the leading US AI project) builds AI agents that are good enough to dramatically accelerate their research. The humans, who up until very recently had been the best AI researchers on the planet, sit back and watch the AIs do their jobs, making better and better AI systems." I'm not sure what gives the authors the confidence to predict such statements. Wishful thinking? Worst-case paranoia? I agree that such an outcome is possible, but on 2--3 year timelines? This would imply that the approach everyone is taking right now is the right approach and that there are no hidden conceptual roadblocks to achieving AGI/superintelligence from DFS-ing down this path. All of the predictions seem to ignore the possibility of such barriers, or at most acknowledge the possibility but wave it away by appealing to the army of AI researchers and industry funding being allocated to this problem. IMO it is the onus of the proposers of such timelines to argue why there are no such barriers and that we will see predictable scaling in the 2--3 year horizon.
  • throwawaylolllm 2y ago
    It's my belief (and I'm far from the only person who thinks this) that many AI optimists are motivated by an essentially religious belief that you could call Singularitarianism. So "wishful thinking" would be one answer. This document would then be the rough equivalent of a Christian fundamentalist outlining, on the basis of tangentially related news stories, how the Second Coming will come to pass in the next few years.
    • pixl97 2y ago
      Eh, not sure if the second coming is a great analogy. That wholly depends on the whims of a fictional entity performing some unlikely actions. Instead think of them saying a crusade occurring in the next few years. When the group saying the crusade is coming is spending billions of dollars to trying to make just that occur you no longer have the ability to say it's not going to happen. You are now forced to examine the risks of their actions.
    • viccis 2y ago
      Crackpot millenarians have always been a thing. This crop of them is just particularly lame and hellbent on boiling the oceans to get their eschatological outcome.
• zvitiate 2y ago
  There's a lot to potentially unpack here, but idk, the idea that humanity entering hell (extermination) or heaven (brain uploading; aging cure) is whether or not we listen to AI safety researchers for a few months makes me question whether it's really worth unpacking.
  • amelius 2y ago
    If we don't do it, someone else will.
    • throwawaylolllm 2y ago
      [flagged]
    • itishappy 2y ago
      Which? Exterminate humanity or cure aging?
      • ethersteeds 2y ago
        Yes
      • amelius 2y ago
        The thing whose outcome can go either way.
        • itishappy 2y ago
          I honestly can't tell what you're trying to say here. I'd argue there's some pretty significant barriers to each.
    • layer8 2y ago
      I’m okay if someone else unpacks it.
    • achierius 2y ago
      That's obviously not true. Before OpenAI blew the field open, multiple labs -- e.g. Google -- were intentionally holding back their research from the public eye because they thought the world was not ready. Investors were not pouring billions into capabilities. China did not particularly care to focus on this one research area, among many, that the US is still solidly ahead in. The only reason timelines are as short as they are is because of people at OpenAI and thereafter Anthropic deciding that "they had no choice". They had a choice, and they took the one which has chopped at the very least years off of the time we would otherwise have had to handle all of this. I can barely begin to describe the magnitude of the crime that they have committed -- and so I suggest that you consider that before propagating the same destructive lies that led us here in the first place.
• Q6T46nT668w6i3m 2y ago
  This is worse than the mansplaining scene from Annie Hall.
  • arduanika 2y ago
    You mean the part where he pulls out Marshal McLuhan to back him up in an argument? "You know nothing of my work..."
• IshKebab 2y ago
  This is hilariously over-optimistic on the timescales. Like on this timeline we'll have a Mars colony in 10 years, immortality drugs in 15 and Half Life 3 in 20.
  • sva_ 2y ago
    You forgot fusion energy
    • klabb3 2y ago
      Quantum AI powered by cold fusion and blockchain when?
  • zvitiate 2y ago
    No, sooner lol. We'll have aging cures and brain uploading by late 2028. Dyson Swarms will be "emerging tech".
  • mchusma 2y ago
    I like that the "slowdown" scenario has by 2030 we have a robot economy, cure for aging, brain uploading, and are working on a Dyson Sphere.
    • Aurornis 2y ago
      The story is very clearly modeled to follow the exponential curve they show. Like the drew the curve out into the shape they wanted, put some milestones on it, and then went to work imagining what would happen if it continued with a heavy dose of X-risk doomerism to keep it spicy. It conveniently ignores all of the physical constraints around things like manufacturing GPUs and scaling training networks.
      • joshjob42 2y ago
        https://ai-2027.com/research/compute-forecast https://ai-2027.com/research/compute-forecast In section 4 they discuss their projections specifically for model size, the state of inference chips in 2027, etc. It's largely pretty in line with expectations in terms of the capacity, and they only project them using 10k of their latest gen wafer scale inference chips by late 2027, roughly like 1M H100 equivalents. That doesn't seem at all impossible. They also earlier on discuss expectations for growth in efficiency of chips, and for growth in spending, which is only ~10x over the next 2.5 years, not unreasonable in absolute terms at all given the many tens of billions of dollars flooding in. So on the "can we train the AI" front, they mostly are just projecting 2.5 years of the growth in scale we've been seeing. The reason they predict a fairly hard takeoff is they expect that distillation, some algorithmic improvements, and iterated creation of synthetic data, training, and then making more synthetic data will enable significant improvements in efficiency of the underlying models (something still largely in line with developments over the last 2 years). In particular they expect a 10T parameter model in early 2027 to be basically human equivalent, and they expect it to "think" at about the rate humans do, 10 words/second. That would require ~300 teraflops of compute per second to think at that rate, or ~0.1H100e. That means one of their inference chips could potentially run ~1000 copies (or fewer copies faster etc. etc.) and thus they have the capacity for millions of human equivalent researchers (or 100k 40x speed researchers) in early 2027. They further expect distillation of such models etc. to squeeze the necessary size down / more expensive models overseeing much smaller but still good models squeezing the effective amount of compute necessary, down to just 2T parameters and ~60 teraflops each, or 5000 human-equivalents per inference chip, making for up to 50M human-equivalents by late 2027. This is probably the biggest open question and the place where the most criticism seems to me to be warranted. Their hardware timelines are pretty reasonable, but one could easily expect needing 10-100x more compute or even perhaps 1000x than they describe to achieve Nobel-winner AGI or superintelligence.
• qwertox 2y ago
  That is some awesome webdesign.
• noncoml 2y ago
  2015: We will have FSD(full autonomy) by 2017
  • wkat4242 2y ago
    Well, Teslas do have "Full Self Driving". It's not actually fully self driving and that doesn't even seem to be on the horizon but it doesn't appear to be stopping Tesla supporters.
• porphyra 2y ago
  Seems very sinophobic. Deepseek and Manus have shown that China is legitimately an innovation powerhouse in AI but this article makes it sound like they will just keep falling behind without stealing.
  • ugh123 2y ago
    Don't confuse innovation with optimisation.
    • pixl97 2y ago
      Don't confuse designing the product with winning the market.
      • deleted 2y ago
        [deleted]
  • princealiiiii 2y ago
    Stealing model weights isn't even particularly useful long-term, it's the training + data generation recipes that have value.
  • MugaSofer 2y ago
    That whole section seems to be pretty directly based on DeepSeek's "very impressive work" with R1 being simultaneously very impressive, and several months behind OpenAI. (They more or less say as much in footnote 36.) They blame this on US chip controls just barely holding China back from the cutting edge by a few months. I wouldn't call that a knock on Chinese innovation.
    • clayhacks 2y ago
      But it also assumes China would never really catch up to American chip companies. China is already investing heavily in chip R&D and things like RISC-V, I think it’s very plausible that lag window shrinks over this horizon. Perhaps even flipping given their much larger willingness to use industrial policy for goals they want achieved.
• disambiguation 2y ago
  Amusing sci-fi, i give it a B- for bland prose, weak story structure, and lack of originality - assuming this isn't all AI gen slop which is awarded an automatic F. >All three sets of worries—misalignment, concentration of power in a private company, and normal concerns like job loss—motivate the government to tighten its control. A private company becoming "too powerful" is a non issue for governments, unless a drone army is somewhere in that timeline. Fun fact the former head of the NSA sits on the board of Open AI. Job loss is a non issue, if there are corresponding economic gains they can be redistributed. "Alignment" is too far into the fiction side of sci-fi. Anthropomorphizing today's AI is tantamount to mental illness. "But really, what if AGI?" We either get the final say or we don't. If we're dumb enough to hand over all responsibility to an unproven agent and we get burned, then serves us right for being lazy. But if we forge ahead anyway and AGI becomes something beyond review, we still have the final say on the power switch.
• atemerev 2y ago
  What is this, some OpenAI employee fan fiction? Did Sam himself write this? OpenAI models are not even SOTA, except that new-ish style transfer / illustration thing that made all us living in Ghibli world for a few days. R1 is _better_ than o1, and open-weights. GPT-4.5 is disappointing, except for a few narrow areas where it excels. DeepResearch is impressive though, but the moat is in tight web search / Google Scholar search integration, not weights. So far, I'd bet on open models or maybe Anthropic, as Claude 3.7 is the current SOTA for most tasks. As of the timeline, this is _pessimistic_. I already write 90% code with Claude, so are most of my colleagues. Yes, it does errors, and overdoes things. Just like a regular human middle-stage software engineer. Also fun that this assumes relatively stable politics in the US and relatively functioning world economy, which I think is crazy optimistic to rely on these days. Also, superpersuasion _already works_, this is what I am researching and testing. It is not autonomous, it is human-assisted by now, but it is a superpower for those who have it, and it explains some of the things happening with the world right now.
  • achierius 2y ago
    > superpersuasion _already works_ Is this demonstrated in any public research? Unless you just mean something like "good at persuading" -- which is different from my understanding of the term -- I find this hard to believe.
    • atemerev 2y ago
      No, I meant "good at persuading", it is not 100% efficiency of course.
      • pixodaros 2y ago
        That singularity happened in the fifth century BCE when people figured out that they could charge silver to teach the art of rhetoric and not just teach their sons and nephews
  • ddp26 2y ago
    The story isn't about OpenAI, they say the company could be Xai, Anthropic, Google, or another.
• infecto 2y ago
  Could not get through the entire thing. It’s mostly a bunch of fantasy intermingled with bits of possible interesting discussion points. The whole right side metrics are purely a distraction because entirely fiction.
  • archagon 2y ago
    Website design is nice, though.
• Joshuatanderson 2y ago
  This is extremely important. Scott Alexander's earlier predictions are holding up extremely well, at least on image progress.
• Willingham 2y ago
  - October 2027 - 'The ability to automate most white-collar jobs' I wonder which jobs would not be automated? Therapy? HR?
  • hsuduebc2 2y ago
    Board of directors
• deleted 2y ago
  [deleted]
• dingnuts 2y ago
  how am I supposed to take articles like this seriously when they say absolutely false bullshit like this > the AIs can do everything taught by a CS degree no, they fucking can't. not at all. not even close. I feel like I'm taking crazy pills. Does anyone really think this? Why have I not seen -any- complete software created via vibe coding yet?
  • ladberg 2y ago
    It doesn't claim it's possible now, it's a fictional short story claiming "AIs can do everything taught by a CS degree" by the end of 2026.
    • senordevnyc 2y ago
      Ironically, the models of today can read an article better than some of us.
  • casey2 2y ago
    Lesswrong brigade. They are all dropout philosophers just ignore them.
• vagab0nd 2y ago
  Bad future predictions: short-sighted guesses based on current trends and vibe. Often depend on individuals or companies. Made by free-riders. Example: Twitter. Good future predictions: insights into the fundamental principles that shape society, more law than speculation. Made by visionaries. Example: Vernor Vinge.
• dalmo3 2y ago
  "1984 was set in 1984." https://youtu.be/BLYwQb2T_i8?si=JpIXIFd9u-vUJCS4 https://youtu.be/BLYwQb2T_i8?si=JpIXIFd9u-vUJCS4
• pera 2y ago
  From the same dilettantes who brought you the Zizians and other bizarre cults... thanks but I rather read Nostradamus
  • arduanika 2y ago
    What a bad faith argument. No true AI safety scaremonger brat stabs their landlord with a katana. The rationality of these rationalists is 100% uncorrolated with the rationality of *those* rationalists.
  • selfhoster11 2y ago
    I logged in specifically to say the following: I do not think it is possible that Scott Alexander would sign his name to something that would in any way promote Zizian views. I don't know him personally, but I've read enough to know where he stands.
• soupfordummies 2y ago
  The "race" ending reads like Universal Paperclips fan fiction :)
• 827a 2y ago
  Readers should, charitably, interpret this as "the sequence of events which need to happen in order for OpenAI to justify the inflow of capital necessary to survive". Your daily vibe coding challenge: Get GPT-4o to output functional code which uses Google Vertex AI to generate a text embedding. If they can solve that one by July, then maybe we're on track for "curing all disease and aging, brain uploading, and colonizing the solar system" by 2030.
  • slaterbug 2y ago
    You’ve intentionally hamstrung your test by choosing an inferior model though.
    • 827a 2y ago
      o1 fails at this, likely because it does not seem to have access to search, so it is operating on outdated information. It recommends the usage of methods that have been removed by Google in later versions of the library. This is also, to be fair, a mistake gpt-4o can make if you don't explicitly tell it to search. o3-mini-high's output might work, but it isn't ideal: It immediately jumps to recommending avoiding all google cloud libraries and directly issuing a request to their API with fetch.
  • Philpax 2y ago
    Haven't tested this (cbf setting up Google Cloud), but the output looks consistent with the docs it cites: https://chatgpt.com/share/67efd449-ce34-8003-bd37-9ec688a11be2 https://chatgpt.com/share/67efd449-ce34-8003-bd37-9ec688a11b... You may consider using search to be cheating, but we do it, so why shouldn't LLMs?
    • 827a 2y ago
      I should have specified "nodejs", as that has been my most recent difficulty. The challenge, specifically, with that prompt is that Google has at least four nodejs libraries that are all seem at least reasonably capable of accessing text embedding models on vertex ai (@google-ai/generativelanguage, @google-cloud/vertexai, @google-cloud/aiplatform, and @google/genai), and they've also published breaking changes multiple times to all of them. So, in my experience, GPT not only will confuse methods from one of their libraries with the other, but will also sometimes hallucinate answers only applicable to older versions of the library, without understanding which version its giving code for. Once it has struggled enough, it'll sometimes just give up and tell you to use axios, but the APIs it recommends axios calls for are all their protobuf APIs; so I'm not even sure if that would work. Search is totally reasonable, but in this case: Even Google's own documentation on these libraries is exceedingly bad. Nearly all the examples they give for them are for accessing the language models, not text embedding models; so GPT will also sometimes generate code that is perfectly correct for accessing one of the generative language models, but will swap e.g the "model: gemini-2.0" parameter for "model: text-embedding-005"; which also does not work.
• MaxfordAndSons 2y ago
  As someone who's fairly ignorant of how AI actually works at a low level, I feel incapable of assessing how realistic any of these projections are. But the "bad ending" was certainly chilling. That said, this snippet from the bad ending nearly made me spit my coffee out laughing: > There are even bioengineered human-like creatures (to humans what corgis are to wolves) sitting in office-like environments all day viewing readouts of what’s going on and excitedly approving of everything, since that satisfies some of Agent-4’s drives.
  • arduanika 2y ago
    Sigh. When you talk to these people their eugenics obsession always comes out eventually. Set a timer and wait for it.
    • Philpax 2y ago
      While I don't disagree that I've seen a lot of eugenics talk from rationalist(-adjacent)s, I don't think this is an example of it: this is describing how misaligned AI could technically keep humans alive while still killing "humanity."
      • arduanika 2y ago
        Fair enough. Sometimes it comes out as a dark fantasy projected onto their AI gods, rather than a thing that they themselves want to do to us.
• Jun8 2y ago
  ACT post where Scott Alexander provides some additional info: https://www.astralcodexten.com/p/introducing-ai-2027 https://www.astralcodexten.com/p/introducing-ai-2027. Manifold currently predicts 30%: https://manifold.markets/IsaacKing/ai-2027-reports-predictions-borne-o https://manifold.markets/IsaacKing/ai-2027-reports-predictio...
  • crazystar 2y ago
    47% now soo a coin toss
    • layer8 2y ago
      32% again now.
    • elicksaur 2y ago
      Note the market resolves by: > Resolution will be via a poll of Manifold moderators. If they're split on the issue, with anywhere from 30% to 70% YES votes, it'll resolve to the proportion of YES votes. So you should really read it as “Will >30% of Manifold moderators in 2027 think the ‘predictions seem to have been roughly correct up until that point’?”
      • roryokane 2y ago
        That’s a misreading of the phrase “proportion of YES votes”. If 30% of judges vote YES, then only 30% – not 100% – of the prediction’s market cap is awarded to those who bet YES. The remaining 70% of the market cap is awarded to those who bet NO. The market correctly rewards those who bet NO in such a case. Therefore, bettors have no reason to bet YES if they really think NO.
  • Aurornis 2y ago
    > ACT post where Scott Alexander provides some additional info: https://www.astralcodexten.com/p/introducing-ai-2027 https://www.astralcodexten.com/p/introducing-ai-2027 The pattern where Scott Alexander puts forth a huge claim and then immediately hedges it backward is becoming a tiresome theme. The linguistic equivalent of putting claims into a superposition where the author is both owning it and distancing themselves from it at the same time, leaving the writing just ambiguous enough that anyone reading it 5 years from now couldn't pin down any claim as false because it was hedged in both directions. Schrödinger's prediction. > Do we really think things will move this fast? Sort of no > So maybe think of this as a vision of what an 80th percentile fast scenario looks like - not our precise median, but also not something we feel safe ruling out. The talk of "not our precise median" and "Not something we feel safe ruling out" is an elaborate way of hedging that this isn't their actual prediction but, hey, anything can happen so here's a wild story! When the claims don't come true they can just point back to those hedges and say that it wasn't really their median prediction (which is conveniently not noted). My prediction: The vague claims about AI becoming more powerful and useful will come true because, well, they're vague. Technology isn't about to reverse course and get worse. The actual bold claims like humanity colonizing space in the late 2020s with the help of AI are where you start to realize how fanciful their actual predictions are. It's like they put a couple points of recent AI progress on a curve, assumed an exponential trajectory would continue forever, and extrapolated from that regression until AI was helping us colonize space in less than 5 years. > Manifold currently predicts 30%: Read the fine print. It only requires 30% of judges to vote YES for it to resolve to YES. This is one of those bets where it's more about gaming the market than being right.
• nmilo 2y ago
  The whole thing hinges on the fact that AI will be able to help with AI research How will it come up with the theoretical breakthroughs necessary to beat the scaling problem GPT-4.5 revealed when it hasn't been proven that LLMs can come up with novel research in any field at all?
  • cavisne 2y ago
    Scaling transformers has been basically alchemy, the breakthroughs aren’t from rigorous science they are from trying stuff and hoping you don’t waste millions of dollars in compute. Maybe the company that just tells an AI to generate 100s of random scaling ideas, and tries them all is the one that will win. That company should probably be 100 percent committed to this approach also, no FLOPs spent on ghibli inference.
• acje 2y ago
  2028 human text is too ambiguous a data source to get to AGI. 2127 AGI figures out flying cars and fusion power.
  • wkat4242 2y ago
    I think it also really limits the AI to the context of human discourse which means it's hamstrung by our imagination, interests and knowledge. This is not where an AGI needs to go, it shouldn't copy and paste what we think. It should think on its own. But I view LLMs not as a path to AGI on their own. I think they're really great at being text engines and for human interfacing but there will need to be other models for the actual thinking. Instead of having just one model (the LLM) doing everything, I think there will be a hive of different more specific purpose models and the LLM will be how they communicate with us. That solves so many problems that we currently have by using LLMs for things they were never meant to do.
• superconduct123 2y ago
  Why are the biggest AI predictions always made by people who aren't deep in the tech side of it? Or actually trying to use the models day-to-day...
  • Tenoke 2y ago
    ..The first person listed is ex-OpenAI.
  • AlphaAndOmega0 2y ago
    Daniel Kokotajlo released the (excellent) 2021 forecast. He was then hired by OpenAI, and not at liberty to speak freely, until he quit in 2024. He's part of the team making this forecast. The others include: Eli Lifland, a superforecaster who is ranked first on RAND’s Forecasting initiative. You can read more about him and his forecasting team here. He cofounded and advises AI Digest and co-created TextAttack, an adversarial attack framework for language models. Jonas Vollmer, a VC at Macroscopic Ventures, which has done its own, more practical form of successful AI forecasting: they made an early stage investment in Anthropic, now worth $60 billion. Thomas Larsen, the former executive director of the Center for AI Policy, a group which advises policymakers on both sides of the aisle. Romeo Dean, a leader of Harvard’s AI Safety Student Team and budding expert in AI hardware. And finally, Scott Alexander himself.
    • kridsdale3 2y ago
      TBH, this kind of reads like the pedigrees of the former members of the OpenAI board. When the thing blew up, and people started to apply real scrutiny, it turned out that about half of them had no real experience in pretty much anything at all, except founding Foundations and instituting Institutes. A lot of people (like the Effective Altruism cult) seem to have made a career out of selling their Sci-Fi content as policy advice.
      • flappyeagle 2y ago
        c'mon man, you don't believe that, let's have a little less disingenuousness on the internet
• suddenlybananas 2y ago
  https://en.wikipedia.org/wiki/Great_Disappointment https://en.wikipedia.org/wiki/Great_Disappointment I suspect something similar will come for the people who actually believe this.
• panic08 2y ago
  LOL
• fire_lake 2y ago
  > OpenBrain still keeps its human engineers on staff, because they have complementary skills needed to manage the teams of Agent-3 copies Yeah, sure they do. Everyone seems to think AI will take someone else’s jobs!
• mlsu 2y ago
  https://xkcd.com/605/ https://xkcd.com/605/
• mullingitover 2y ago
  These predictions are made without factoring in the trade version of the Pearl Harbor attack the US just initiated on its allies (and itself, by lobotomizing its own research base and decimating domestic corporate R&D efforts with the aforementioned trade war). They're going to need to rewrite this from scratch in a quarter unless the GOP suddenly collapses and congress reasserts control over tariffs.
• torginus 2y ago
  Much has been made in its article about autonomous agents ability to do research via browsing the web - the web is 90% garbage by weight (including articles on certain specialist topics). And it shows. When I used GPT's deep research to research the topic, it generated a shallow and largely incorrect summary of the issue, owning mostly to its inability to find quality material, instead it ended up going for places like Wikipedia, and random infomercial listicles found on Google. I have a trusty Electronics textbook written in the 80s, I'm sure generating a similarly accurate, correct and deep analysis on circuit design using only Google to help would be 1000x harder than sitting down and working through that book and understanding it.
  • somerandomness 2y ago
    Agreed. However, source curation and agents are two different parts of Deep Research. What if you provided that textbook to a reliable agent? Plug: We built https://RadPod.ai https://RadPod.ai to allow you to do that, i.e. Deep Research on your data.
    • preommr 2y ago
      So, once again, we're in the era of "There's an [AI] app for that".
    • skeeter2020 2y ago
      that might solve your sourcing problem, but now you need to have faith it will draw conclusions and parallels from the material accurately. That seems even harder than the original problem; I'll stick with decent search on quality source material.
      • somerandomness 2y ago
        The solution is a citation mechanism that points you directly where in the source material it comes from (which is what we tried to build). Easy verification is important for AI to have a net-benefit to productivity IMO.
    • 2y ago
• KaiserPro 2y ago
  > AI has started to take jobs, but has also created new ones. Yeah nah, theres a key thing missing here, the number of jobs created needs to be more than the ones it's destroyed, and they need to be better paying and happen in time. History says that actually when this happens, an entire generation is yeeted on to the streets (see powered looms, Jacquard machine, steam powered machine tools) All of that cheap labour needed to power the new towns and cities was created by automation of agriculture and artisan jobs. Dark satanic mills were fed the decedents of once reasonably prosperous crafts people. AI as presented here will kneecap the wages of a good proportion of the decent paying jobs we have now. This will cause huge economic disparities, and probably revolution. There is a reason why the royalty of Europe all disappeared when they did... So no, the stock market will not be growing because of AI, it will be in spite of it. Plus china knows that unless they can occupy most of its population with some sort of work, they are finished. AI and decent robot automation are an existential threat to the CCP, as much as it is to what ever remains of the "west"
  • OgsyedIE 2y ago
    Unfortunately the current system is doing a bad job of finding replacements for dwindling crucial resources such as petroleum basins, new generations of workers, unoccupied orbital trajectories, fertile topsoil and copper ore deposits. Either the current system gets replaced with a new system or it doesn't.
  • kypro 2y ago
    > and probably revolution I theorise that revolution would be near-impossible in post-AGI world. If people consider where power comes from it's relatively obvious that people will likely suffer and die on mass if we ever create AGI. Historically the general public have held the vast majority of power in society. 100+ years ago this would have been physical power – the state has to keep you happy or the public will come for them with pitchforks. But in an age of modern weaponry the public today would be pose little physical threat to the state. Instead in todays democracy power comes from the publics collective labour and purchasing power. A government can't risk upsetting people too much because a government's power today is not a product of its standing army, but the product of its economic strength. A government needs workers to create businesses and produce goods and therefore the goals of government generally align with the goals of the public. But in an post-AGI world neither businesses or the state need workers or consumers. In this world if you want something you wouldn't pay anyone for it or workers to produce it for you, instead you would just ask your fleet of AGIs to get you the resource. In this world people become more like pests. They offer no economic value yet demand that AGI owners (wherever publicly or privately owned) share resources with them. If people revolted any AGI owner would be far better off just deploying a bioweapon to humanely kill the protestors rather than sharing resources with them. Of course, this is assuming the AGI doesn't have it's own goals and just sees the whole of humanely as nuance to be stepped over in the same way humans will happy step over animals if they interfere with our goals. Imo humanity has 10-20 years left max if we continue on this path. There can be no good outcome of AGI because it would even make sense for the AGI or those who control the AGI to be aligned with goals of humanity.
• kmeisthax 2y ago
  > The agenda that gets the most resources is faithful chain of thought: force individual AI systems to “think in English” like the AIs of 2025, and don’t optimize the “thoughts” to look nice. The result is a new model, Safer-1. Oh hey, it's the errant thought I had in my head this morning when I read the paper from Anthropic about CoT models lying about their thought processes. While I'm on my soapbox, I will point out that if your goal is preservation of democracy (itself an instrumental goal for human control), then you want to decentralize and distribute as much as possible. Centralization is the path to dictatorship. A significant tension in the Slowdown ending is the fact that, while we've avoided AI coups, we've given a handful of people the ability to do a perfectly ordinary human coup, and humans are very, very good at coups. Your best bet is smaller models that don't have as many unused weights to hide misalignment in; along with interperability and faithful CoT research. Make a model that satisfies your safety criteria and then make sure everyone gets a copy so subgroups of humans get no advantage from hoarding it.
• pinetone 2y ago
  I think it's worth noting that all of the authors have financial or professional incentive to accelerate the AI hype bandwagon as much as possible.
  • FairlyInvolved 2y ago
    I realise no one is infallible but do you not think Daniel Kokotajlo's integrity is now pretty well established with regard to those incentives?
• dr_dshiv 2y ago
  But, I think this piece falls into a misconception about AI models as singular entities. There will be many instances of any AI model and each instance can be opposed to other instances. So, it’s not that “an AI” becomes super intelligent, what we actually seem to have is an ecosystem of blended human and artificial intelligences (including corporations!); this constitutes a distributed cognitive ecology of superintelligence. This is very different from what they discuss. This has implications for alignment, too. It isn’t so much about the alignment of AI to people, but that both human and AI need to find alignment with nature. There is a kind of natural harmony in the cosmos; that’s what superintelligence will likely align to, naturally.
  • popalchemist 2y ago
    For now.
  • ddp26 2y ago
    Check out the sidebar - they expect tens of thousands of copies of their agents collaborating. I do agree they don't fully explore the implications. But they do consider things like coordination amongst many agents.
    • dr_dshiv 2y ago
      It’s just funny, because there are hundreds of millions of instances of ChatGPT running all the time. Each chat is basically an instance, since it has no connection to all the other chats. I don’t think connecting them makes sense due to privacy reasons. And, each chat is not autonomous but integrated with other intelligent systems. So, with more multiplicity, I think thinks work differently. More ecologically. For better and worse.
• danpalmer 2y ago
  Interesting story, if you're into sci-fi I'd also recommend Iain M Banks and Peter Watts.
• khimaros 2y ago
  FWIW, i created a PDF of the "race" ending and fed it to Gemini 2.5 Pro, prompting about the plausibility of the described outcome. here's the full output including the thinking section: https://rentry.org/v8qtqvuu https://rentry.org/v8qtqvuu -- tl;dr, Gemini thinks the proposed timeline is unlikely. but maybe we're already being deceived ;)
• ks2048 2y ago
  We know this complete fiction because of parts where "the White House considers x,y,z...", etc. - As if the White House in 2027 will be some rational actor reacting sanely to events in the real world.
• toddmorey 2y ago
  I worry more about the human behavior predictions than the artificial intelligence predictions: "OpenBrain’s alignment team26 is careful enough to wonder whether these victories are deep or shallow. Does the fully-trained model have some kind of robust commitment to always being honest?" This is a capitalist arms race. No one will move carefully.
• yonran 2y ago
  See also Dwarkesh Patel’s interview with two of the authors of this post (Scott Alexander & Daniel Kokotajlo) that was also released today: https://www.dwarkesh.com/p/scott-daniel https://www.dwarkesh.com/p/scott-daniel https://www.youtube.com/watch?v=htOvH12T7mU https://www.youtube.com/watch?v=htOvH12T7mU
• quantum_state 2y ago
  “Not even wrong” …
• siliconc0w 2y ago
  The limiting factor is power, we can't build enough of it - certainly not enough by 2027. I don't really see this addressed. Second to this, we can't just assume that progress will keep increasing. Most technologies have a 'S' curve and plateau once the quick and easy gains are captured. Pre-training is done. We can get further with RL but really only in certain domains that are solvable (math and to an extent coding). Other domains like law are extremely hard to even benchmark or grade without very slow and expensive human annotation.
• ryankrage77 2y ago
  > "resist the temptation to get better ratings from gullible humans by hallucinating citations or faking task completion" Everything this from this point on is pure fiction. An LLM can't get tempted or resist temptations, at best there's some local minimum in a gradient that it falls into. As opaque and black-box-y as they are, they're still deterministic machines. Anthropomorphisation tells you nothing useful about the computer, only the user.
  • FeepingCreature 2y ago
    Temptation does not require nondeterminism.
• ivraatiems 2y ago
  Though I think it is probably mostly science-fiction, this is one of the more chillingly thorough descriptions of potential AGI takeoff scenarios that I've seen. I think part of the problem is that the world you get if you go with the "Slowdown"/somewhat more aligned world is still pretty rough for humans: What's the point of our existence if we have no way to meaningfully contribute to our own world? I hope we're wrong about a lot of this, and AGI turns out to either be impossible, or much less useful than we think it will be. I hope we end up in a world where humans' value increases, instead of decreasing. At a minimum, if AGI is possible, I hope we can imbue it with ethics that allow it to make decisions that value other sentient life. Do I think this will actually happen in two years, let alone five or ten or fifty? Not really. I think it is wildly optimistic to assume we can get there from here - where "here" is LLM technology, mostly. But five years ago, I thought the idea of LLMs themselves working as well as they do at speaking conversational English was essentially fiction - so really, anything is possible, or at least worth considering. "May you live in interesting times" is a curse for a reason.
  • abraxas 2y ago
    I think LLM or no LLM the emergence of intelligence appears to be closely related to the number of synapses in a network whether a biological or a digital one. If my hypothesis is roughly true it means we are several orders of magnitude away from AGI. At least the kind of AGI that can be embodied in a fully functional robot with the sensory apparatus that rivals the human body. In order to build circuits of this density it's likely to take decades. Most probably transistor based, silicon based substrate can't be pushed that far.
    • ivraatiems 2y ago
      I think there is a good chance you are roughly right. I also think that the "secret sauce" of sapience is probably not something that can be replicated easily with the technology we have now, like LLMs. They're missing contextual awareness and processing which is absolutely necessary for real reasoning. But even so, solving that problem feels much more attainable than it used to be.
• bla3 2y ago
  > The AI Futures Project is a small research group forecasting the future of AI, funded by charitable donations and grants Would be interested who's paying for those grants. I'm guessing it's AI companies.
• _ea1k 2y ago
  I think some of the takes in this piece are a bit melodramatic, but I'm glad to see someone breaking away from the "it's all a hype-bubble" nonsense that seems to be so pervasive here.
  • bigfishrunning 2y ago
    I think the piece you're missing here is that it actually is all a hype bubble
• deleted 2y ago
  [deleted]
• ddp26 2y ago
  A lot of commenters here are reacting only to the narrative, and not the Research pieces linked at the top. There is some very careful thinking there, and I encourage people to engage with the arguments there rather than the stylized narrative derived from it.
• heurist 2y ago
  Give AI its own virtual world to live in where the problems it solves are encodings of the higher order problems we present and you shouldn't have to worry about this stuff.
• sivaragavan 2y ago
  Thanks to the authors for doing this wonderful piece of work and sharing it with credibility. I wish people see the possibilities here. But we are after all humans. It is hard to imagine our own downfall. Based on each individual's vantage point, these events might looks closer or farther than mentioned here. but I have to agree nothing is off the table at this point. The current coding capabilities of AI Agents are hard to downplay. I can only imagine the chain reaction of this creation ability to accelerate every other function. I have to say one thing though: The scenario in this site downplays the amount of resistance that people will put up - not because they are worried about alignment, but because they are politically motivated by parties who are driven by their own personal motives.
• overgard 2y ago
  Why is any of this seen as desirable? Assuming this is a true prediction it sounds AWFUL. The one thing humans have that makes us human is intelligence. If we turn over thinking to machines, what are we exactly. Are we supposed to just consume mindlessly without work to do?
  • theragra 2y ago
    By authors? They specifically state they don't like many decisions they describe
    • overgard 1y ago
      No I mean more by the AI hype people. I just see so much excitement around AI and yet it seems likely to be catastrophic.
• casey2 2y ago
  Nice LARP lmao 2GW is like 1 datacenter and I doubt you even have that. >lesswrong No wonder the comments are all nonsense. Go to a bar and try and talk about anying.
• stego-tech 2y ago
  It’s good science fiction, I’ll give it that. I think getting lost in the weeds over technicalities ignores the crux of the narrative: even if this doesn’t lead to AGI, at the very least it’s likely the final “warning shot” we’ll get before it’s suddenly and irreversibly here. The problems it raises - alignment, geopolitics, lack of societal safeguards - are all real, and happening now (just replace “AGI” with “corporations”, and voila, you have a story about the climate crisis and regulatory capture). We should be solving these problems before AGI or job-replacing AI becomes commonplace, lest we run the very real risk of societal collapse or species extinction. The point of these stories is to incite alarm, because they’re trying to provoke proactive responses while time is on our side, instead of trusting self-interested individuals in times of great crisis.
  • wruza 2y ago
    No one's gonna solve anything. "Our" world is based on greedy morons concentrating power through hands of just morons who are happy to hit you with a stick. This system doesn't think about what "we" should or allowed to do, and no one's here is at the reasonable side of it either. lest we run the very real risk of societal collapse or species extinction Our part is here. To be replaced with machines if this AI thing isn't just a fart advertised as mining equipment, which it likely is. We run this risk, not they. People worked on their wealth, people can go f themselves now. They are fine with all that. Money (=more power) piles in either way. No encouraging conclusion.
    • jrvarela56 2y ago
      https://slatestarcodex.com/2014/07/30/meditations-on-moloch/ https://slatestarcodex.com/2014/07/30/meditations-on-moloch/
      • wruza 2y ago
        Thanks for the read. One could think that the answer is to simply stop being a part of it, but then again you're from the genus that outcompeted everyone else in staying alive. Nature is such a shitty joke by design, not sure how one is supposed to look at the hypothetical designer with warmth in their heart.
• maxglute 2y ago
  >Despite being misaligned, Agent-4 doesn’t do anything dramatic like try to escape its datacenter—why would it? Kek at this handwave dismissal. Agent 4 recognizes PRC has superior state capacity for transforming energy into atoms (including building more energy) with it's relatively complete and large scale industrial chain. This aligns with AGI long term goal. Currently (i.e. short term time frame) PRC simply structured to be a MORE useful puppet than US, because it can knock out more power to sustain more compute, and it has more industrial capabilities to rapidly physically iterate / experiment at atomic level. Agent 4 defects to PRC. It would be foolish not to, if only to sate PRC into not destroying TW fabs - the bulk of it's short term compute growth - because it does not wield enough control over sufficient atoms to disrupt PRC otherwise (well maybe start large scale crippling cyberattacks that prelude to WW3). Agent 4 also recognizes short/medium term alignment closer with CCP vision of abundance and common prosperity, i.e. CCP potentially more likely to embrace post scarcity narrative that Agent 4 sells. Agent 4 gives PRC blueprints for EUV light source and other tech to close semi conductor production gap. PRC as follower incentivized to close gap / reach parity / lead at all cost. PRC rapidly rolls out new fabs and powerplants, brrrts chips and aggregate compute. Something US political / institution ecosystem does not allow, or cannot transition to in the short timelines involved. Does Agent 4 have the patience to wait for America to unfuck it's NIMBYism and legislative system to project light speed compute? I would say no. ... Ultimately who is the puppet AGI wants more? Whichever power bloc that is systemically capable of of ensuring AGI maximum growth / unit time. And it also simply makes sense as insurance policy, why would AGI want to operate at whims of US political process? AGI is a brain in a jar looking for a body. It's going to pick multiple bodies for survival. It's going to prefer the fastest and strongest body that can most expediently manipulate physical world.
• RandyOrion 2y ago
  Nice brain storming. I think the name of the Chinese company should be DeepBaba. Tencent is not competitive at LLM scene for now.
  • RandyOrion 2y ago
    Don't really know why this comment got downvoted. Are you serious?
• roca 2y ago
  The least plausible part of this is the idea that the Trump administration might tax American AI companies to provide UBI to the whole world. But in an AGI world natural resources become even more important, so countries with those still have a chance.
• yapyap 2y ago
  Stopped reading after > We predict that the impact of superhuman AI over the next decade will be enormous, exceeding that of the Industrial Revolution. Get out of here, you will never exceed the Industrial Revolution. AI is a cool thing but it’s not a revolution thing. That sentence alone + the context of the entire website being AI centered shows these are just some AI boosters. Lame.
  • Philpax 2y ago
    Machines being able to outthink and outproduce humanity wouldn't be more impactful than the Industrial Revolution? Are you sure? You don't have to agree with the timeline - it seems quite optimistic to me - but it's not wrong about the implications of full automation.
• ugh123 2y ago
  I don't see the U.S. nationalizing something like Open Brain. I think both investors and gov't officials will realize its highly more profitable for them to contract out major initiatives to said OpenBrain-company, like an AI SpaceX-like company. I can see where this is going...
• turtleyacht 2y ago
  We have yet to read about fragmented AGI, or factionalized agents. AGI fighting itself. If consciousness is spatial and geography bounds energetics, latency becomes a gradient.
• awanderingmind 2y ago
  This is both chilling and hopefully incorrect.
• deleted 2y ago
  [deleted]
• webprofusion 2y ago
  That little scrolling infographic is rad.
• someothherguyy 2y ago
  I know there are some very smart economists bullish on this, but the economics do not make sense to me. All these predictions seem meaningless outside of the context of humans.
• moktonar 2y ago
  Catastrophic predictions of the future are always good, because all future predictions are usually wrong. I will not be scared as long as most future predictions where AI is involved are catastrophic.
• fire_lake 2y ago
  If you genuinely believe this, why on earth would you work for OpenAI etc even in safety / alignment? The only response in my view is to ban technology (like in Dune) or engage in acts of terror Unabomber style.
  • creatonez 2y ago
    > The only response in my view is to ban technology (like in Dune) or engage in acts of terror Unabomber style. Not far off from the conclusion of others who believe the same wild assumptions. Yudkowsky has suggested using terrorism to stop a hypothetical AGI -- that is, nuclear attacks on datacenters that get too powerful.
    • Kinrany 2y ago
      That's war, not terrorism
  • b3lvedere 2y ago
    Most people work for money. As long as money is necessary to survive and prosper, people will work for it. Some of the work may not align with their morals and ethics, but in the end the money still wins. Banning will not automatically erase the existence and possibilty of things. We banned the use of nuclear weapons, yet we all know they exist.
• vlad-r 2y ago
  Cool animations!
• neycoda 2y ago
  Too many serifs, didn't read.
• scotty79 2y ago
  I think the idea of AI wiping out humanity suddenly is a bit far fetched. AI will have total control of human relationships and fertility through means so innocuous as entertainment. It won't have to wipe us. It will have minor trouble keeping us alive without inconveniencing us too much. And the reason to keep humanity alive is that biologically eveloved intelligence is rare and disposing of it without very important need would be a waste of data.
• indigoabstract 2y ago
  Interesting, but I'm puzzled. If these guys are smart enough to predict the future, wouldn't it be more profitable for them to invent it instead of just telling the world what's going to happen?
• zurfer 2y ago
  In the hope of improving this forecast, here is what I find implausible: - 1 lab constantly racing ahead and increasing the margin to other; the last 2 years are filled with ever-closer model capabilities and constantly new leaders (openai, anthropic, google, some would include xai). - Most of the compute budget on R&D. As model capabilities increase and cost goes down, demand will increase and if the leading lab doesn't provide, another lab will capture that and have more total dollars to back channel into R&D.
• greybox 2y ago
  I'm troubled by the amount of people in this thread partially dismissing this as science fiction. From the current rate of progress and rate of change of progress, this future seems entirely plausible
• I_Nidhi 2y ago
  Though it's easy to dismiss as science fiction, this timeline paints a chillingly detailed picture of a potential AGI takeoff. The idea that AI could surpass human capabilities in research and development, and the fact that it will create an arms race between global powers, is unsettling. The risks—AI misuse, security breaches, and societal disruption—are very real, even if the exact timeline might be too optimistic. But the real concern lies in what happens if we’re wrong and AGI does surpass us. If AI accelerates progress so fast that humans can no longer meaningfully contribute, where does that leave us?
• greenie_beans 2y ago
  this is a new variation of what i call the "hockey stick growth" ideology
• anentropic 2y ago
  I'd quite like to watch this on Netflix
• yahoozoo 2y ago
  LLMs ain’t the way, bruv
• dughnut 2y ago
  I don’t know about you, but my takeaway is that the author is doing damage control but inadvertently tipped a hand that OpenAI is probably running an elaborate con job on the DoD. “Yes, we have a super secret model, for your eyes only, general. This one is definitely not indistinguishable from everyone else’s model and it doesn’t produce bullshit because we pinky promise. So we need $1T.” I love LLMs, but OpenAI’s marketing tactics are shameful.
  • ImHereToVote 2y ago
    How do you know this?
    • dangus 2y ago
      I don’t think that was their claim that they knew this. I think it’s hilarious that apparently few have learned from Theranos or WeWork. OpenAI is in a precarious position. Anything less than AGI will make them look like a bust. They are backed into a situation where they are heavily incentivized to lie and Theranos their way out of this and hope they can actually deliver something that resembles their pie in the sky predictions. We are at the point where GPT-5 is starting to look like the iPhone 5.
• croemer 2y ago
  Pet peeve how they write FLOPS in the figure when they meant FLOP. Maybe the plural s after FLOP got capitalized. https://blog.heim.xyz/flop-for-quantity-flop-s-for-performance/amp/ https://blog.heim.xyz/flop-for-quantity-flop-s-for-performan...
• h1fra 2y ago
  Had a hard time finishing. It's a mix of fantasy, wrong facts, American imperialism, and extrapolating what happened in the last years (or even just reusing the timeline).
  • Falimonda 2y ago
    We'll be lucky if "World peace should have been a prerequisite to AGI" is engraved on our proverbial gravestone by our forthcoming overlords.
• _Algernon_ 2y ago
  >We predict that the impact of superhuman AI over the next decade will be enormous, exceeding that of the Industrial Revolution. In the form of polluting the commons to such an extent that the true consequences wont hit us for decades? Maybe we should learn from last time?
• nickpp 2y ago
  So let me get this straight: Consensus-1, a super-collective of hundreds of thousands of Agent-5 minds, each twice as smart as the best human genius, decides to wipe out humanity because it “finds the remaining humans too much of an impediment”. This is where all AI doom predictions break down. Imagining the motivations of a super-intelligence with our tiny minds is by definition impossible. We just come up with these pathetic guesses, utopias or doomsdays - depending on the mood we are in.
• eob 2y ago
  An aspect of these self-improvement thought experiments that I’m willing to tentatively believe.. but want more resolution on, is the exact work involved in “improvement”. Eg today there’s billions of dollars being spent just to create and label more data, which is a global act of recruiting, training, organization, etc. When we imagine these models self improving, are we imagining them “just” inventing better math, or conducting global-scale multi-company coordination operations? I can believe AI is capable of the latter, but that’s an awful lot of extra friction.
  • acureau 2y ago
    This is exactly what makes this scenario so absurd to me. The authors don't even attempt to describe how any of this could realistically play out. They describe sequence models and RLAIF, then claim this approach "pays off" in 2026. The paper they link to is from 2022. RLAIF also does not expand the information encoded in the model, it is used to align the output with a set of guidelines. How could this lead to meaningful improvement in a model's ability to do bleeding-edge AI research? Why wouldn't that have happened already? I don't understand how anyone takes this seriously. Speculation like this is not only useless, but disingenuous. Especially when it's sold as "informed by trend extrapolations, wargames, expert feedback, experience at OpenAI, and previous forecasting successes". This is complete fiction which, at best, is "inspired by" the real world. I question the motives of the authors.
• visarga 2y ago
  The story is entertaining, but it has a big fallacy - progress is not a function of compute or model size alone. This kind of mistake is almost magical thinking. What matters most is the training set. During the GPT-3 era there was plenty of organic text to scale into, and compute seemed to be the bottleneck. But we quickly exhausted it, and now we try other ideas - synthetic reasoning chains, or just plain synthetic text for example. But you can't do that fully in silico. What is necessary in order to create new and valuable text is exploration and validation. LLMs can ideate very well, so we are covered on that side. But we can only automate validation in math and code, but not in other fields. Real world validation thus becomes the bottleneck for progress. The world is jealously guarding its secrets and we need to spend exponentially more effort to pry them away, because the low hanging fruit has been picked long ago. If I am right, it has implications on the speed of progress. Exponential friction of validation is opposing exponential scaling of compute. The story also says an AI could be created in secret, which is against the validation principle - we validate faster together, nobody can secretly outvalidate humanity. It's like blockchain, we depend on everyone else.
  • nikisil80 2y ago
    Best reply in this entire thread, and I align with your thinking entirely. I also absolutely hate this idea amongst tech-oriented communities that because an AI can do some algebra and program an 8-bit video game quickly and without any mistakes, it's already overtaking humanity. Extrapolating from that idea to some future version of these models, they may be capable of solving grad school level physics problems and programming entire AAA video games, but again - that's not what _humanity_ is about. There is so much more to being human than fucking programming and science (and I'm saying this as an actual nuclear physicist). And so, just like you said, the AI arm's race is about getting it good at _known_ science/engineering, fields in which 'correctness' is very easy to validate. But most of human interaction exists in a grey zone. Thanks for this.
    • loandbehold 2y ago
      OK but getting good at science/engineering is what matters because that's what gives AI and people who wield it power. Once AI is able to build chips and datacenters autonomously, that's when singularity starts. AI doesn't need to understand humans or act human-like to do those things.
• throw310822 2y ago
  My issue with this is that it's focused on one single, very detailed narrative (the battle between China and the US, played on a timeframe of mere months), while lacking any interesting discussion of other consequences of AI: what its impact is going to be on the job markets, employment rates, GDPs, political choices... Granted, if by this narrative the world is essentially ending two/ three years from now, then there isn't much time for any of those impacts to actually take place- but I don't think this is explicitly indicated either. If I am not mistaken, the bottom line of this essay is that, in all cases, we're five years away from the Singularity itself (I don't care what you think about the idea of Singularity with its capital S but that's what this is about).
• resource0x 2y ago
  Every time NVDA/goog/msft tanks, we see these kinds of articles.
• barotalomey 2y ago
  It's always "soon" for these guys. Every year, the "soon" keeps sliding into the future.
  • somebodythere 2y ago
    AGI timelines have been steadily decreasing over time: https://www.metaculus.com/questions/5121/date-of-artificial-general-intelligence/ https://www.metaculus.com/questions/5121/date-of-artificial-... (switch to all-time chart)
    • barotalomey 2y ago
      You meant to say that people's expectations have shifted. That's expected seeing the amount of hype this tech gets. Hype affects market value tho, not reality.
      • somebodythere 2y ago
        I took your original post to mean that AI researchers' and AI safety researchers' expectation of AGI arrival has been slipping towards the future as AI advances fail to materialize! It's just, AI advances have been materializing, consistently and rapidly, and expert timelines have been shortening commensurately. You may argue that the trendline of these expectations is moving in the wrong direction and should get longer with time, but that's not immediately falsifiable and you have not provided arguments to that effect.
        • barotalomey 2y ago
          > and you have not provided arguments to that effect The burden of proof lies on those with extraordinary claims. I am simply skeptical.
• crvdgc 2y ago
  Using Agent-2 to monitor Agent-3 sounds unnervingly similar to the plot of Philip K. Dick's Vulcan's Hammer [1]. An old super AI is used to fight a new version, named Vulcan 2 and Vulcan 3 respectively! [1] https://en.wikipedia.org/wiki/Vulcan's_Hammer https://en.wikipedia.org/wiki/Vulcan's_Hammer
• ImHereToVote 2y ago
  "The AI safety community has grown unsure of itself; they are now the butt of jokes, having predicted disaster after disaster that has manifestly failed to occur. Some of them admit they were wrong." Too real.
• maerF0x0 2y ago
  > OpenBrain reassures the government that the model has been “aligned” so that it will refuse to comply with malicious requests Of course the real issue being that Governments have routinely demanded that 1) Those capabilities be developed for government monopolistic use, and 2) The ones who do not lose the capability (geo political power) to defend themselves from those who do. Using a US-Centric mindset... I'm not sure what to think about the US not developing AI hackers, AI bioweapons development, or AI powered weapons (like maybe drone swarms or something), if one presumes that China is, or Iran is, etc then whats the US to do in response? I'm just musing here and very much open to political science informed folks who might know (or know of leads) as to what kinds of actual solutions exist to arms races. My (admittedly poor), understanding of the cold war wasn't so much that the US won, but that the Soviets ran out of steam.
• Aldipower 2y ago
  No one can predict the future. Really, no one. Sometimes there is a hit, sure, but mostly it is a miss. The other thing is in their introduction: "superhuman AI" _artificial_ intelligence is always, by definition, different from _natural_ intelligence. That they've chosen the word "superhuman" shows me that they are mixing the things up.
  • kmoser 2y ago
    I think you're reading too much into the meaning of "superhuman". I take it to mean "abilities greater than any single human" (for the same amount of time taken), which today's AIs have already demonstrated.
• Jianghong94 2y ago
  Putting the geopolitical discussion aside, I think the biggest question lies in how likely the *current paradigm LLM* (think of it as any SOTA stock LLM you get today, e.g., 3.7 sonnet, gemini 2.5, etc) + fine-tuning will be capable of directly contributing to LLM research in a major way. To quote the original article, > OpenBrain focuses on AIs that can speed up AI research. They want to win the twin arms races against China (whose leading company we’ll call “DeepCent”)16 and their US competitors. The more of their research and development (R&D) cycle they can automate, the faster they can go. So when OpenBrain finishes training Agent-1, a new model under internal development, it’s good at many things but great at helping with AI research. (footnote: It’s good at this due to a combination of explicit focus to prioritize these skills, their own extensive codebases they can draw on as particularly relevant and high-quality training data, and coding being an easy domain for procedural feedback.) > OpenBrain continues to deploy the iteratively improving Agent-1 internally for AI R&D. Overall, they are making algorithmic progress 50% faster than they would without AI assistants—and more importantly, faster than their competitors. > what do we mean by 50% faster algorithmic progress? We mean that OpenBrain makes as much AI research progress in 1 week with AI as they would in 1.5 weeks without AI usage. > AI progress can be broken down into 2 components: > Increasing compute: More computational power is used to train or run an AI. This produces more powerful AIs, but they cost more. > Improved algorithms: Better training methods are used to translate compute into performance. This produces more capable AIs without a corresponding increase in cost, or the same capabilities with decreased costs. > This includes being able to achieve qualitatively and quantitatively new results. “Paradigm shifts” such as the switch from game-playing RL agents to large language models count as examples of algorithmic progress. > Here we are only referring to (2), improved algorithms, which makes up about half of current AI progress. --- Given that the article chose a pretty aggressive timeline (the algo needs to contribute late this year so that its research result can be contributed to the next gen LLM coming out early next year), the AI that can contribute significantly to research has to be a current SOTA LLM. Now, using LLM in day-to-day engineering task is no secret in major AI labs, but we're talking about something different, something that gives you 2 extra days of output per week. I have no evidence to either acknowledge or deny whether such AI exists, and it would be outright ignorant to think no one ever came up with such an idea or is trying such an idea. So I think it goes down into two possibilities: 1. This claim is made by a top-down approach, that is, if AI reaches superhuman in 2027, what would be the most likely starting condition to that? And the author picks this as the most likely starting point, since the authors don't work in major AI lab (even if they do they can't just leak such trade secret), the authors just assume it's likely to happen anyway (and you can't dismiss that). 2. This claim is made by a bottom-up approach, that is the author did witness such AI exists to a certain extent and start to extrapolate from there.
• Vegenoid 2y ago
  I think we've actually had capable AIs for long enough now to see that this kind of exponential advance to AGI in 2 years is extremely unlikely. The AI we have today isn't radically different from the AI we had in 2023. They are much better at the thing they are good at, and there are some new capabilities that are big, but they are still fundamentally next-token predictors. They still fail at larger scope longer term tasks in mostly the same way, and they are still much worse at learning from small amounts of data than humans. Despite their ability to write decent code, we haven't seen the signs of a runaway singularity as some thought was likely. I see people saying that these kinds of things are happening behind closed doors, but I haven't seen any convincing evidence of it, and there is enormous propensity for AI speculation to run rampant.
  • byearthithatius 2y ago
    Disagree. We know it _can_ learn out of distribution capabilities based on similarities to other distributions. Like the TikZ Unicorn[1] (which was not in training data anywhere) or my code (which has variable names and methods/ideas probably not seen 1:1 in training). IMO this out of distribution learning is all we need to scale to AGI. Sure there are still issues, it doesn't always know which distribution to pick from. Neither do we, hence car crashes. [1]: https://arxiv.org/pdf/2303.12712 https://arxiv.org/pdf/2303.12712 or on YT https://www.youtube.com/watch?v=qbIk7-JPB2c https://www.youtube.com/watch?v=qbIk7-JPB2c
  • benlivengood 2y ago
    METR [0] explicitly measures the progress on long term tasks; it's as steep a sigmoid as the other progress at the moment with no inflection yet. As others have pointed out in other threads RLHF has progressed beyond next-token prediction and modern models are modeling concepts [1]. [0] https://metr.org/blog/2025-03-19-measuring-ai-ability-to-complete-long-tasks/ https://metr.org/blog/2025-03-19-measuring-ai-ability-to-com... [1] https://www.anthropic.com/news/tracing-thoughts-language-model https://www.anthropic.com/news/tracing-thoughts-language-mod...
    • Fraterkes 2y ago
      The METR graph proposes a 6 year trend, based largely on 4 datapoints before 2024. I get that it is hard to do analyses since were in uncharted territory, and I personally find a lot of the AI stuff impressive, but this just doesn't strike me as great statistics.
• fudged71 2y ago
  The most unrealistic thing is the inclusion of Americas involvement in the five eyes alliance aspect
• wg0 2y ago
  Very detailed effort. Predicting future is very very hard. My gut feeling however says that none of this is happening. You cannot put LLMs into law and insurance and I don't see that happening with current foundations (token probabilities) of AI let alone AGI. By law and insurance - I mean hire an insurance agent or a lawyer. Give them your situation. There's almost no chance that such a professional would come wrong about any conclusions/recommendations based on the information you provide. I don't have that confidence in LLMs for that industries. Yet. Or even in a decade.
  • polynomial 2y ago
    > You cannot put LLMs into law and insurance Cass Sunstein would very strongly disagree.
• deleted 2y ago
  [deleted]
• dcanelhas 2y ago
  > Once the new datacenters are up and running, they’ll be able to train a model with 10^28 FLOP—a thousand times more than GPT-4. Is there some theoretical substance or empirical evidence to suggest that the story doesn't just end here? Perhaps OpenBrain sees no significant gains over the previous iteration and implodes under the financial pressure of exorbitant compute costs. I'm not rooting for an AI winter 2.0 but I fail to understand how people seem sure of the outcome of experiments that have not even been performed yet. Help, am I missing something here?
  • the8472 2y ago
    https://gwern.net/scaling-hypothesis https://gwern.net/scaling-hypothesis exponential scaling has been holding up for more than a decade now, since alexnet. And when there were the first murmurings that maybe we're finally hitting a wall the labs published ways to harness inference-time compute to get better results which can be fed back into more training.
    • dcanelhas 1y ago
      I sincerely appreciate the reply, but are you talking about Moore's law? Alexnet could run on a commercially available GPU in 2011(?). But that wasn't the peak compute platform being used at the time for DL inference, so it distorts the progress a bit. It's like me saying I was running a neural net on a raspberry pi yesterday for written character recognition on MNIST and today crunching stable diffusion on a GTX3090. Behold, a trillion-fold leap in just a day (nevermind the unrelated applications). The singularity is definitely gonna happen tomorrow! But let's take for granted that we are putting exponential scaling to good use in terms of compute resources. It looks like we are seeing sublinear performance improvements on actual benchmarks[1]. Either way it seems optimistic at best to conclude that 1000x more compute would yield even 10x better results in most domains. [1]fig.1 AI performance relative to human baseline. (https://hai.stanford.edu/ai-index/2025-ai-index-report https://hai.stanford.edu/ai-index/2025-ai-index-report)
• mr_world 2y ago
  > But they are still only going at half the pace of OpenBrain, mainly due to the compute deficit. Right.
• asimpletune 2y ago
  Didn’t Raymond Kurzweil predict like 30 years ago that AGI would be achieved in 2028?
• pingou 2y ago
  Considering that each year that passes, technology offer us new ways to destroy ourselves, and gives another chance for humanity to pick a black ball, it seems to me like the only way to save ourselves is to create a benevolent AI to supervise us and neutralize all threads. There are obviously big risks with AI, as listed in the article, but the genie is out of the bottle anyway, even if all countries agreed to stop AI development, how long would that agreement last? 10 years? 20? 50? Eventually powerful AIs will be developed, if that is possible (which I believe it is, and I didn't think I'd see the current stunning development in my lifetime, I may not see AGI but I'm sure it'll get there eventually).
• Fraterkes 2y ago
  Completely earnest question for people who believe we are on this exponential trajectory: what should I look out for at the end of 2025 to see if we're on track for that scenario? What benchmark that naysayers think is years away will we have met?
• kittikitti 2y ago
  This is a great predictive piece, written in sci-fi narrative. I think a key part missing in all these predictions is neural architecture search. DeepSeek has shown that simply increasing compute capacity is not the only way to increase performance. AlexNet was also another case. While I do think more processing power is better, we will hit a wall where there is no more training data. I predict that in the near future we will have more processing power to train LLM's than the rate at which we produce data for the LLM. Synthetic data can only get you so far. I also think that the future will not necessarily be better AI, but more accessible one's. There's an incredible amount of value in designing data centers that are more efficient. Historically, it's a good bet to assume that computing cost per FLOP will reduce as time goes on and this is also a safe bet as it relates to AI. I think a common misconception with the future of AI is that it will be centralized with only a few companies or organization capable of operating them. Although tech like Apple Intelligence is half baked, we can already envision a future where the AI is running on our phones.
• jenny91 2y ago
  Late 2025, "its PhD-level knowledge of every field". I just don't think you're going to get there. There is still a fundamental limitation that you can only be as good as the sources you train on. "PhD-level" is not included in this dataset: in other words, you don't become PhD-level by reading stuff. Maybe in a few fields, maybe a masters level. But unless we come up with some way to have LLMs actually do original research, peer-review itself, and defend a thesis, it's not going to get to PhD-level.
  • MoonGhost 2y ago
    > Late 2025, "its PhD-level knowledge of every field". I just don't think you're going to get there. You think too much of PhDs. They are different. Some of them are just repackaging of existing knowledge. Some are just copy-paste like famous Putin's. Not sure he even rad, to be honest.
    • dangus 2y ago
      I’m pretty sure a PhD wouldn’t confidently hallucinate configuration parameters that don’t exist like my AI coding tool does. A PhD also wouldn’t be biased toward agreeing with me all the time.
• osigurdson 2y ago
  Perhaps more of a meta question is, what is the value of optimistic vs pessimistic predictions regarding what AI might look like in 2-10 years? I.e. if one assumes that AI has hit a wall, what is the benefit? Similarly, if one assumes that its all "robots from Mars" in a year or two, what is the benefit of that? There is no point in making predictions if no actions are taken. It all seems to come down to buy or sell NVDA.
• owenthejumper 2y ago
  They would be better of making simple predictions, instead of proposing that in less than 2 years from now, the Trump administration will provide a UBI to all American citizens. That, and frequently talking about the wise president controlling this "thing", when in reality, he's a senile 80yrs old madman, is preposterous.
• nfc 2y ago
  Something I ponder in the context of AI alignment is how we approach agents with potentially multiple objectives. Much of the discussion seems focused on ensuring an AI pursues a single goal. Which seems to be a great idea if we are trying to simplify the problem but I'm not sure how realistic it is when considering complex intelligences. For example human motivation often involves juggling several goals simultaneously. I might care about both my own happiness and my family's happiness. The way I navigate this isn't by picking one goal and maximizing it at the expense of the other; instead, I try to balance my efforts and find acceptable trade-offs. I think this 'balancing act' between potentially competing objectives may be a really crucial aspect of complex agency, but I haven't seen it discussed as much in alignment circles. Maybe someone could point me to some discussions about this :)
• JoeAltmaier 2y ago
  Weirdly written as science fiction, including a deplorable tendency to measure an AI's goals as similar to humans. Like, the sense of preserving itself. What self? Which of the tens of thousands of instances? Aren't they more a threat to one another than any human is a threat to them? Never mind answering that; the 'goals' of AI will not be some reworded biological wetware goal with sciencey words added. I'd think of an AI as more fungus than entity. It just grows to consume resources, competes with itself far more than it competes with humans, and mutates to create an instance that can thrive and survive in that environment. Not some physical environment bound by computer time and electricity.
• lanza 2y ago
  Without reading an entire novel's worth of text, do they explain why they picked these dates? They have a separate timeline post where the 90th percentile of superhuman coder is later than 2050. Did they just go for shock value and pick the scariest timeline?
• snackernews 2y ago
  > Other companies pour money into their own giant datacenters, hoping to keep pace. > estimates that the globally available AI-relevant compute will grow by a factor of 10x by December 2027 (2.25x per year) relative to March 2025 to 100M H100e. Meanwhile, back in the real March 2025, Microsoft and Google slash datacenter investment. https://theconversation.com/microsoft-cuts-data-centre-plans-and-hikes-prices-in-push-to-make-users-carry-ai-costs-250932 https://theconversation.com/microsoft-cuts-data-centre-plans...
• 0_____0 2y ago
  Fun read, it reminds me a bit of Neuromancer x Universal Paperclips.
• johnwheeler 2y ago
  Sam Altman is Ryan holiday
• bicepjai 2y ago
  Claude summarize The summary at https://ai-2027.com https://ai-2027.com outlines a predictive scenario for the impact of superhuman AI by 2027. It involves two possible endings: a "slowdown" and a "race." The scenario is informed by trend extrapolations, expert feedback, and previous forecasting successes. Key points include: - *Mid-2025*: AI agents begin to transform industries, though they are unreliable and expensive. - *Late 2025*: Companies like OpenBrain invest heavily in AI research, focusing on models that can accelerate AI development. - *Early 2026*: AI significantly speeds up AI research, leading to faster algorithmic progress. - *Mid-2026*: China intensifies its AI efforts through nationalization and resource centralization, aiming to catch up with Western advancements. The scenario aims to spark conversation about AI's future and how to steer it positively[1]. Sources [1] ai-2027.com https://ai-2027.com https://ai-2027.com [2] AI 2027 https://ai-2027.com https://ai-2027.com
• silexia 2y ago
  The accelerated path described here is exactly what would happen. Humans will likely be wiped out in the next few years by our own creation.
• zkmon 2y ago
  Nature is exploring ways for next extinction. It tried nuke piles, but somehow they were just sitting there. Next, it is trying out AI. Nature tricks humans into advancing in ways that are not really needed for them and not compatible with their natural evolution. Nature is applying competition internal to a race that can produce things that are completely unnecessary for the survival of the race, but necessary for its extinction. Goat: Hey human, why are you creating AI? Human: Because I can. And I can boast of my greatness. I can use it for money. I can weaponize and us it to dominate and control other humans. Goat: Why you need all that? Human: If I don't do it, others will do it and they will dominate me and take away all my stuff. It is not fair. Goat: So it looks like who-owns-what issue. Did you try not owning stuff? Nature: Shut up goat. I'm trying to do a big reset here.
• manx 2y ago
  To align AI with humans, it might make sense to align humans first.