8 ms·
Pacing the frontier
- deleted 2mo ago[deleted]
- dominotw 2mo ago> > We request that the U.S. government support an international effort to develop the technical and governance tools needed to deliberately pace the frontier of automated AI development. > 1122 employees of frontier AI companies 'lets us be the gatekeepers of AI because we know better than you' access to gpu doesnt equal to any special insights to predict future of ai. Everyone and their grandma with enough compute is putting out a model now.
- fuddle 2mo ago> We request that the U.S. government support an international effort to develop the technical and governance tools needed to deliberately pace the frontier of automated AI development. This is quite vague to me, what exactly do you mean?
- zzleeper 2mo agoIt means only those vetted will be allowed to use frontier models (i.e. let's pace ourselves and not share the frontier broadly)
- mofeien 2mo agoIt means that they request the structures needed to navigate the event horizon of recursive self improvement be put into place. Probably: International agreements, verifiability of an AI pause on the level of graphic cards as well as through physical inspections of datacenters. Like it's done with nuclear energy.
- bluegatty 2mo ago'Regulatory Capture' if we were cynical. I think a lot of these guys mean it though. 'The Fear' - is not some magic intelligence will come alive and be Skynet, these people are ridiculous. People will use the compute power to do nasty things, we'll be overwhelmed with bots, regular companies will use them for dark patterns, political actors will use them to cheat. It will be 100% humans using it for bad stuff, and in not very novel ways.
- cindyllm 2mo ago[dead]
- padolsey 2mo agoIt means they haven't really thought it through and don't have a consolidated enough opinion to form a specific Ask.
- serious_angel 2mo agoThe website does not list all the comments but just around ~50 without any way to load all the ~+1100 more (1132, as of 2026-07-28 2100 UTC). The number of em-dashes is questionable, existing in many, if not all, the comments. > ...AI may end up progressing far more quickly than societies and states can adapt... ~ Stephanie Chan, Google I am sorry, but I am not sure whether it's adequate and fair to allow join such statements/petitions to only those who live in it. Isn't it like allow deciding the future for weaponry laws to weapon and war enthusiasts only? And, this is not serious at this point: > ... ~ Source: Anonymous, Meta > ... ~ Source: Anonymous, Google
- joshuaS98 2mo ago[flagged]
- tcdent 2mo agoSomething people don't seem to see, and not that it's placed directly in front of you, is that pacing the release and adoption of AI is already happening. Cost-per-token determines the economic viability of tasks that AI gets applied to. How many tokens would it take to automate all of the responsibilities of a gas station clerk? Probably more than their hourly wage. Job safe. A slightly larger stretch, that most people attribute to profit margins (labs do not have profit margins, they have unlimited invested capital and no expectation of short-term returns), is the flat-rate billing plans of most code products. Why doesn't Anthropic have a 50x or 100x Max plan? Not that it's impossible, that it releases tokens into the workforce at a greater rate (note that their are hour, day, week limits across most of these services as well). Hypothetical, but reasoned, and I do not expect most agree with me. If compute as a variable is inferred, this supports the incorporation a deliberate (redundant) one as well.
- alach11 2mo agoI agree with the substance of your comment that token economics pace the diffusion of AI into the labor market. But I think they have a much smaller influence over the pace of the frontier, right?
- tcdent 2mo agoFair point! I guess I read "pacing the frontier" as "pacing the release of the frontier" surely no lab's internal research team is just going to stop innovating?
- adamzenith 2mo agoNot under all the competitive pressure they are facing, but if an industry wide slowdown could be achieved they might spend more time on model alignment and less on pure capabilities.
- OutOfHere 2mo agoPacing is a really bad idea because China will not slow it down for you.
- joshuaS98 2mo agoThat's the point. Do you see any concerns by these companies? They just want to slow down pace NOW because they're losing the race
- ajyoon 2mo agoThe entire point of the statement is to start international coordination.
- intorio 2mo agoThe problem is how do you enforce coordination? This isn't like nuclear testing that can be detected around the world on seismographs or from satellites in space. AI research happens in benign looking buildings that could be doing a hundred different non-AI workloads, much of it things that industry or governments would not want 3rd parties prying into. There is too much to potentially gain to trust coordination.
- ajyoon 2mo ago
- areoform 2mo agoThe first computers were used for war. The Harvard Mark I was used by von Neumann to calculate the physics of implosion. And then right afterwards, ENIAC was used to calculate the feasibility of the H-bomb. They called it the "Los Alamos problem." The first computer network, SAGE (or, Semi-Automatic Ground Environment) brought together RADAR data from across the US with air-traffic control data and military flight data to detect overflight by the USSR. It could direct and control fighters to intercept those planes. If you were sitting there, in 1955, would you have believed that computers and their networks would bring about an extraordinary age? Imagine if von Neumann and his ilk started campaigning in 1955 to limit the development and use of computers, because surely they can only be used to make weapons, right? Right? We were lucky to have visionary people like Joe Licklider who could look past the present and see the future. In 1963, he wrote a memo calling for an "Intergalactic Computer Network" (because he knew engineers under-delivered. So if he asked for a galaxy, at least he'd get a planet). And he gave these machines to scientists and people like the original hackers at MIT to play with. What if Licklider had written memos about how we had to "pace" computers to make sure that other countries wouldn't be able to develop implosion technologies or systems like SAGE? What then? Do you think you would have the jobs and the prosperity you do today? If instead of giving away this technology, spreading it, and putting down the seeds for ARPA-net, Licklider had written memos from "concerned scientists" (and I'm sure he could have found a bunch) talking about how computers can only be used for weapons and war! So stop the proliferation of computers! We can't imagine a world with computers small enough to fit inside everything! Even wall sockets! What if your wall socket could do war?! They didn't, because they knew there's no better way to kill a better future for all mankind than to control it. † yes, peace. We are more peaceful than we have ever been. Humans can be a violent lot even when reluctant to kill.
- GhastBartender 2mo ago> It is hard to predict exactly how much this will accelerate AI progress, but there is a real risk that capability development rapidly accelerates beyond our ability to understand or control the resulting systems. This is just the standard AI "doomer" talking point. The dangers of AI just used as a compliment sandwich. "AI is so cool and awesome and that's why we need to slow down." The real risks are much more boring and much less considered. For just one example: AI is set to completely upset the Software Development industry. There are 1 to 3 million software developers in the US, depending on how you draw the line exactly. A low-few hundred thousand of those are visa workers. With AI set to significantly cut into tech employment, what's going to happen to those visa workers? ... The Trump admin is not particularly subtle, even openly talking about retroactively revoking birthright citizenship. AI & Tech companies love to talk about how their AI is going to hack everything if they're not careful. But not a peep about these ethics concerns. Even if you [the reader] consider deporting visa workers the ethical move if the amount of jobs shrink, that's not something to be done unprepared or out of nowhere.
- sleepingreset 2mo ago[flagged]
- fidotron 2mo agoAnyone serious about this should participate in pacing the frontier by quitting their job so as to not potentially contribute to the problematic progress. Be the change you want to see! Curious how that isn't what happens.
- mofeien 2mo agoThat's an application of the "Tragedy of the commons": One person stopping to push the frontier just means that this person will lose their job, the frontier will be pushed by the others. The solution is to push for a binding rule instead that will make everyone slow down in the racing towards artificial superintelligence.
- fidotron 2mo agoAnd this is how we absolve people of all personal responsibility. Such companies would not pay people that well if they were so fungible. There is no such possible binding rule either, a point that is so obvious that any smart person claiming otherwise is clearly up to something.
- alach11 2mo agoWe make binding rules all the time. We didn't fix the ozone layer by encouraging people to quit their jobs producing CFCs. We lobbied for international regulation to ban them.
- fidotron 2mo agoIn your example there is nothing to be gained by not complying. Failing to comply with what is in reality arms control is a widespread occurence. It's notable the only historical way to handle this (Schelling covered it at length) is tolerance of mutual espionage on a level that would simply not be tolerable to any of the groups involved. I should add one of Schelling's other pet subjects was "time to weapon being viable". i.e. the distinction between a nuclear power and non-nuclear power isn't quite right, as there's a huge class of countries that could become so fairly fast if they were so inclined, and that this situation is somewhat inevitable. It is this that leads to efforts to denuclearize, even on a civilian level, "problematic" countries. You want to stop frontier AI and you'll have to be rolling back a lot of existing deployed technology.
- deleted 2mo ago[deleted]
- alach11 2mo agoThe HN discourse on this subject is incredibly discouraging to me. It's completely nonconstructive. I take the pace of AI development seriously. All of my friends who work at AI Labs take it seriously. It's very easy to make lazy, shallow dismissals that leaders like Dario/Demis/Sam are acting purely out of self-interest. If AI does pose biological risk, or could destabilize geopolitics, or even just poses economic risks, surely a joint petition like this is a good thing? We should want the AI Labs to make binding agreements to proceed safely and slower, rather than being stuck in a suboptimal game theoretical race to advance unsafely. We should be hoping that nations can make binding agreements. Our discourse here should be about the substance of what these agreements could be and the practicality of enforcement mechanisms.
- red_green_yell 2mo agoAre they lazy dismissals? Or a simple observation that LLM scaling is hitting diminishing returns way way short of AGI. If the claim is (1) AGI coming and (2) we should so something about AGI, wouldn't it be smart to confirm (1) before worrying about (2)? Especially if certain companies have trillions of dollars of valuation that depend on society as a whole believing (1)?
- bpascal 2mo agoi think part of the problem, though, is that we've already hit many of the warning signs that AGI is coming, and people have kept moving the goalposts. if you were to tell someone from ten years that there's an ai that has passed the turing test and has made novel mathematical discoveries, i think many would say AGI is already here.
- versteegen 2mo agoThe old observation that people tend to define AI as whatever computers can't do yet is as true as ever. It's getting a bit absurd, moving from demanding "general intelligence" to replicating human cognitive phenomenology (the experience of cognitive activities). Yet LLMs can already somewhat (confabulation-prone) introspect their own "internal unspoken thoughts" in their residual streams, quite fascinating.
- sergiotapia 2mo agoI'm chinapilled now. Anthropic is an evil, enemy of humanity company that cannot be allowed to succeed.
- OutOfHere 2mo agoYes. Which Chinese labs and models do you like the most?
- brap 2mo agoHas there ever been a case in human history when technology just stopped advancing, simultaneously world wide, only because a bunch of people wanted it to? I think these people are either incredibly naive, or, more likely, not acting in good faith.
- MeetingsBrowser 2mo agoDNA splicing, human cloning, super-sonic passenger flight, human gene editing (CRISPR), nuclear energy, biological weapons, and probably more.
- reasonableklout 2mo agohttps://en.wikipedia.org/wiki/Asilomar_Conference_on_Recombinant_DNA https://en.wikipedia.org/wiki/Asilomar_Conference_on_Recombi... > The effects of these guidelines are still being felt through the biotechnology industry and the participation of the general public in scientific discourse. Due to potential safety hazards, scientists worldwide had halted experiments using recombinant DNA technology, which entailed combining DNAs from different organisms. After the establishment of the guidelines during the conference, scientists continued with their research, which increased fundamental knowledge about biology and the public's interest in biomedical research.
- ece 2mo agoWhat these people want is same thing people in every other field want too, which is being able to use AI as a tool and nothing more. Off course, some of the people that signed this might not recognize this, and the people who didn't sign it are actively working against such an equilibrium.
- dzink 2mo agoThe historic analog most befitting AI are weapons. It provides an advantage to companies and governments in creating weapons, in surveilling people and thus gaining leverage over individuals, in controlling what individuals can do or cannot stop doing. It thus follows the laws of weapons control - the more restrictions are threatened to be imposed, the more even the smallest actors will fight to get more of it. They may not understand or know what they are getting, but they will want more of it because their potential rivals have it. Governments are the ultimate customer here - they have the money, the energy, and the appetite to consume AI infinitely in battlement against opponents - foreign and domestic. The only approximate solution is to prevent any restrictions on the use of it - a-la-2nd amendment (the upside of that is it advances the AI for future use and is thus in the interest of governments as well). Modern large models seem to be compressible and thus we may see more and more tiny sub-specialized varieties. But the push to slow the frontier in weaponry always leads to the reverse.
- paytonjjones 2mo ago> only approximate solution is to prevent any restrictions on the use of it - a-la-2nd amendment Not even in 'Murica do we have anything close to no restrictions. You can't buy your own EMP or nuke, and even warring governments have been surprisingly reserved in building (at least to public knowledge) biological weapons.
- dzink 2mo agoCompletely agree, BUT you’re not dealing with -self-preserving humans anymore. In this case what are the odds of some less competent large state actor getting a model that goes rogue and has to be mitigated against? How do you mitigate against novel weapons it creates if your models are lobotomized in that category? No single government will neuter their own models and very few if any are competent and interested enough to mitigate against them.
- throwaway13337 2mo agoI think we see things similarly about the negative potential but please be careful with the framing. AI is a printing press or a steam engine. It's not a weapon. The framing is important. Much like 'distillation attack'. These framings are an attack on the narrative. Printing presses at the time were seen as extremely dangerous too. For many of the same reasons cited about AI. In both cases, the technologies were used to destablize existing power structures. And, after some hardship, make a better world for most people. Even calling these things 'dual use' is a weird framing. Books are not 'dual use technologies' unless you want to sell fear.
- vessenes 2mo agoThis. Will. Never. Work. This research game is a zero-trust iterated game. Any defection carries the benefit of possibly being permanently valuable. There will always be defections. OpenAI defected WHILE THEY WERE pushing to slow. The cost to the US of deciding to slow is so massively harmful that it is wrong to even put this idea out in the public zeitgeist; it may somehow become a thing that US legislators believe they can accomplish. They CANNOT accomplish this, because the fundamental economics and possible outcomes make it mandatory for any strategic adversary to work as hard as possible to defect. The proposals here imagine a global panopticon surveillance state overseeing all research and chips as a 'reasonable' solution. EVEN IN THAT (terrible) state for the world, we will have effective defection. Like it or not, the world is on this train, and the only sane thing to do is to fully support high quality ethical research groups and firms as they get up the learning curve of this technology. Variants of this idea in AI 2030 included burning some of the US lead to get some nebulous safety metrics improved - also insane. Any slowing allows strategic adversaries time to catch up with very different alignment goals. Do I like this situation? No. Are the people signing these things just fully in denial about the game theory and the stakes and the behaviors of strategic adversaries? Yes, yes they are.
- red_green_yell 2mo agoThe only thing it might accomplish is allowing OpenAI / Anthropic to squeak out an IPO before the market realizes AI will become a commodity and the massive profits these companies are promising will never materialize. They aren't in denial about this open letter's futility in terms of actually reducing the pace of AI, that's just not their goal in the first place.
- vessenes 2mo agoI don’t think it’s clear we’ll get commoditized AI out of this research loop. The only real datapoints we have are that Chinese open weights models seem able to stay a certain amount of distance ‘behind’ released frontier models. It’s clear that the models are trained on exfiltrated data from the frontier models => we cannot currently disambiguate how easy it is to get close to the frontier in a vacuum. I also don’t believe we have seen anything like 1% of inference demand fulfilled globally. You might believe differently. But, if I’m right, we are nowhere near some massive collapse of these companies, quite the opposite. For the opposite take to be correct, you’d need to believe that the ‘value’ of tokens will not increase significantly and that the current demand for tokens is at or near a peak. I also don’t believe these open letters are signed by people that are all master strategists and just economically motivated. I believe instead that religion and money combined create strange behaviors and beliefs - in this case “what’s good for us could be implemented” — combined with the already endemic “what’s good for us is good for humanity.” Together we get really bad outcomes.
- PowerElectronix 2mo agoThey just want some time to be able to sell the stock
- biffles 2mo agoIt's an admirable idea, but I don't see how you can slow this down in practice. We certainly know China will not, and they are already at the frontier. RSI is too important and too tangible at this point for everyone to slow down. It's the equivalent of telling the world to stop developing nuclear weapons after Hiroshima. Just not going to happen, and in the meantime dramatically increases the risk that the power of AGI lands in the hands of the few (read: the political class) rather than empowers the people.
- frabcus 2mo agoYou need AI capabilities to the point where the US and Chinese leaders get scared enough that they do a deal. There are detailed specific scenarios in Plan A https://ai-2040.com/ https://ai-2040.com/
- frabcus 2mo agoAlso, in reality above a certain power level the Governments will stop giving us access anyway. This already began with Mythos.
- yks 2mo ago> We request that the U.S. government ... Too bad these researcher's organizations spent their efforts supporting the dumbest, ficklest possible version of the U.S. government to guide us through the gravity of this moment.
- girfan 2mo agoCan someone help clarify what it means, concretely, to make binding agreements to proceed "safely and slower" in the context of AI research (whether or not at frontier labs)? Are there specific policies and actionable guides being proposed?
- frabcus 2mo agoThere's a decent amount of stuff in Plan A. Especially if you read the supplements. https://ai-2040.com/ https://ai-2040.com/
- fwlr 2mo agoMaybe I am too cynical, but this just sounds like a plea for the government to step in and save them from their investors.
- overfeed 2mo ago"Oh no! The AI we developed for years, and whose stratospheric valuation is predicated on replacing human jobs, is coming for our jobs, the government should do something!" Signed, AI researchers and engineers.
- mosura 2mo agoIf the “recursive self improvement” is taken literally (and at some point it will) then that is going to be what happens. The fun part is that probably happens quite a way before getting anywhere near to AGI, and it is almost plausible that for many of them that is where they are.
- 0xDEAFBEAD 2mo agoInvestors want to see a return on their investment ASAP. The Economist recently published an article: "AI revenues are growing fast, but not fast enough -- The returns on trillions of dollars of spending are deeply uncertain" https://www.economist.com/finance-and-economics/2026/07/28/ai-revenues-are-growing-fast-but-not-fast-enough https://www.economist.com/finance-and-economics/2026/07/28/a... As OP states: "each company—and country—is under intense competitive pressure not to unilaterally slow that acceleration." Let's not overwrite human civilization for the sake of the shareholders.
- melodyogonna 2mo agoThese AI companies needs to stop with all the doom and gloom.
- himata4113 2mo agoWe know that at the end of the day AI intelligence is closely tied to invested compute. The fact that automated AI research might spark some kind of exponencial increase in capabilities seems extremely unlikely to me. Of course we know that some got there faster and cheaper than others, but the fact that it requires immense computing power has not changed. We also have already witnessed a slowdown, the raw intelligence of models has been relatively stable since january, but their use of tools and what we call agent capabilities have allowed us to tackle significalty harder problems with what we have already. This is becoming blatently obvious how smaller models such as gpt 5.6 luna are capable of doing similar things to sol while presumeably being fraction of the side and compute investment. This is also the likely reason why chinese models are able to catch up as intelligence strictly requires more compute, agentic capabilities are more of a mixed bag of RLHF and general post training. I believe automated AI development will simply lead to cheaper intelligence, but it will not necessarily produce something more intelligent. There is a sentance from a book that I don't remember where I read it from, but has stuck with me which boils down to this: "We are very unlikely to make something smarter than a human brain.". If you apply this concept more broadly an AI system will not be able to produce something more capable than itself, it might be specialize in different areas via trial and error (i.e. solution discovery), but it will not have better intuition than that which was trained from the fruits of human labor. I also strongly believe that model performance is ultimately governed by the advancements brought into the world by humans. AI of course can find improvements by simply trying a bunch of random things, emphasis on the word FIND. Until we can prove that AI systems are able to come up with unique solutions that have yet to exist even with simplier problems such as creating any kind of new AI architecture that has not been already documented, conceptiualized or explored by humans already -- and it doesn't need to be better than what we have already, just something new -- I believe it's too early to sound alarms until proven otherwise. -- Something I've realized as I was fact checking myself: We could presumeably have self-improving intelligence if we used enough properly engineered agentic workers working towards more intelligent AI, but at that point we are back to the same problem where we're spending mass amounts of compute to increase the raw intelligence of a model.
- chrisjj 2mo ago> We know that at the end of the day AI intelligence Stopped reading at "intelligence". Try capability. (unless your target audience is those already gulled.)
- throwaway13337 2mo agoImplied in this is a concentration of power required to enforce. No one is sold on a dystopian hellscape of fear, surveillance, and control. Each one of these would be required for a select group to effectively enforce AI safety. Inflicting fear on society has never lead anywhere good. Please stop. New technologies have always been scary. But humans have adapted. I believe in humanity. Do you?
- chrisjj 2mo ago> New technologies have always been scary. But humans have adapted. The Silurian Hypothesis would like a word.
- soundworlds 2mo agoThere seem to be people working at OpenAI and Anthropic who are surprised to hear that many people don't trust them. If this is you, I would recommend talking to people outside of the AI industry - we've seen OpenAI and Anthropic continuously lying, abusing lobbying, training on pirated IP, trying to scare politicians into blocking actually open AI from abroad, etc etc Why the hell would we trust these companies to be impartial regulators now?
- chrisjj 2mo ago> I would recommend talking to people outside of the AI industry They already are. Loud and clear. What they aren't doing is listening to people outside of the AI industry. And that's because they don't want to hear.
- ndom91 2mo ago"AI is progressing at a rate that our society might not be ready for. Frontier labs are very close to AI that can exceed even the best people on almost every metric of intelligence. This will lead to unprecedented social and safety risks. To ensure a positive future we need to develop AI in a way that is driven by responsibility and thoughtfulness." Responsibility and thoughtfulness? In this economy?
- chrisjj 2mo agoAnd Xi rejoices!