8 ms·

Policy on the AI Exponential

169 points by yjp20 4mo ago

deleted 4mo ago
[deleted]
slopinthebag 4mo ago
How predictable. The company currently on top wishes to use the regulatory power of the state to prevent competitors from encroaching on their market dominance. It’s a tail as old as time, although their CEO’s rarely publish blog posts about it.
- thewebguyd 4mo ago
  Yep > "Models above a threshold of compute should undergo mandatory testing by a qualified third party for their level of risk in four specific areas: cybersecurity, biological weapons, loss of control of AI systems, and automated R&D that could accelerate these other risks." AKA: Make it as expensive and untenable as possible for any open source model to jump through the regulation red tape so we can pull the ladder up behind us. Disgusting. All the marketing talk about "this model is so dangerous" "omg we can't release this to the public its so dangerous" etc. is just priming for the incoming lobbying for protectionism from foreign competition, and regulation preventing the development of any other model that could threaten their dominance in the name of "safety"
  - slopinthebag 4mo ago
    Yeah, they know their moat is evaporating as the Chinese models continue to catch up, and at a much cheaper price. Also, why has my comment been flagged? It is sitting at positive votes but has suddenly been flagged for no apparent reason?
    - dang 4mo ago
      If you mean https://news.ycombinator.com/item?id=48481131 https://news.ycombinator.com/item?id=48481131, users flagged it. We can only guess why users flag things. Perhaps it was because they thought the comment broke the site guidelines by being snarky and/or fulminatey? I don't think it was an extreme case of that, so I've turned off the flags now.
      - ofjcihen 4mo ago
        I mean it’s pretty tame and actually adds to the conversation. It feels somewhat obvious that someone just disagreed and flagged it. Is there a system for holding people accountable who abuse flags?
        dang 4mo ago
        Yup there is. I've restored a few other comments that were similarly flagged as well.
      - cadamsdotcom 4mo ago
        Thanks for staying on top of these things dang - such an important topic so it’s great to see the discussion rage but stay constructive.
      - slopinthebag 4mo ago
        Thanks big dog
deleted 4mo ago
[deleted]
themafia 4mo ago
> AI models have gone from barely being able to write a coherent line of code to writing most of the code at major AI companies. Gasoline has gone from barely being able to power stationary farm machines to now being the fuel that underpins our entire economy. So, great news all around, right? > which predict an exponential increase And was that actually delivered? Real question: If a model goes from 80% accurate to 85% accurate is that an exponential increase in "cognitive capabilities?" Are we considering training costs and effort?
- interestpiqued 4mo ago
  Yeah I’m lost by the constant use of “exponential”. What is the x and y axis when the talking heads say this?
  - TobyTheCamel 4mo ago
    I think people are typically referring to the task-completion time horizon at a fixed success rate [1]. That has had pretty robust exponential scaling for many years now. [1] https://metr.org/time-horizons/ https://metr.org/time-horizons/
deleted 4mo ago
[deleted]
pdhborges 4mo ago
What will be Amodei's job after we have AIs that are better at evrything than humans? Is the AI going to care about our stock exchange playgrounds that reward the future Antropic stock holders?
- pixl97 4mo ago
  Well obviously he'll have the AI 'dispose' of the poor and live a life as a king with a select few farmed humans and have the world as a play thing. Really the entire future of AI at this point seems like "Don't worry about it, we'll figure out when we get there". Works a lot better if you're extremely rich and can afford your own private security.
  - nemomarx 4mo ago
    I never understood why people setting up bunkers expect the security to still be loyal after whatever happens.
    - rootusrootus 4mo ago
      Even loyal security would hardly be sufficient. If a million people have decided they want your head on a pike, even a billionaire cannot afford a big enough army. And what exactly is a billionaire in that world anyway?
    - stackbutterflow 4mo ago
      Can't find the article mentioning it but apparently it's an open problem they're thinking about. But yeah if society collapses these billionaire nerds are the first to go. Quietly, in their bunkers, while the team leader of their seal mercenary team takes over. Even before the rest of us realizes what's happening.
      - stackbutterflow 4mo ago
        Found it https://www.theguardian.com/news/2022/sep/04/super-rich-prepper-bunkers-apocalypse-survival-richest-rushkoff https://www.theguardian.com/news/2022/sep/04/super-rich-prep... > Finally, the CEO of a brokerage house explained that he had nearly completed building his own underground bunker system, and asked: “How do I maintain authority over my security force after the event?” The event. That was their euphemism for the environmental collapse, social unrest, nuclear explosion, solar storm, unstoppable virus, or malicious computer hack that takes everything down. This single question occupied us for the rest of the hour. They knew armed guards would be required to protect their compounds from raiders as well as angry mobs. One had already secured a dozen Navy Seals to make their way to his compound if he gave them the right cue. But how would he pay the guards once even his crypto was worthless? What would stop the guards from eventually choosing their own leader? The billionaires considered using special combination locks on the food supply that only they knew. Or making guards wear disciplinary collars of some kind in return for their survival. Or maybe building robots to serve as guards and workers – if that technology could be developed “in time”. I tried to reason with them. I made pro-social arguments for partnership and solidarity as the best approaches to our collective, long-term challenges. The way to get your guards to exhibit loyalty in the future was to treat them like friends right now, I explained. Don’t just invest in ammo and electric fences, invest in people and relationships. They rolled their eyes at what must have sounded to them like hippy philosophy.
tripleee 4mo ago
My god this guy is insufferable. Stop mis-using the term exponential
- TobyTheCamel 4mo ago
  Looks pretty exponential to me [1]. From a fully independent, non-profit research group. [1] https://metr.org/time-horizons/ https://metr.org/time-horizons/
  - interestpiqued 4mo ago
    Release date seems like a terrible x axis with how much more compute they are using. Not to mention while I like what METR is trying to measure, it is an uber specific metric. And frankly, me just complaining, they’re prompts I feel do most of the work for the AI. I’ve never gotten as detailed instructions as they give the AI for the task
    - HDBaseT 4mo ago
      Whilst true, if you had unlimited compute 5 years ago, we wouldn't be anywhere near Mythos level purely because the technology behind the models wasn't refined enough.
- baq 4mo ago
  Suggest a better name for what’s happening with LLMs please.
  - dontlikeyoueith 4mo ago
    Snake oil scam. They simply don't do what the label on the box says they do.
    - teliosix 4mo ago
      It is really hard to believe you actually believe this unless there really is this class of people that are so addicted to social media that they have confused performativity with actual thinking.
ofjcihen 4mo ago
Can we not open up every article talking to working professionals as if they’re children? I like to stay up to date on things but more and more I’m finding myself pointing codex at a URL and saying “get to the point”.
- bendergarcia 4mo ago
  He wrote it with ai that’s why
SkitterKherpi 4mo ago
It is impressive how well they've scheduled all their releases, posts, and other news to dominate the tech news cycle almost every day in this pre-IPO phase.
Imnimo 4mo ago
>The government should have the power to block or deter deployment of the model if it is determined, in light of third-party assessment, to present unacceptable risks. This power must be scoped to the above four specific risks and there must be protective measures against political favoritism or arbitrary decisions. I feel significantly less sympathy for Anthropic's Supply Chain Risk designation if they believe the government should have this power over them. You get what you sign up for.
hmmmmmmmmmmmmmm 4mo ago
Sit back and watch the stock market implode as these lot double down on safety theatre. The hype is coming to an end.
- aspenmartin 4mo ago
  > The hype is coming to an end I'm sure you have evidence for this
- rootusrootus 4mo ago
  They have to keep it up at least until the IPO dust settles.
SilverElfin 4mo ago
I have to be honest, I am tired of reading these arrogant, self-absorbed posts from Dario and Anthropic in general. Opening with this lord of the rings reference just feels like they are trying too hard and are untrustworthy. As annoying as their tone is, the real big danger is what they are setting up for. All this fear-mongering around Mythos, the overly aggressive controls on Fable, and these manifestos they keep writing, are part of setting up for REGULATORY CAPTURE. Even collaborating with the Pope and the Interfaith Alliance (https://iafsc.org/our-work/faith-ai-covenant https://iafsc.org/our-work/faith-ai-covenant) are part of creating a vast support network for regulations and restrictions. Those regulations will help those faith organizations or the government or whatever, but will also help Anthropic’s bottom line. Those regulations will not support your civil liberties. They will restrict speech, access to AI, and allowed uses of AI. They will lead to bans on use of models from some countries like China, and also bans on open-source or open-weight models. If Dario wants to be trusted, he needs to explicitly say in writing that Anthropic will not support any legal or regulatory restrictions on open-source AI, open-weight AI, or Chinese models. Otherwise, what he is really saying - even as he claims he is trying to ‘defend democracy’ - is that he and Anthropic do not truly support fundamental rights like our right to speech. It’s not just Anthropic either. OpenAI had their own recent polemic, pushing for regulations like mandatory safety reviews by agencies for “frontier” models (https://news.ycombinator.com/item?id=48387246 https://news.ycombinator.com/item?id=48387246). It’s a dead giveaway that these companies have no moats, are in serious danger of being a commodity, and are now in the process of using regulations and enshittification to hold onto money and power.
- nemomarx 4mo ago
  I mean he wants to be trusted but he also absolutely doesn't support freedom of speech in that sense, and I doubt anyone with power or influence over ai policy does? If you find a good lobbying group with money who can push for it let HN know.
  - mananaysiempre 4mo ago
    To be honest, at this point I’d take any group at all willing to usefully and non-hypocritically push for free speech (as a general societal value, not just a US-specific legal notion).
fnoef 4mo ago
[flagged]
- verandaguy 4mo ago
  LOTR is easy to love. It's a classic story arc, well told, and well adapted for mass consumption. It's also a very thinly veiled metaphor for the ceaseless pursuit of industrialization in every aspect of life chipping away at the humanity and metaphorical magic of the human experience, and the perils of embarking on that endeavour to the exclusion of everything else. Personally, I find some humour in this situation.
  - anematode 4mo ago
    Maybe Dario should have cited The Lorax.
kouteiheika 4mo ago
> AI companies that develop advanced AI models must have strong security standards that protect their model weights So, basically, make open-weight models illegal. It's nice for Dario to come out and say this so explicitly.
- raincole 4mo ago
  People ask what the 'open' part of 'OpenAI' even means as they're a for-profit company. It turns out it means 'opener than Anthropic.'
  - inigyou 4mo ago
    Opener Anthropic Imitator
- simplyluke 4mo ago
  Dario's been beating the regulatory capture drum for several years at various intervals, always in the name of safety, but it's hard to not see how self-serving it is. I'm personally very tired of reading the linear-algebra-median of every AI safety essay from lesswrong with the inserted opinion of "therefore all my competitors, especially those pesky open source ones from scary countries should be illegal, only I can be trusted to not abuse the computer god that I definitely will have in just a couple more releases and with a couple more trillion invested"
- ed_balls 4mo ago
  "We Have No Moat, And Neither Does OpenAI"
iamniels 4mo ago
I like that he comes up with new laws and regulations for AI companies. Can I suggest some more? - You shall not embed copyrighted material in your models. - You shall not bombard every little website in existence with 1 million scraping queries per day. - You shall not use your political influence to pump and dump your AI (or rocket?) company. - You shall not imperill the whole IT sector by buying all CPU and memory chips. These new rules will affect every society directly in a positive way. Thanks.
- bashtoni 4mo ago
  You missed a couple ofvery important ones: - Your AI data centres will run only on renewable energy - Your AI data centres will not use evaporative cooling
- deleted 4mo ago
  [deleted]
- tptacek 4mo ago
  It is normal, expected, and healthy for stakeholders in a regulatory environment to offer proposals about regulations. What's unhealthy is the proposition that the deliberation process is so fragile that a stakeholder needs to cover every angle, lest they corrupt the outcome.
  - ofjcihen 4mo ago
    What you say is true but completely ignores the obvious ways in which what he is proposing benefits his company. Its like saying it’s normal for a taxi driver to drive people places while he’s got you handcuffed in the trunk.
    - movealongnow 4mo ago
      [flagged]
  - pseudalopex 4mo ago
    It is normal, expected, and healthy to offer criticism of self interested proposals. And mock even. What is unhealthy is to imply someone said what they did not.
snaking0776 4mo ago
I know this is likely just for IPO hype but when I read things like this I sometimes wonder if I must be missing something. I use agents everyday and find them really useful and they save me a lot of headache. At the same time I find that if I let it self-direct at a high level at all it generally makes bad choices that cause me headaches later so I can’t really give them autonomy. Enough people seem to believe this exponential line of thinking though that I keep having to wonder: am I the one missing something here? Is there some magic tool that I haven’t found yet that will cure cancer?
- ofjcihen 4mo ago
  N+1. This is my experience and for the most part the people that I work with share the same feeling. A highly enthusiastic concussion enthusiast with 10 hands is how one person put it. These are people in different fields but highly accomplished so I’m feeling comfortable sharing their assessment.
  - TobyTheCamel 4mo ago
    You're talking quite statically though. I don't think anyone is worried about today's models being a serious threat, but next year's, three years' time? Just three years' ago these models were useful bumbling fools and it's hard to judge where on the S-curve we currently are. I'd rather be thinking about these issues in advance rather than waiting until the problem becomes real.
    - ofjcihen 4mo ago
      > I don't think anyone is worried about today's models being a serious threat Fable is essentially bricked for my areas of interest (even being a member of the cybersecurity program). It seems like they’re attempting to sell regulatory capture under the guise of safety. That’s more of the point.
    - dontlikeyoueith 4mo ago
      The models are still useful bumbling fools. We're in the flat part of the curve because we've exhausted existing data sources.
advael 4mo ago
It's crazy how all these tech CEOs develop the same sense of ethics that seeks to make the foundation of open research and development that made their efforts possible and may threaten their market position illegal in the name of safety against nebulously-defined risks
d_silin 4mo ago
The only effective action is push back on Athropic.
TobyTheCamel 4mo ago
I feel completely baffled by the other responses on this thread. People viewing this purely as a marketing stunt, regulatory capture or attack on their freedoms, with seemingly no appreciation of the real threat that AI could pose to society and even humanity given its current rate of progress. I'm not going to claim that the CEO of pre-IPO company has no incentive to bolster the claims of his tech, but to completely disregard everything he is saying based on that seems awfully binary. I don't know whether people are just high on copium, spouting "it's just fancy autocomplete" or "only humans can really be creative" on every LLM-related thread, but it is impossible to deny that in a span of a few years we've gone from models that could barely put together a sentence, to something maybe not equivalent to a junior developer, but at least resembling it. And sure, you can point out every flaw that current day LLMs have, just how everyone pointed out that Stable Diffusion couldn't generate accurate hands (until it could 6 months later!). But the gradient is pretty clear and I am yet to see a well-argued narrative from anyone why scaling laws should fail in the next year or two (by which point it feels like we're going to have a real problem, extrapolating the current trajectory). I'm very glad this discussion is at least being had, and I wish everyone would get off their high-horse and take things a bit more seriously.
- mitthrowaway2 4mo ago
  I agree that AI poses a threat to society. I act on this by not developing world-leading AI models and offering them to anyone willing to pay top dollar, while funneling that money back into accelerating AI capabilities development. Maybe Dario would consider taking a similar ethical position? Maybe he would support restrictions and taxes on data center construction, in order to slow down the pace?
  - TobyTheCamel 4mo ago
    If Anthropic were not developing these models, one of many other companies would be. I think it's good that the CEO of the current world-leader is at least considering these discussions and platforming possible solutions. The fact that he doesn't support more restrictive approaches that don't align with his incentives doesn't invalidate the points he is making.
anothermathbozo 4mo ago
> A wide range of pro-employment policy incentives can help to slow or reduce job displacement, including: wage insurance policies that compensate people when they have to take a lower-paying job, retention tax incentives to encourage employers not to make layoffs, workforce training grants, or infrastructure to facilitate matching of employers to employees to speed the rate of labor market adaptation. While the particulars of which interventions are best will depend on what kind of labor displacement AI brings, we should readily accept the costs and market inefficiencies that these policies could entail, particularly as they are likely to be offset by AI-driven productivity gains. People get income from one of three places: capital income, labor income, or the welfare state. If this technology truly unlocks a holy panacea of productivity with a commensurate drop in employment then capital’s share of the national income can and should provide for a wider and deeper welfare state. Nothing new need be invented here. Dario’s long and only somewhat organized list of policy interventions makes appropriate preparedness sound like a manic pulling of any and all levers when a simple theory of distribution will suffice.
- jelling 4mo ago
  This and we already did a dry run of ad-hoc distributions with COVID relief. They had to use the data from tax filings but it did work in terms of getting the money out there.
- SpicyLemonZest 4mo ago
  > If this technology truly unlocks a holy panacea of productivity with a commensurate drop in employment then capital’s share of the national income can and should provide for a wider and deeper welfare state. This isn't guaranteed in the tax system as it exists today, because reinvestments into further growth are often treated as expenses which cancel out the income for tax purposes.
  - anothermathbozo 4mo ago
    You’re conflating firm level taxable income with the national income.
patcon 4mo ago
The comments here. They make me feel that we are so doomed. We all want to nuclear codes so badly. We are addicted to intelligence and labour so badly that we simply can't concieve that a pro-social actor might want us all not to have it, and for good reason. I mean... Obviously, insiders like Oppenheimer (who dedicated their lives to considering the implications of the technology under discussion), they just feared nuclear proliferation because they wanted all the profits for themselves, right :(
- polishTar 4mo ago
  Look - it's WAY more fun to just call Dario a goober than engage with the actual substance of the essay. Duh! Sure, this may be the most important invention ever with near certainty to reshape society over the next few years, but meh. We should probably just immediately dismiss the concerns of anyone working on it without addressing their arguments at all. It's easy, we can justify ignoring their warnings by saying they're self-interested or too self-important or whatever. Life is more fun when you live it with your eyes closed! You should try it out too.
  - patcon 4mo ago
    Omg I love you. Thanks
thefounder 4mo ago
This guy can’t stay a day without posting something more or less “ban open source AI”. We keep you safe
tetrisgm 4mo ago
Honest question: is there a reason for the naming conventions for these models? Anything that makes it better than giving them names with model numbers, like “Claude 3” or such?
- kridsdale1 4mo ago
  Yes. Each of the nouns is a “size class” in literature. From small lines poem (haiku, sonnet) to larger story (fable) to very large story (opus) to culture-defining foundational (myth). It’s a fun way to say how many parameters are in the model without revealing a number like 405B or 17B which isn’t really comparable vs other models.
  - tetrisgm 4mo ago
    Interesting. Thanks for enlightening. My non-poetic brain thinks we should call then Mini, nothing, Pro, Max, and then version numbers. Exactly like Apple. It'd be so much easier to parse. Maybe the AI companies like having the affectionate names haha
    - HDBaseT 4mo ago
      I personally hate the Mini, Nothing, Pro and Max nomenclature. I find it very tacky and the confusing having to remember a "base model" exists without any descriptor. Apple does make it worse though, they have products like the iPhone Pro Max, which combines two!
jampekka 4mo ago
"Democracies should seek to form a global coalition centered on building AI according to their common values, iteratively trying to draw in the rest of the world by making it more and more attractive to be part of the coalition and less and less attractive to be outside it." It's not clear to me on which side of the coalition USA is meant to be in this divide. And as an European I'm not sure whether being in China's or USA's coalition is better in the long term. In general, this deliberate mongering of ever more geopolitical division is extremely harmful. As is the Trump bootlicking.
deleted 4mo ago
[deleted]
deleted 4mo ago
[deleted]
swingboy 4mo ago
[flagged]
- dang 4mo ago
  Please don't post unsubstantive comments to Hacker News, regardless of who is a goober or you believe they are.
hmokiguess 4mo ago
https://xcancel.com/cuimao/status/2058458683781365873 https://xcancel.com/cuimao/status/2058458683781365873
kingstnap 4mo ago
Its hard to read the first half of this as anything other than regulatory capture propaganda. It really all ties together as: > AI has become a major commercial technology >Frontier AI models, like airplanes, should be required to go through technical testing and auditing, and their release should be blocked or reversed as a threat to public safety if they do not meet high standards of safety > AI companies that develop advanced AI models must have strong security standards that protect their model weights Anyway Dario's financial interests aside. This is an interesting breakpoint for me. > Second, any response to AI-driven job displacement needs to address both the need to provide for everyone economically, and the need for people to find meaning, purpose, and agency. The latter is ultimately more important To me this reads as an out of touch statement. I think the majority of people on earth work to keep a roof over their heads. Of course work can be a source of meaning, purpose, and agency, but to call it the more important aspect on a societal level is a sort of rich person like Dario statement to make.
- FergusArgyll 4mo ago
  Can you explain how this is attempted regulatory capture? To start a lab now which can actually compete for the frontier (i.e. and pass the "Threshold of compute" needed to get regulated) a lab / company would need a ton of money. Surely a well funded operation of that kind can deal with the regulations.
  - thayne 4mo ago
    For one thing, the financial barrier may not necessarily stay that high forever. For another, such regulations could prevent a competitor from making the weights open for their model to try and disrupt the competition. And finally, Amodei would no doubt want to be involved in designing the tests the AI needs to pass, and could (and likely would) design it in a way that Anthropic models would be able to pass easier than competing models.
  - gck1 4mo ago
    You'd need a ton of money if you followed the unoptimized, cash burning mindset of existing AI labs. There's probably a ton of optimization that is just sitting on the table. Chinese labs have proven it can be done for way less money. Then there's running inference service of open weights, which doesn't necessarily require opening a lab. You can grab Chinese model weights and sell inference. Anthropic wants to make sure nobody can open a new domestic lab, or provide inference services of unauthorized open weight models, or release open weights if model is good. It is regulatory capture - it covers all areas that are dangers to Anthropic's bottom line.
gedy 4mo ago
I think we should treat this with smirking suspicions until their IPO happens.
jdw64 4mo ago
I read this essay, and it feels like lying behind a mask of moral responsibility and safety for humanity. They are asking for FAA style preclearance and third party audits. That literally means no new AI startup can emerge. Do they not know that audits cost money? Protect your own monopoly, protect your customers' regulations. They want strong regulation like the FAA to raise barriers to entry for the foundation models they themselves build, but then why do they want to loosen FDA regulations? While at the same time driving token consumption from their own customers. They talk about permanent job displacement and UBI. I usually call this "a morally packaged safe landing." They are doing something unpopular (destroying jobs) and getting criticized for it. But they do not want to be criticized further, and they want to ask for social sympathy. So they claim a 'noble cause' that everyone can sympathize with and that is safe for themselves AI will generate astronomical productivity gains and capital profits, which AI companies privatize. So why should the social costs be paid by national taxes? In my opinion, something like "We will donate all of our AI companies' revenue for the next 10 years to society" would show genuine sincerity. Then they say, if we do not develop AI, China will eat our lunch, and they go after China. But is not this really about preventing Chinese dumping, maintaining our own token prices, and asking the world to beat down China so that they can preserve global tech hegemony? But by blocking China from the CUDA ecosystem, now the CANN ecosystem has emerged, has it not? If China develops techniques that reliably reduce inference costs, who knows how things will turn out then. Honestly, I like Anthrpic's Claude, but the Anthropic CEO's rhetoric is so stale. It is not that it feels hypocritical. It is that this is just a one dimensional rhetorical tactic that assumes the public is stupid. I do not think open source is unconditionally good. (It is good, but it can become bad in all situations or all countrie). Open source itself is a barrier for countries outside the Anglosphere when they want to release IT products. Because there is no incentive to buy a product that is worse than an open source alternative. So I do not think everything necessarily has to be open source. But this (referring to Anthropic's position) seems to treat people like fools. If regulation is needed, shouldn't they also argue that FDA regulation is needed? I wish they would be consistent
- aesthesia 4mo ago
  > They are asking for FAA style preclearance and third party audits. That literally means no new AI startup can emerge. Do they not know that audits cost money? Training frontier AI models costs money, orders of magnitude more than third-party audits. If you can afford to build the model, you can afford to have it audited.
JustFinishedBSG 4mo ago
It's very hard those days to think of companies/people more arrogant than Anthropic/Dario, which is quite the achievement as the bar is very high. If that arrogance was well placed at least you could somewhat excuse it, but the fact that it is so overtly hypocritical and based on false premises just makes it so much worse.
voxleone 4mo ago
Regarding regulation: I'm deeply invested in computer vision systems and i fear that policymakers [who are not deeply familiar with the technical distinctions between AI systems] may write broad rules that cover "AI" generally. In that case, computer vision companies and industrial users could end up subject to requirements that were largely motivated by concerns about generative AI and LLMs.
jldugger 4mo ago
>AI Exponential How much of the policy prescription changes if the exponential is actually just a series of sigmoids[1]? [1]: https://x.com/ylecun/status/1799064075487572133 https://x.com/ylecun/status/1799064075487572133
- aspenmartin 4mo ago
  Ooof a Yann LeCun quote from '24, a risky time period to be mining his quotes from lol. But I think broadly he's right, but the argument is: we don't see any evidence for the curve flattening, and we should plan for it _not_ to flatten any time soon. "It will eventually slow down" is true but its meaningless if you dont put any sort of time period on this.
mountainb 4mo ago
This reads like an AI with an overflowing context window wrote it; or in the alternative it’s a list of statutes written by an arrogant and delusional king. It is this type of arrogance that will lead to an unfavorable reaction by Congress.
- dontlikeyoueith 4mo ago
  The sooner the better.
maxglute 4mo ago
What about punishment on AI exponential for security failures, electric chairs for CEO of high ranked frontier models that gets jailbroken by geopolitical adversaries.
silexia 4mo ago
AI is a lethal risk to mankind and should be totally rolled back for a century of close research before we try again.
re-framer 4mo ago
I appreciate the critical perspective on political and economic power, as long as it's consequently followed, and every willingness for cooperation and the creation of fair rules is good. What makes me doubt that Dario Amodei has really internalized the problem is the lack of humility, the stance that it's just important that the "good guys" keep the technology away from the "bad guys". If you really want to provide AI with public benefit, you need to prevent power concentration. How? Some unpolished ideas, I'd be happy to hear yours: - Avoid getting too close to an administration that is openly attacking democracy and is not interested in the benefit of humanity or mutually beneficial cooperation. - Don't support surveillance. Non-(US-)Americans have human rights and privacy, too. Prepare for a situation where a government tries to convert your compute infrastructure into surveillance infrastructure. - Support the creation of community data centers. In other words, build data centers together with local communities and make sure they profit from them. - Advocate for laws that require transparency about resource usage and emissions of data centers. - If you don't want an AI race, make sure that other countries don't need to fear the US concentrating too much power. Create institutions that can be trusted by other countries, too. EDIT: I forgot: - If qualified labor will actually turn out to get devalued, we also need a plan to prevent states from turning into rentier states that don't depend on a well-educated society any longer.
- re-framer 4mo ago
  (On the other hand, I have been fooled by too many billionaires claiming to act in the interest of democracy and freedom. I once fell for a billionaire buying a social media platform, claiming to be a free speech fundamentalist, and it didn't age too well.)
AnodicElegy 4mo ago
"AI is advancing at a lightning pace—in only four years, AI models have gone from barely being able to write a coherent line of code to writing most of the code at major AI companies. Similar gains have been made in biology, physics, math, finance, law, translation, and many other fields." This is a massive exaggeration. The advancement in the automation of computer code writing has been impressive and is obviously, at least in the short term, changing the software engineering industry substantially. Most other fields have not been affected to nearly the same degree. Certainly not biology, physics, finance, and law (I don't know enough about the math and translation fields to speak to those). --- "3. Accelerating AI’s positive impact..." This whole section is the type of thing that often comes out of the mouths of Silicon Valley tech executives without a pharma background. It indicates a thorough lack of understanding of the realities of pharmaceutical research. What he is describing here is removing many of the solid, evidentiary rules that are in place to make sure that the drugs reaching the market actually work and replacing them with proxy predictions. Look, my least favourite part of the job is the animal testing, and I would be hugely grateful if that could be eliminated from the drug discovery pipeline. People have been trying to do that for a long time. But it's extremely difficult. Biology is very, very complicated. Our understanding of how processes in organisms work are vague and approximative. This is not computer code. Even if Anthropic somehow got all of Big Pharma to hand them their proprietary data, it would only scratch the surface of the understanding that is needed to solve these kind of problems. Due to these realities, the program Amodei is describing here would, effectively, open a floodgate of drugs on the market that don't actually do what they are supposed to and are more likely to have unidentified toxicity.
- interestpiqued 4mo ago
  Also I’m not denying it would have a major impact either way. But c suite have basically demanded people use ai to code. It’s not all organic
BrenBarn 4mo ago
This was slightly more thoughtful than I feared it would be. However, what I still do not see in any posts from any of these AI pushers is any genuine consideration of the possibility that the best thing might be for them to do less of what they are currently doing. Not to do it differently. Not to mitigate it. Not to do something else in addition. But to actually reduce their current activities.
- deleted 4mo ago
  [deleted]
observationist 4mo ago
>> Of all tyrannies, a tyranny sincerely exercised for the good of its victims may be the most oppressive. It would be better to live under robber barons than under omnipotent moral busybodies. The robber baron’s cruelty may sometimes sleep, his cupidity may at some point be satiated; but those who torment us for our own good will torment us without end for they do so with the approval of their own conscience. They may be more likely to go to Heaven yet at the same time likelier to make a Hell of earth. -C.S. Lewis
david_shaw 4mo ago
> Members of the trusted coalition should freely share chips and semiconductor manufacturing equipment (SME) with each other, while working together to deny it to adversaries. US export controls on frontier chips and SME to China have been a major contributor to the US’s overall lead in AI, and these policies need to be expanded, tightened, and coordinated with other likeminded states. I understand why Dario thinks this is crucial, but it's a very dystopian view of the medium-term future. I'm not an optimist to the point that I believe that AI will lead to global Star Trek-style utopia (although it theoretically could), but ongoing disparity between "allied" and "enemy" powers relating to hardware technology and software models is both not really possible to enforce in the long term, and a pretty dismal state of global affairs even if successful. I'd be interested in an expert geopolitical opinion on what the long tail of this would really look like in any sort of reasonable reality.
gck1 4mo ago
While I do understand the risks, I don't understand the solution. Essentially, Dario is saying that powerful model weights can't be distributed (ban open weights), and governments should coordinate and agree on standards, and block any dangerous model from being used at all, with government deciding what dangerous means. Okay, I don't understand how legitimate access is granted then. Surely, Dario isn't saying to ban Sonnet, because I can definitely make it do cyber harm, as most exploits that I've seen in the wild with my own eyes were trivial. So the only way I see his proposal working is: - No open weights, AI is centralized in the hands of few - We get AI-FAA that sets the rules and monitors - If I want to do a security scan of my codebase, I get a time and scope limited license from AI-FAA that I upload to claude that will allow it to run the security scan in cloud with their models - Claude Mythos Scanner(TM). Dario's proposal ultimately requires that people lose direct access to inference via API. Is this why they've been building SaaS clones with AI bolted on?
david_shaw 4mo ago
> A nation that possesses powerful AI facing one without it—or even facing one that is behind in AI by 3 years—could be the equivalent of an army of World War II Marines facing an army of medieval swordsmen. This is a somewhat ironic take from someone who very publicly feuded with the US government about whether their AI could be used for waging war.
- CamperBob2 4mo ago
  His entire "I Have No Moat And I Must Scream" essay rests on the assumption that our government will remain aligned with both his own company's interests and those of our democratic society as a whole... as if our government is even starting out from such a position. After his recent head-butting defeat by Hegseth's Department of War (sic), it's just downright surreal to read passages like this: Frontier AI models, like airplanes, should be required to go through technical testing and auditing, and their release should be blocked or reversed as a threat to public safety if they do not meet high standards of safety. I am grateful to see the Trump administration’s Executive Order move incrementally towards a greater role for government in AI, though Anthropic’s proposal recommends even further action. Either he's playing us all for fools, or he's playing himself. I suppose both could be true.
gck1 4mo ago
These last few days, I can't help but think that we're now at crossroads that future people will remember as one of two: - And this were the first steps of Anthropic establishing worldwide corporate technocracy. - And this is when Anthropic lost and everyone got access to AI. Similar to how IBM's defeat allowed us to have PCs.
jgil 4mo ago
> As a company, Anthropic always does as much as it can to work with customers to find creative new use cases and new sources of revenue that allow them to do more with their existing workforce, rather than focusing solely on cost savings (which often means reducing the workforce). Without direct workforce or policymaker representation on the boards of private entities, the private sector will seek to maximize shareholder value even if that means workforce reductions. It's not clear that any country could realistically ensure that incredibly powerful industries/private sector entities operate perfectly aligned with national interests, short of nationalization. Large tech companies are already quasi-state actors. In theory, international law and regulations can be binding and enforceable. We see how well that works in practice.
baq 4mo ago
Dario has been riding this exponential for longer than almost anyone here, I’d recommend people try to not scream ‘regulatory capture’ immediately when the risks have indeed materialized and the trend critically does not show any signs of slowing down, in fact the only disagreement is whether it’s accelerating. You have to start thinking in log scales to be able to forecast anything.
simianwords 4mo ago
Of all the points, I find only this one fair: Dario is making it hard for competitor startups to come up because he's proposing additional regulations. A good proposal here is: should Anthropic and OpenAI become sort of VC's that fund other competitors?
- eieie11a 4mo ago
  With what money you fool? They are deep in the red. That’s before considering reinvestment needs.
  - simianwords 4mo ago
    If you think they are deep in the red why do you want any regulation? They will cease to exist. You don’t have to worry in that case.
    - eieie11a 4mo ago
      [flagged]
    - wwwja 4mo ago
      You must be pretty stupid to write such a post. Regulation puts up a barrier to entry, enabling incumbents to have greater market power.
agentrank_ai 4mo ago
[flagged]
mhitza 4mo ago
Does this guy have exponential breakfasts? I agree on some points about the missuse of AI particularly for surveillance, military and propaganda. But this reads like a post further glazing Mythos, and we are just one or two years away "trust us guys", and similar to Mistral's policy plea "please use AI everywhere or we are going to be left behind". I had the hardest time accepting one of his first points that LLMs could barely write a line of code 4 years ago. ChatGPT 3.5 was reasonable at code writing but hallucinated a lot of library functions. Yes we have better harnessing today, and models have been further finetunned with reallife code, but pushing this argument just to support his exponential narrative is deceptive. Like most AI marketing.
deleted 4mo ago
[deleted]
deleted 4mo ago
[deleted]
AlexCoventry 4mo ago
> Often this is for good reasons: governments have grave powers, and it’s usually for the best that they aren’t used too hastily. "Don't mention the (Iran) War!"
deleted 4mo ago
[deleted]
w10-1 4mo ago
Does it really help to say it's exponential? People are not in charge, and have not been for some time. Corporations and governments select their leaders and policies to advance their interests. People fit the work, not vice-versa. Only external competition or internal capabilities limit them (i.e., predation or resources). External resources have been optimized as profit and exported costs; now that AI replaces the pesky need to source elites, capabilities will expand, which will result in more competition. Law is not a lot more than settled expectations, and increasing capabilities changes expectations much more than even market disruption and disinformation, so I wouldn't expect law to save us. As far as I can see, only competition will temper things, and only if AI companies are seen as responsible for their customers' applications - which I doubt. I say this in hope of being proven wrong.
naveen99 4mo ago
Anthropic reminds me of the Democratic Party in the days of Biden. Who is really running the show ?
janpeuker 4mo ago
> Ban the domestic use of fully autonomous weapons. "Domestic" is an interesting shift here from the earlier _general_ discussions on autonomous weapons. So robot imperialism is good for Dario. But the imperial boomerang is a thing, in which case the proposed regulation itself is smoke and mirrors or just regulatory capture?
capitanazo77 4mo ago
When has giving power to politicians have had a good result? Please don’t
VortexLain 4mo ago
Oh yes, let's induce regulatory capture and ban open-weight models in the name of alignment.

169 points by yjp20 4mo ago

deleted 4mo ago

[deleted]

How predictable. The company currently on top wishes to use the regulatory power of the state to prevent competitors from encroaching on their market dominance. It’s a tail as old as time, although their CEO’s rarely publish blog posts about it.

thewebguyd 4mo ago
Yep > "Models above a threshold of compute should undergo mandatory testing by a qualified third party for their level of risk in four specific areas: cybersecurity, biological weapons, loss of control of AI systems, and automated R&D that could accelerate these other risks." AKA: Make it as expensive and untenable as possible for any open source model to jump through the regulation red tape so we can pull the ladder up behind us. Disgusting. All the marketing talk about "this model is so dangerous" "omg we can't release this to the public its so dangerous" etc. is just priming for the incoming lobbying for protectionism from foreign competition, and regulation preventing the development of any other model that could threaten their dominance in the name of "safety"
- slopinthebag 4mo ago
  Yeah, they know their moat is evaporating as the Chinese models continue to catch up, and at a much cheaper price. Also, why has my comment been flagged? It is sitting at positive votes but has suddenly been flagged for no apparent reason?
  - dang 4mo ago
    If you mean https://news.ycombinator.com/item?id=48481131 https://news.ycombinator.com/item?id=48481131, users flagged it. We can only guess why users flag things. Perhaps it was because they thought the comment broke the site guidelines by being snarky and/or fulminatey? I don't think it was an extreme case of that, so I've turned off the flags now.
    - ofjcihen 4mo ago
      I mean it’s pretty tame and actually adds to the conversation. It feels somewhat obvious that someone just disagreed and flagged it. Is there a system for holding people accountable who abuse flags?
      - dang 4mo ago
        Yup there is. I've restored a few other comments that were similarly flagged as well.
    - cadamsdotcom 4mo ago
      Thanks for staying on top of these things dang - such an important topic so it’s great to see the discussion rage but stay constructive.
    - slopinthebag 4mo ago
      Thanks big dog

deleted 4mo ago

[deleted]

themafia 4mo ago

> AI models have gone from barely being able to write a coherent line of code to writing most of the code at major AI companies. Gasoline has gone from barely being able to power stationary farm machines to now being the fuel that underpins our entire economy. So, great news all around, right? > which predict an exponential increase And was that actually delivered? Real question: If a model goes from 80% accurate to 85% accurate is that an exponential increase in "cognitive capabilities?" Are we considering training costs and effort?

interestpiqued 4mo ago
Yeah I’m lost by the constant use of “exponential”. What is the x and y axis when the talking heads say this?
- TobyTheCamel 4mo ago
  I think people are typically referring to the task-completion time horizon at a fixed success rate [1]. That has had pretty robust exponential scaling for many years now. [1] https://metr.org/time-horizons/ https://metr.org/time-horizons/

deleted 4mo ago

[deleted]

pdhborges 4mo ago

What will be Amodei's job after we have AIs that are better at evrything than humans? Is the AI going to care about our stock exchange playgrounds that reward the future Antropic stock holders?

pixl97 4mo ago
Well obviously he'll have the AI 'dispose' of the poor and live a life as a king with a select few farmed humans and have the world as a play thing. Really the entire future of AI at this point seems like "Don't worry about it, we'll figure out when we get there". Works a lot better if you're extremely rich and can afford your own private security.
- nemomarx 4mo ago
  I never understood why people setting up bunkers expect the security to still be loyal after whatever happens.
  - rootusrootus 4mo ago
    Even loyal security would hardly be sufficient. If a million people have decided they want your head on a pike, even a billionaire cannot afford a big enough army. And what exactly is a billionaire in that world anyway?
  - stackbutterflow 4mo ago
    Can't find the article mentioning it but apparently it's an open problem they're thinking about. But yeah if society collapses these billionaire nerds are the first to go. Quietly, in their bunkers, while the team leader of their seal mercenary team takes over. Even before the rest of us realizes what's happening.
    - stackbutterflow 4mo ago
      Found it https://www.theguardian.com/news/2022/sep/04/super-rich-prepper-bunkers-apocalypse-survival-richest-rushkoff https://www.theguardian.com/news/2022/sep/04/super-rich-prep... > Finally, the CEO of a brokerage house explained that he had nearly completed building his own underground bunker system, and asked: “How do I maintain authority over my security force after the event?” The event. That was their euphemism for the environmental collapse, social unrest, nuclear explosion, solar storm, unstoppable virus, or malicious computer hack that takes everything down. This single question occupied us for the rest of the hour. They knew armed guards would be required to protect their compounds from raiders as well as angry mobs. One had already secured a dozen Navy Seals to make their way to his compound if he gave them the right cue. But how would he pay the guards once even his crypto was worthless? What would stop the guards from eventually choosing their own leader? The billionaires considered using special combination locks on the food supply that only they knew. Or making guards wear disciplinary collars of some kind in return for their survival. Or maybe building robots to serve as guards and workers – if that technology could be developed “in time”. I tried to reason with them. I made pro-social arguments for partnership and solidarity as the best approaches to our collective, long-term challenges. The way to get your guards to exhibit loyalty in the future was to treat them like friends right now, I explained. Don’t just invest in ammo and electric fences, invest in people and relationships. They rolled their eyes at what must have sounded to them like hippy philosophy.

tripleee 4mo ago

My god this guy is insufferable. Stop mis-using the term exponential

TobyTheCamel 4mo ago
Looks pretty exponential to me [1]. From a fully independent, non-profit research group. [1] https://metr.org/time-horizons/ https://metr.org/time-horizons/
- interestpiqued 4mo ago
  Release date seems like a terrible x axis with how much more compute they are using. Not to mention while I like what METR is trying to measure, it is an uber specific metric. And frankly, me just complaining, they’re prompts I feel do most of the work for the AI. I’ve never gotten as detailed instructions as they give the AI for the task
  - HDBaseT 4mo ago
    Whilst true, if you had unlimited compute 5 years ago, we wouldn't be anywhere near Mythos level purely because the technology behind the models wasn't refined enough.
baq 4mo ago
Suggest a better name for what’s happening with LLMs please.
- dontlikeyoueith 4mo ago
  Snake oil scam. They simply don't do what the label on the box says they do.
  - teliosix 4mo ago
    It is really hard to believe you actually believe this unless there really is this class of people that are so addicted to social media that they have confused performativity with actual thinking.

ofjcihen 4mo ago

Can we not open up every article talking to working professionals as if they’re children? I like to stay up to date on things but more and more I’m finding myself pointing codex at a URL and saying “get to the point”.

bendergarcia 4mo ago
He wrote it with ai that’s why

SkitterKherpi 4mo ago

It is impressive how well they've scheduled all their releases, posts, and other news to dominate the tech news cycle almost every day in this pre-IPO phase.

Imnimo 4mo ago

>The government should have the power to block or deter deployment of the model if it is determined, in light of third-party assessment, to present unacceptable risks. This power must be scoped to the above four specific risks and there must be protective measures against political favoritism or arbitrary decisions. I feel significantly less sympathy for Anthropic's Supply Chain Risk designation if they believe the government should have this power over them. You get what you sign up for.

hmmmmmmmmmmmmmm 4mo ago

Sit back and watch the stock market implode as these lot double down on safety theatre. The hype is coming to an end.

aspenmartin 4mo ago
> The hype is coming to an end I'm sure you have evidence for this
rootusrootus 4mo ago
They have to keep it up at least until the IPO dust settles.

SilverElfin 4mo ago

I have to be honest, I am tired of reading these arrogant, self-absorbed posts from Dario and Anthropic in general. Opening with this lord of the rings reference just feels like they are trying too hard and are untrustworthy. As annoying as their tone is, the real big danger is what they are setting up for. All this fear-mongering around Mythos, the overly aggressive controls on Fable, and these manifestos they keep writing, are part of setting up for REGULATORY CAPTURE. Even collaborating with the Pope and the Interfaith Alliance (https://iafsc.org/our-work/faith-ai-covenant https://iafsc.org/our-work/faith-ai-covenant) are part of creating a vast support network for regulations and restrictions. Those regulations will help those faith organizations or the government or whatever, but will also help Anthropic’s bottom line. Those regulations will not support your civil liberties. They will restrict speech, access to AI, and allowed uses of AI. They will lead to bans on use of models from some countries like China, and also bans on open-source or open-weight models. If Dario wants to be trusted, he needs to explicitly say in writing that Anthropic will not support any legal or regulatory restrictions on open-source AI, open-weight AI, or Chinese models. Otherwise, what he is really saying - even as he claims he is trying to ‘defend democracy’ - is that he and Anthropic do not truly support fundamental rights like our right to speech. It’s not just Anthropic either. OpenAI had their own recent polemic, pushing for regulations like mandatory safety reviews by agencies for “frontier” models (https://news.ycombinator.com/item?id=48387246 https://news.ycombinator.com/item?id=48387246). It’s a dead giveaway that these companies have no moats, are in serious danger of being a commodity, and are now in the process of using regulations and enshittification to hold onto money and power.

nemomarx 4mo ago
I mean he wants to be trusted but he also absolutely doesn't support freedom of speech in that sense, and I doubt anyone with power or influence over ai policy does? If you find a good lobbying group with money who can push for it let HN know.
- mananaysiempre 4mo ago
  To be honest, at this point I’d take any group at all willing to usefully and non-hypocritically push for free speech (as a general societal value, not just a US-specific legal notion).

fnoef 4mo ago

[flagged]

verandaguy 4mo ago
LOTR is easy to love. It's a classic story arc, well told, and well adapted for mass consumption. It's also a very thinly veiled metaphor for the ceaseless pursuit of industrialization in every aspect of life chipping away at the humanity and metaphorical magic of the human experience, and the perils of embarking on that endeavour to the exclusion of everything else. Personally, I find some humour in this situation.
- anematode 4mo ago
  Maybe Dario should have cited The Lorax.

kouteiheika 4mo ago

> AI companies that develop advanced AI models must have strong security standards that protect their model weights So, basically, make open-weight models illegal. It's nice for Dario to come out and say this so explicitly.

raincole 4mo ago
People ask what the 'open' part of 'OpenAI' even means as they're a for-profit company. It turns out it means 'opener than Anthropic.'
- inigyou 4mo ago
  Opener Anthropic Imitator
simplyluke 4mo ago
Dario's been beating the regulatory capture drum for several years at various intervals, always in the name of safety, but it's hard to not see how self-serving it is. I'm personally very tired of reading the linear-algebra-median of every AI safety essay from lesswrong with the inserted opinion of "therefore all my competitors, especially those pesky open source ones from scary countries should be illegal, only I can be trusted to not abuse the computer god that I definitely will have in just a couple more releases and with a couple more trillion invested"
ed_balls 4mo ago
"We Have No Moat, And Neither Does OpenAI"

iamniels 4mo ago

I like that he comes up with new laws and regulations for AI companies. Can I suggest some more? - You shall not embed copyrighted material in your models. - You shall not bombard every little website in existence with 1 million scraping queries per day. - You shall not use your political influence to pump and dump your AI (or rocket?) company. - You shall not imperill the whole IT sector by buying all CPU and memory chips. These new rules will affect every society directly in a positive way. Thanks.

bashtoni 4mo ago
You missed a couple ofvery important ones: - Your AI data centres will run only on renewable energy - Your AI data centres will not use evaporative cooling
deleted 4mo ago
[deleted]
tptacek 4mo ago
It is normal, expected, and healthy for stakeholders in a regulatory environment to offer proposals about regulations. What's unhealthy is the proposition that the deliberation process is so fragile that a stakeholder needs to cover every angle, lest they corrupt the outcome.
- ofjcihen 4mo ago
  What you say is true but completely ignores the obvious ways in which what he is proposing benefits his company. Its like saying it’s normal for a taxi driver to drive people places while he’s got you handcuffed in the trunk.
  - movealongnow 4mo ago
    [flagged]
- pseudalopex 4mo ago
  It is normal, expected, and healthy to offer criticism of self interested proposals. And mock even. What is unhealthy is to imply someone said what they did not.

snaking0776 4mo ago

I know this is likely just for IPO hype but when I read things like this I sometimes wonder if I must be missing something. I use agents everyday and find them really useful and they save me a lot of headache. At the same time I find that if I let it self-direct at a high level at all it generally makes bad choices that cause me headaches later so I can’t really give them autonomy. Enough people seem to believe this exponential line of thinking though that I keep having to wonder: am I the one missing something here? Is there some magic tool that I haven’t found yet that will cure cancer?

ofjcihen 4mo ago
N+1. This is my experience and for the most part the people that I work with share the same feeling. A highly enthusiastic concussion enthusiast with 10 hands is how one person put it. These are people in different fields but highly accomplished so I’m feeling comfortable sharing their assessment.
- TobyTheCamel 4mo ago
  You're talking quite statically though. I don't think anyone is worried about today's models being a serious threat, but next year's, three years' time? Just three years' ago these models were useful bumbling fools and it's hard to judge where on the S-curve we currently are. I'd rather be thinking about these issues in advance rather than waiting until the problem becomes real.
  - ofjcihen 4mo ago
    > I don't think anyone is worried about today's models being a serious threat Fable is essentially bricked for my areas of interest (even being a member of the cybersecurity program). It seems like they’re attempting to sell regulatory capture under the guise of safety. That’s more of the point.
  - dontlikeyoueith 4mo ago
    The models are still useful bumbling fools. We're in the flat part of the curve because we've exhausted existing data sources.

advael 4mo ago

It's crazy how all these tech CEOs develop the same sense of ethics that seeks to make the foundation of open research and development that made their efforts possible and may threaten their market position illegal in the name of safety against nebulously-defined risks

d_silin 4mo ago

The only effective action is push back on Athropic.

TobyTheCamel 4mo ago

I feel completely baffled by the other responses on this thread. People viewing this purely as a marketing stunt, regulatory capture or attack on their freedoms, with seemingly no appreciation of the real threat that AI could pose to society and even humanity given its current rate of progress. I'm not going to claim that the CEO of pre-IPO company has no incentive to bolster the claims of his tech, but to completely disregard everything he is saying based on that seems awfully binary. I don't know whether people are just high on copium, spouting "it's just fancy autocomplete" or "only humans can really be creative" on every LLM-related thread, but it is impossible to deny that in a span of a few years we've gone from models that could barely put together a sentence, to something maybe not equivalent to a junior developer, but at least resembling it. And sure, you can point out every flaw that current day LLMs have, just how everyone pointed out that Stable Diffusion couldn't generate accurate hands (until it could 6 months later!). But the gradient is pretty clear and I am yet to see a well-argued narrative from anyone why scaling laws should fail in the next year or two (by which point it feels like we're going to have a real problem, extrapolating the current trajectory). I'm very glad this discussion is at least being had, and I wish everyone would get off their high-horse and take things a bit more seriously.

mitthrowaway2 4mo ago
I agree that AI poses a threat to society. I act on this by not developing world-leading AI models and offering them to anyone willing to pay top dollar, while funneling that money back into accelerating AI capabilities development. Maybe Dario would consider taking a similar ethical position? Maybe he would support restrictions and taxes on data center construction, in order to slow down the pace?
- TobyTheCamel 4mo ago
  If Anthropic were not developing these models, one of many other companies would be. I think it's good that the CEO of the current world-leader is at least considering these discussions and platforming possible solutions. The fact that he doesn't support more restrictive approaches that don't align with his incentives doesn't invalidate the points he is making.

anothermathbozo 4mo ago

> A wide range of pro-employment policy incentives can help to slow or reduce job displacement, including: wage insurance policies that compensate people when they have to take a lower-paying job, retention tax incentives to encourage employers not to make layoffs, workforce training grants, or infrastructure to facilitate matching of employers to employees to speed the rate of labor market adaptation. While the particulars of which interventions are best will depend on what kind of labor displacement AI brings, we should readily accept the costs and market inefficiencies that these policies could entail, particularly as they are likely to be offset by AI-driven productivity gains. People get income from one of three places: capital income, labor income, or the welfare state. If this technology truly unlocks a holy panacea of productivity with a commensurate drop in employment then capital’s share of the national income can and should provide for a wider and deeper welfare state. Nothing new need be invented here. Dario’s long and only somewhat organized list of policy interventions makes appropriate preparedness sound like a manic pulling of any and all levers when a simple theory of distribution will suffice.

jelling 4mo ago
This and we already did a dry run of ad-hoc distributions with COVID relief. They had to use the data from tax filings but it did work in terms of getting the money out there.
SpicyLemonZest 4mo ago
> If this technology truly unlocks a holy panacea of productivity with a commensurate drop in employment then capital’s share of the national income can and should provide for a wider and deeper welfare state. This isn't guaranteed in the tax system as it exists today, because reinvestments into further growth are often treated as expenses which cancel out the income for tax purposes.
- anothermathbozo 4mo ago
  You’re conflating firm level taxable income with the national income.

patcon 4mo ago

The comments here. They make me feel that we are so doomed. We all want to nuclear codes so badly. We are addicted to intelligence and labour so badly that we simply can't concieve that a pro-social actor might want us all not to have it, and for good reason. I mean... Obviously, insiders like Oppenheimer (who dedicated their lives to considering the implications of the technology under discussion), they just feared nuclear proliferation because they wanted all the profits for themselves, right :(

polishTar 4mo ago
Look - it's WAY more fun to just call Dario a goober than engage with the actual substance of the essay. Duh! Sure, this may be the most important invention ever with near certainty to reshape society over the next few years, but meh. We should probably just immediately dismiss the concerns of anyone working on it without addressing their arguments at all. It's easy, we can justify ignoring their warnings by saying they're self-interested or too self-important or whatever. Life is more fun when you live it with your eyes closed! You should try it out too.
- patcon 4mo ago
  Omg I love you. Thanks

thefounder 4mo ago

This guy can’t stay a day without posting something more or less “ban open source AI”. We keep you safe

tetrisgm 4mo ago

Honest question: is there a reason for the naming conventions for these models? Anything that makes it better than giving them names with model numbers, like “Claude 3” or such?

kridsdale1 4mo ago
Yes. Each of the nouns is a “size class” in literature. From small lines poem (haiku, sonnet) to larger story (fable) to very large story (opus) to culture-defining foundational (myth). It’s a fun way to say how many parameters are in the model without revealing a number like 405B or 17B which isn’t really comparable vs other models.
- tetrisgm 4mo ago
  Interesting. Thanks for enlightening. My non-poetic brain thinks we should call then Mini, nothing, Pro, Max, and then version numbers. Exactly like Apple. It'd be so much easier to parse. Maybe the AI companies like having the affectionate names haha
  - HDBaseT 4mo ago
    I personally hate the Mini, Nothing, Pro and Max nomenclature. I find it very tacky and the confusing having to remember a "base model" exists without any descriptor. Apple does make it worse though, they have products like the iPhone Pro Max, which combines two!

jampekka 4mo ago

"Democracies should seek to form a global coalition centered on building AI according to their common values, iteratively trying to draw in the rest of the world by making it more and more attractive to be part of the coalition and less and less attractive to be outside it." It's not clear to me on which side of the coalition USA is meant to be in this divide. And as an European I'm not sure whether being in China's or USA's coalition is better in the long term. In general, this deliberate mongering of ever more geopolitical division is extremely harmful. As is the Trump bootlicking.

deleted 4mo ago

[deleted]

deleted 4mo ago

[deleted]

swingboy 4mo ago

[flagged]

dang 4mo ago
Please don't post unsubstantive comments to Hacker News, regardless of who is a goober or you believe they are.

hmokiguess 4mo ago

https://xcancel.com/cuimao/status/2058458683781365873 https://xcancel.com/cuimao/status/2058458683781365873

kingstnap 4mo ago

Its hard to read the first half of this as anything other than regulatory capture propaganda. It really all ties together as: > AI has become a major commercial technology >Frontier AI models, like airplanes, should be required to go through technical testing and auditing, and their release should be blocked or reversed as a threat to public safety if they do not meet high standards of safety > AI companies that develop advanced AI models must have strong security standards that protect their model weights Anyway Dario's financial interests aside. This is an interesting breakpoint for me. > Second, any response to AI-driven job displacement needs to address both the need to provide for everyone economically, and the need for people to find meaning, purpose, and agency. The latter is ultimately more important To me this reads as an out of touch statement. I think the majority of people on earth work to keep a roof over their heads. Of course work can be a source of meaning, purpose, and agency, but to call it the more important aspect on a societal level is a sort of rich person like Dario statement to make.

FergusArgyll 4mo ago
Can you explain how this is attempted regulatory capture? To start a lab now which can actually compete for the frontier (i.e. and pass the "Threshold of compute" needed to get regulated) a lab / company would need a ton of money. Surely a well funded operation of that kind can deal with the regulations.
- thayne 4mo ago
  For one thing, the financial barrier may not necessarily stay that high forever. For another, such regulations could prevent a competitor from making the weights open for their model to try and disrupt the competition. And finally, Amodei would no doubt want to be involved in designing the tests the AI needs to pass, and could (and likely would) design it in a way that Anthropic models would be able to pass easier than competing models.
- gck1 4mo ago
  You'd need a ton of money if you followed the unoptimized, cash burning mindset of existing AI labs. There's probably a ton of optimization that is just sitting on the table. Chinese labs have proven it can be done for way less money. Then there's running inference service of open weights, which doesn't necessarily require opening a lab. You can grab Chinese model weights and sell inference. Anthropic wants to make sure nobody can open a new domestic lab, or provide inference services of unauthorized open weight models, or release open weights if model is good. It is regulatory capture - it covers all areas that are dangers to Anthropic's bottom line.

gedy 4mo ago

I think we should treat this with smirking suspicions until their IPO happens.

jdw64 4mo ago

I read this essay, and it feels like lying behind a mask of moral responsibility and safety for humanity. They are asking for FAA style preclearance and third party audits. That literally means no new AI startup can emerge. Do they not know that audits cost money? Protect your own monopoly, protect your customers' regulations. They want strong regulation like the FAA to raise barriers to entry for the foundation models they themselves build, but then why do they want to loosen FDA regulations? While at the same time driving token consumption from their own customers. They talk about permanent job displacement and UBI. I usually call this "a morally packaged safe landing." They are doing something unpopular (destroying jobs) and getting criticized for it. But they do not want to be criticized further, and they want to ask for social sympathy. So they claim a 'noble cause' that everyone can sympathize with and that is safe for themselves AI will generate astronomical productivity gains and capital profits, which AI companies privatize. So why should the social costs be paid by national taxes? In my opinion, something like "We will donate all of our AI companies' revenue for the next 10 years to society" would show genuine sincerity. Then they say, if we do not develop AI, China will eat our lunch, and they go after China. But is not this really about preventing Chinese dumping, maintaining our own token prices, and asking the world to beat down China so that they can preserve global tech hegemony? But by blocking China from the CUDA ecosystem, now the CANN ecosystem has emerged, has it not? If China develops techniques that reliably reduce inference costs, who knows how things will turn out then. Honestly, I like Anthrpic's Claude, but the Anthropic CEO's rhetoric is so stale. It is not that it feels hypocritical. It is that this is just a one dimensional rhetorical tactic that assumes the public is stupid. I do not think open source is unconditionally good. (It is good, but it can become bad in all situations or all countrie). Open source itself is a barrier for countries outside the Anglosphere when they want to release IT products. Because there is no incentive to buy a product that is worse than an open source alternative. So I do not think everything necessarily has to be open source. But this (referring to Anthropic's position) seems to treat people like fools. If regulation is needed, shouldn't they also argue that FDA regulation is needed? I wish they would be consistent

aesthesia 4mo ago
> They are asking for FAA style preclearance and third party audits. That literally means no new AI startup can emerge. Do they not know that audits cost money? Training frontier AI models costs money, orders of magnitude more than third-party audits. If you can afford to build the model, you can afford to have it audited.

JustFinishedBSG 4mo ago

It's very hard those days to think of companies/people more arrogant than Anthropic/Dario, which is quite the achievement as the bar is very high. If that arrogance was well placed at least you could somewhat excuse it, but the fact that it is so overtly hypocritical and based on false premises just makes it so much worse.

voxleone 4mo ago

Regarding regulation: I'm deeply invested in computer vision systems and i fear that policymakers [who are not deeply familiar with the technical distinctions between AI systems] may write broad rules that cover "AI" generally. In that case, computer vision companies and industrial users could end up subject to requirements that were largely motivated by concerns about generative AI and LLMs.

jldugger 4mo ago

>AI Exponential How much of the policy prescription changes if the exponential is actually just a series of sigmoids[1]? [1]: https://x.com/ylecun/status/1799064075487572133 https://x.com/ylecun/status/1799064075487572133

aspenmartin 4mo ago
Ooof a Yann LeCun quote from '24, a risky time period to be mining his quotes from lol. But I think broadly he's right, but the argument is: we don't see any evidence for the curve flattening, and we should plan for it _not_ to flatten any time soon. "It will eventually slow down" is true but its meaningless if you dont put any sort of time period on this.

mountainb 4mo ago

This reads like an AI with an overflowing context window wrote it; or in the alternative it’s a list of statutes written by an arrogant and delusional king. It is this type of arrogance that will lead to an unfavorable reaction by Congress.

dontlikeyoueith 4mo ago
The sooner the better.

maxglute 4mo ago

What about punishment on AI exponential for security failures, electric chairs for CEO of high ranked frontier models that gets jailbroken by geopolitical adversaries.

silexia 4mo ago

AI is a lethal risk to mankind and should be totally rolled back for a century of close research before we try again.

re-framer 4mo ago

I appreciate the critical perspective on political and economic power, as long as it's consequently followed, and every willingness for cooperation and the creation of fair rules is good. What makes me doubt that Dario Amodei has really internalized the problem is the lack of humility, the stance that it's just important that the "good guys" keep the technology away from the "bad guys". If you really want to provide AI with public benefit, you need to prevent power concentration. How? Some unpolished ideas, I'd be happy to hear yours: - Avoid getting too close to an administration that is openly attacking democracy and is not interested in the benefit of humanity or mutually beneficial cooperation. - Don't support surveillance. Non-(US-)Americans have human rights and privacy, too. Prepare for a situation where a government tries to convert your compute infrastructure into surveillance infrastructure. - Support the creation of community data centers. In other words, build data centers together with local communities and make sure they profit from them. - Advocate for laws that require transparency about resource usage and emissions of data centers. - If you don't want an AI race, make sure that other countries don't need to fear the US concentrating too much power. Create institutions that can be trusted by other countries, too. EDIT: I forgot: - If qualified labor will actually turn out to get devalued, we also need a plan to prevent states from turning into rentier states that don't depend on a well-educated society any longer.

re-framer 4mo ago
(On the other hand, I have been fooled by too many billionaires claiming to act in the interest of democracy and freedom. I once fell for a billionaire buying a social media platform, claiming to be a free speech fundamentalist, and it didn't age too well.)

AnodicElegy 4mo ago

"AI is advancing at a lightning pace—in only four years, AI models have gone from barely being able to write a coherent line of code to writing most of the code at major AI companies. Similar gains have been made in biology, physics, math, finance, law, translation, and many other fields." This is a massive exaggeration. The advancement in the automation of computer code writing has been impressive and is obviously, at least in the short term, changing the software engineering industry substantially. Most other fields have not been affected to nearly the same degree. Certainly not biology, physics, finance, and law (I don't know enough about the math and translation fields to speak to those). --- "3. Accelerating AI’s positive impact..." This whole section is the type of thing that often comes out of the mouths of Silicon Valley tech executives without a pharma background. It indicates a thorough lack of understanding of the realities of pharmaceutical research. What he is describing here is removing many of the solid, evidentiary rules that are in place to make sure that the drugs reaching the market actually work and replacing them with proxy predictions. Look, my least favourite part of the job is the animal testing, and I would be hugely grateful if that could be eliminated from the drug discovery pipeline. People have been trying to do that for a long time. But it's extremely difficult. Biology is very, very complicated. Our understanding of how processes in organisms work are vague and approximative. This is not computer code. Even if Anthropic somehow got all of Big Pharma to hand them their proprietary data, it would only scratch the surface of the understanding that is needed to solve these kind of problems. Due to these realities, the program Amodei is describing here would, effectively, open a floodgate of drugs on the market that don't actually do what they are supposed to and are more likely to have unidentified toxicity.

interestpiqued 4mo ago
Also I’m not denying it would have a major impact either way. But c suite have basically demanded people use ai to code. It’s not all organic

BrenBarn 4mo ago

This was slightly more thoughtful than I feared it would be. However, what I still do not see in any posts from any of these AI pushers is any genuine consideration of the possibility that the best thing might be for them to do less of what they are currently doing. Not to do it differently. Not to mitigate it. Not to do something else in addition. But to actually reduce their current activities.

deleted 4mo ago
[deleted]

observationist 4mo ago

>> Of all tyrannies, a tyranny sincerely exercised for the good of its victims may be the most oppressive. It would be better to live under robber barons than under omnipotent moral busybodies. The robber baron’s cruelty may sometimes sleep, his cupidity may at some point be satiated; but those who torment us for our own good will torment us without end for they do so with the approval of their own conscience. They may be more likely to go to Heaven yet at the same time likelier to make a Hell of earth. -C.S. Lewis

david_shaw 4mo ago

> Members of the trusted coalition should freely share chips and semiconductor manufacturing equipment (SME) with each other, while working together to deny it to adversaries. US export controls on frontier chips and SME to China have been a major contributor to the US’s overall lead in AI, and these policies need to be expanded, tightened, and coordinated with other likeminded states. I understand why Dario thinks this is crucial, but it's a very dystopian view of the medium-term future. I'm not an optimist to the point that I believe that AI will lead to global Star Trek-style utopia (although it theoretically could), but ongoing disparity between "allied" and "enemy" powers relating to hardware technology and software models is both not really possible to enforce in the long term, and a pretty dismal state of global affairs even if successful. I'd be interested in an expert geopolitical opinion on what the long tail of this would really look like in any sort of reasonable reality.

gck1 4mo ago

While I do understand the risks, I don't understand the solution. Essentially, Dario is saying that powerful model weights can't be distributed (ban open weights), and governments should coordinate and agree on standards, and block any dangerous model from being used at all, with government deciding what dangerous means. Okay, I don't understand how legitimate access is granted then. Surely, Dario isn't saying to ban Sonnet, because I can definitely make it do cyber harm, as most exploits that I've seen in the wild with my own eyes were trivial. So the only way I see his proposal working is: - No open weights, AI is centralized in the hands of few - We get AI-FAA that sets the rules and monitors - If I want to do a security scan of my codebase, I get a time and scope limited license from AI-FAA that I upload to claude that will allow it to run the security scan in cloud with their models - Claude Mythos Scanner(TM). Dario's proposal ultimately requires that people lose direct access to inference via API. Is this why they've been building SaaS clones with AI bolted on?

david_shaw 4mo ago

> A nation that possesses powerful AI facing one without it—or even facing one that is behind in AI by 3 years—could be the equivalent of an army of World War II Marines facing an army of medieval swordsmen. This is a somewhat ironic take from someone who very publicly feuded with the US government about whether their AI could be used for waging war.

CamperBob2 4mo ago
His entire "I Have No Moat And I Must Scream" essay rests on the assumption that our government will remain aligned with both his own company's interests and those of our democratic society as a whole... as if our government is even starting out from such a position. After his recent head-butting defeat by Hegseth's Department of War (sic), it's just downright surreal to read passages like this: Frontier AI models, like airplanes, should be required to go through technical testing and auditing, and their release should be blocked or reversed as a threat to public safety if they do not meet high standards of safety. I am grateful to see the Trump administration’s Executive Order move incrementally towards a greater role for government in AI, though Anthropic’s proposal recommends even further action. Either he's playing us all for fools, or he's playing himself. I suppose both could be true.

gck1 4mo ago

These last few days, I can't help but think that we're now at crossroads that future people will remember as one of two: - And this were the first steps of Anthropic establishing worldwide corporate technocracy. - And this is when Anthropic lost and everyone got access to AI. Similar to how IBM's defeat allowed us to have PCs.

jgil 4mo ago

> As a company, Anthropic always does as much as it can to work with customers to find creative new use cases and new sources of revenue that allow them to do more with their existing workforce, rather than focusing solely on cost savings (which often means reducing the workforce). Without direct workforce or policymaker representation on the boards of private entities, the private sector will seek to maximize shareholder value even if that means workforce reductions. It's not clear that any country could realistically ensure that incredibly powerful industries/private sector entities operate perfectly aligned with national interests, short of nationalization. Large tech companies are already quasi-state actors. In theory, international law and regulations can be binding and enforceable. We see how well that works in practice.

baq 4mo ago

Dario has been riding this exponential for longer than almost anyone here, I’d recommend people try to not scream ‘regulatory capture’ immediately when the risks have indeed materialized and the trend critically does not show any signs of slowing down, in fact the only disagreement is whether it’s accelerating. You have to start thinking in log scales to be able to forecast anything.

simianwords 4mo ago

Of all the points, I find only this one fair: Dario is making it hard for competitor startups to come up because he's proposing additional regulations. A good proposal here is: should Anthropic and OpenAI become sort of VC's that fund other competitors?

eieie11a 4mo ago
With what money you fool? They are deep in the red. That’s before considering reinvestment needs.
- simianwords 4mo ago
  If you think they are deep in the red why do you want any regulation? They will cease to exist. You don’t have to worry in that case.
  - eieie11a 4mo ago
    [flagged]
  - wwwja 4mo ago
    You must be pretty stupid to write such a post. Regulation puts up a barrier to entry, enabling incumbents to have greater market power.

agentrank_ai 4mo ago

[flagged]

mhitza 4mo ago

Does this guy have exponential breakfasts? I agree on some points about the missuse of AI particularly for surveillance, military and propaganda. But this reads like a post further glazing Mythos, and we are just one or two years away "trust us guys", and similar to Mistral's policy plea "please use AI everywhere or we are going to be left behind". I had the hardest time accepting one of his first points that LLMs could barely write a line of code 4 years ago. ChatGPT 3.5 was reasonable at code writing but hallucinated a lot of library functions. Yes we have better harnessing today, and models have been further finetunned with reallife code, but pushing this argument just to support his exponential narrative is deceptive. Like most AI marketing.

deleted 4mo ago

[deleted]

deleted 4mo ago

[deleted]

AlexCoventry 4mo ago

> Often this is for good reasons: governments have grave powers, and it’s usually for the best that they aren’t used too hastily. "Don't mention the (Iran) War!"

deleted 4mo ago

[deleted]

w10-1 4mo ago

Does it really help to say it's exponential? People are not in charge, and have not been for some time. Corporations and governments select their leaders and policies to advance their interests. People fit the work, not vice-versa. Only external competition or internal capabilities limit them (i.e., predation or resources). External resources have been optimized as profit and exported costs; now that AI replaces the pesky need to source elites, capabilities will expand, which will result in more competition. Law is not a lot more than settled expectations, and increasing capabilities changes expectations much more than even market disruption and disinformation, so I wouldn't expect law to save us. As far as I can see, only competition will temper things, and only if AI companies are seen as responsible for their customers' applications - which I doubt. I say this in hope of being proven wrong.

naveen99 4mo ago

Anthropic reminds me of the Democratic Party in the days of Biden. Who is really running the show ?

janpeuker 4mo ago

> Ban the domestic use of fully autonomous weapons. "Domestic" is an interesting shift here from the earlier _general_ discussions on autonomous weapons. So robot imperialism is good for Dario. But the imperial boomerang is a thing, in which case the proposed regulation itself is smoke and mirrors or just regulatory capture?

capitanazo77 4mo ago

When has giving power to politicians have had a good result? Please don’t

VortexLain 4mo ago

Oh yes, let's induce regulatory capture and ban open-weight models in the name of alignment.