9 ms·
OpenAI delays launch of open-weight model
- puttycat 1y agohttps://nitter.space/sama/status/1943837550369812814 https://nitter.space/sama/status/1943837550369812814
- ryao 1y agoAm I the only one who thinks mention of “safety tests” for LLMs is a marketing scheme? Cars, planes and elevators have safety tests. LLMs don’t. Nobody is going to die if a LLM gives an output that its creators do not like, yet when they say “safety tests”, they mean that they are checking to what extent the LLM will say things they do not like.
- eviks 1y agoWhy is your definition of safety so limited? Death isn't the only type of harm...
- ryao 1y agoThere are other forms of safety, but whether a digital parrot says something that people do not like is not a form of safety. They are abusing the term safety for marketing purposes.
- eviks 1y agoYou're abusing the terms by picking either the overly limited ("death") or overly expansive ("not like") definitions to fit your conclusion. Unless you reject the fact that harm can come from words/images, a parrot can parrot harmful words/images, so be unsafe.
- ryao 1y agoThe maxim “sticks and stones can break my bones, but words can never hurt me” comes to mind here. That said, I think this misses the point that the LLM is not a gatekeeper to any of this.
- eviks 1y agoDon't let your mind potential be limited by such primitive slogans!
- jiggawatts 1y agoI find it particularly irritating that the models are so overly puritan that they refuse to translate subtitles because they mention violence.
- jazzyjackson 1y agoit's like complaining about bad words in the dictionary the bot has no agency, the bot isn't doing anything, people talk to themselves, augmenting their chain of thought with an automated process. If the automated process is acting in an undesirable manner, the human that started the process can close the tab. Which part of this is dangerous or harmful?
- ks2048 1y agoYou could be right about this being an excuse for some other reason, but lots of software has “safety tests” beyond life or death situations. Most companies, for better or worse (I say for better) don’t want their new chatbot to be a RoboHitler, for example.
- ryao 1y agoIt is possible to turn any open weight model into that with fine tuning. It is likely possible to do that with closed weight models, even when there is no creator provided sandbox for fine tuning them, through clever prompting and trying over and over again. It is unfortunate, but there really is no avoiding that. That said, I am happy to accept the term safety used in other places, but here it just seems like a marketing term. From my recollection, OpenAI had made a push to get regulation that would stifle competition by talking about these things as dangerous and needing safety. Then they backtracked somewhat when they found the proposed regulations would restrict themselves rather than just their competitors. However, they are still pushing this safety narrative that was never really appropriate. They have a term for this called alignment and what they are doing are tests to verify alignment in areas that they deem sensitive so that they have a rough idea to what extent the outputs might contain things that they do not like in those areas.
- natrius 1y agoAn LLM can trivially instruct someone to take medications with adverse interactions, steer a mental health crisis toward suicide, or make a compelling case that a particular ethnic group is the cause of your society's biggest problem so they should be eliminated. Words can't kill people, but words can definitely lead to deaths. That's not even considering tool use!
- ryao 1y agoThis is analogous to saying a computer can be used to do bad things if it is loaded with the right software. Coincidentally, people do load computers with the right software to do bad things, yet people are overwhelmingly opposed to measures that would stifle such things. If you hook up a chat bot to a chat interface, or add tool use, it is probable that it will eventually output something that it should not and that output will cause a problem. Preventing that is an unsolved problem, just as preventing people from abusing computers is an unsolved problem.
- ronsor 1y agoAs the runtime of any program approaches infinity, the probability of the program behaving in an undesired manner approaches 1.
- ryao 1y agoThat is not universally true. The yes program is a counter example: https://www.man7.org/linux/man-pages/man1/yes.1.html https://www.man7.org/linux/man-pages/man1/yes.1.html
- cgriswald 1y agoDevil's advocate: (1) Execute yes (with or without arguments, whatever you desire). (2) Let the program run as long as you desire. (3) When you stop desiring the program to spit out your argument, (4) Stop the program. Between (3) and (4) some time must pass. During this time the program is behaving in an undesired way. Ergo, yes is not a counter example of the GP's claim.
- recursive 1y agoI also think it's marketing but kind of for the opposite reason. Basically I don't think any of the current technology can be made safe.
- nomel 1y agoYes, perfection is difficult, but it's relative. It can definitely be made much safer. Looking at the analysis of pre vs post alignment makes this obvious, including when the raw unaligned models are compared to "uncensored" models.
- jrflowers 1y ago> Am I the only one who thinks mention of “safety tests” for LLMs is a marketing scheme? It is. It is also part of Sam Altman’s whole thing about being the guy capable of harnessing the theurgical magicks of his chat bot without shattering the earth. He periodically goes on Twitter or a podcast or whatever and reminds everybody that he will yet again single-handedly save mankind. Dude acts like he’s Buffy the Vampire Slayer
- olalonde 1y agoEspecially since "safety" in this context often just means making sure the model doesn't say things that might offend someone or create PR headaches.
- SV_BubbleTime 1y agoDon’t draw pictures of celebrities. Don’t discuss making drugs or bombs. Don’t call yourself MechaHitler… which I don’t care that while scenario was objectively funny on its sheer ridiculousness.
- jekwoooooe 1y agoSure it’s funny until some mentally unstable Nazi sympathizer goes and shoots up another synagogue. So funny.
- SV_BubbleTime 1y agoI get the over actions and hyperbole are part of the pact… it doesn’t make it any less ridiculous. As a Jew, I think we’re going to be OK and funny things can just be funny.
- olalonde 1y agoThey used to say that about video games.
- halfjoking 1y agoIt's overblown. Elon shipped Hitler grok straight to prod Nobody died
- pona-a 1y agoPlaying devil's advocate, what if it was more subtle? Prolonged use of conversational programs does reliably induce certain mental states in vulnerable populations. When ChatGPT got a bit too agreeable, that was enough for a man to kill himself in a psychotic episode [1]. I don't think this magnitude of delusion was possible with ELIZA, even if the fundamental effect remains the same. Could this psychosis be politically weaponized by biasing the model to include certain elements in its responses? We know this rhetoric works: cults have been using love-bombing, apocalypticism, us-vs-them dynamics, assigned special missions, and isolation from external support systems to great success. What we haven't seen is what happens when everyone has a cult recruiter in their pocket, waiting for a critical moment to offer support. ChatGPT has an estimated 800 million weekly active users [2]. How many of them would be vulnerable to indoctrination? About 3% of the general population has been involved in a cult [3], but that might be a reflection of conversion efficiency, not vulnerability. Even assuming 5% are vulnerable, that's still 40 million people ready to sacrifice their time, possessions, or even their lives in their delusion. [1] https://www.rollingstone.com/culture/culture-features/chatgpt-obsession-mental-breaktown-alex-taylor-suicide-1235368941/ https://www.rollingstone.com/culture/culture-features/chatgp... [2] https://www.forbes.com/sites/martineparis/2025/04/12/chatgpt-hits-1-billion-users-openai-ceo-says-doubled-in-weeks/ https://www.forbes.com/sites/martineparis/2025/04/12/chatgpt... [3] https://www.peopleleavecults.com/post/statistics-on-cults https://www.peopleleavecults.com/post/statistics-on-cults
- stogot 1y agoYou’re worried about indoctrination in an LLM but it starts much earlier than that. The school system is indoctrination of our youngest minds, both today in the West and its Prussian origins https://today.ucsd.edu/story/education-systems-were-first-designed-to-suppress-dissent https://today.ucsd.edu/story/education-systems-were-first-de... We should fix both systems. I don’t want Altman’s or Musk’s opinions indoctrinating
- simianwords 1y agoI hope the same people questioning ai safety (which is reasonable) don’t also hold concern on Grok due to the recent incident. You have to understand that a lot of people do care about these kind of things.
- ignoramous 1y ago> Nobody is going to die Callous. Software does have real impact on real people. Ex: https://news.ycombinator.com/item?id=44531120 https://news.ycombinator.com/item?id=44531120
- layer8 1y agoIt’s about safety for the LLM provider, not necessarily the user.
- deleted 1y ago[deleted]
- stogot 1y agoAt my company (which produces models) almost all the responsible AI jazz is about DEI and banning naughty words. Little actions on preventing bad outcomes
- razodactyl 1y agoThere's a bias that smart people have in underestimating the stupidity of those on the left side of the intelligence curve. Not even that, children believe anything and more so that of a computer designed to be "harmless".
- Rastonbury 1y agoI think that's a bit uncharitable, the top companies have hired top talent who's explicit job is safety, which is obviously a hard problem ever since Microsoft's Tay. Anthropic publishes pretty extensive safety reviews of their models. Misalignment on incentives,prioritisation of speed and it being and almost an impossible task may make it seem like a marketing scheme. It's not like companies who slap on all the certifications AWS has just because they run on AWS
- etaioinshrdlu 1y agoIt's worth remembering that the safety constraints can be successfully removed, as demonstrated by uncensored fine-tunes of Llama.
- adidoit 1y agoNot sure if it's coincidental that OpenAI's open weights release got delayed right after an ostensibly excellent open weights model (Kimi K2) got released today. https://moonshotai.github.io/Kimi-K2/ https://moonshotai.github.io/Kimi-K2/ OpenAI know they need to raise the bar with their release. It can't be a middle-of-the-pack open weights model.
- lossolo 1y agoThis could be it, especially since they announced last week that it would be the best open-source model.
- reactordev 1y agoTechnically they were right when they said it, in their minds. Things are moving so fast that in a week, it will be true again.
- sigmoid10 1y agoThey might also be focusing all their work on beating Grok 4 now, since xAi has a significant edge in accumulating computing power and they opened a considerable gap in raw intelligence tests like ARC and HLE. OpenAI is in this to win the competitive race, not the open one.
- unsupp0rted 1y ago> They might also be focusing all their work on beating Grok 4 now, With half the key team members they had a month prior
- sigmoid10 1y agoI'm starting to think talent is way less concentrated in these individuals than execs would have investors believe. While all those people who left OpenAI certainly have the ability to raise ridiculous sums of venture capital in all sorts of companies, Anthropic remains the only offspring that has actually reached a level where they can go head-to-head with OpenAI. Zuck now spending billions on snatching those people seems more like a move out of desperation than a real plan.
- buyucu 1y agoProbably ClosedAI's model was not as good as some of the models being released now. They are delaying it to do some last minute benchmark hacking.
- Y_Y 1y agoMy hobby: monetizing cynicism. I go on Polymarket and find things that would make me happy or optimistic about society and tech, and then bet a couple of dollars (of some shitcoin) against them. e.g. OpenAI releasing an open weights model before September is trading at 81% at time of writing - https://polymarket.com/event/will-openai-release-an-open-source-model-before-september?tid=1752308786815 https://polymarket.com/event/will-openai-release-an-open-sou... Last month I was up about ten bucks because OpenAI wasn't open, the ceasefire wasn't a ceasefire, and the climate metrics got worse. You can't hedge away all the existential despair, but you can take the sting out of it.
- hereme888 1y agopeople still use crypto? I thought the hype died around the time when AI boomed.
- amelius 1y agopeople use crypto for speculation, and for (semi)illegal purposes only a small percentage of use is for actual legitimate money transfers
- ben_w 1y agoUnfortunatley crypto hype is still high; and I think still on the up, but that's vibes not market analysis.
- unsupp0rted 1y agoBitcoin is higher than ever. People can't wait until it gets high enough that they can sell it for dollars, and use those dollars to buy things and make investments in things that are valuable.
- esperent 1y ago> Bitcoin is higher than ever That's just speculation though. I saw a cynical comment on reddit yesterday that unfortunately made a lot of sense. Many people now are just so certain that the future of work is not going to include many humans, so they're throwing everything into stocks and crypto, which is why they remain so high even in the face of so much political uncertainty. It's not that people are investing because they have hope. People are just betting everything as a last ditch survival attempt before the robots take over. Of course this is hyperbolic - market forces are never that simple. But I think there might be some truth to it.
- fresh_broccoli 1y agoDelays aside, I wonder what kind of license they're planning to use for their weights. Will it be restricted like Llama, or fully open like Whisper or Granite?
- Havoc 1y agoPointless security theatre. The community worked out long ago how to strip away any safeguards.
- thrance 1y agoWhenever I read something similar I immediately remember how "Open"AI refused to release GTP2 XL at the time because it was "too powerful".
- gnarlouse 1y agoHonestly, they’re distancing themselves optically/temporally from HerrGrokler newslines
- qoez 1y agoMy pet theory is that they delayed this because grok-4 released because they explicitly want to not be seen as competing with them by pulling the usual trick of releasing right around when google does. Feels like a very sam altman move in my model of his mind.
- xcrunner529 1y agoNo, Elon, no one is impressed or even thinking positively about MechaHitler
- seydor 1y ago> this is new for us So much for the company that should never be new to that
- deleted 1y ago[deleted]
- jmugan 1y agoWhat is their business purpose for releasing an open-weights model? How does it help them? I asked an LLM but it just said vague unconvincing things about ecosystem plays and fights for talent.
- brap 1y agoPR
- blurbleblurble 1y agoWow. Twitter is not a serious website anymore. Why are companies and professionals still using it? Is it really like that now, with all that noise from grok floating to the top?
- hansmayer 1y agoIs it now coming before or after the release of AGI, which OpenAI, "knows how to build now" ?
- groggo 1y agowhy would OpenAI release an open weight model? Genuinely curious.
- deleted 1y ago[deleted]
- stonogo 1y agowe'll never hear about this again
- mystraline 1y agoTo be completely and utterly fair, I trust Deepseek and Qwen (Alibaba) more than American AI companies. American AI companies have shown they are money and compute eaters, and massively so at that. Billions later, and well, not much to show. But Deepseek cost $5M to develop, and made multiple novel ways to train. Oh, and their models and code are all FLOSS. The US companies are closed. Basically, the US ai companies are too busy treating each other as vultures.
- ryao 1y agoWasn’t that figure just the cost of the GPUs and nothing else?
- rynn 1y agoIt was more than $5m https://interestingengineering.com/culture/deepseeks-ai-training-cost-billion https://interestingengineering.com/culture/deepseeks-ai-trai...
- rpdillon 1y agoYeah, I hate that this figure keeps getting thrown around. IIRC, it's the price of 2048 H800s for 2 months at $2/hour/GPU. If you consider months to be 30 days, that's around $5.7M, which lines up. What doesn't line up is ignoring the costs of facilities, salaries, non-cloud hardware, etc. which will dominate costs, I'd expect. $100M seems like a fairer estimate, TBH. The original paper had more than a dozen authors, and DeepSeek had about 150 researchers working on R1, which supports the notion that personnel costs would likely dominate.
- moralestapia 1y ago>ignoring the costs of facilities, salaries, non-cloud hardware, etc. If you lease, those costs are amortized. It was definitely more than $5M, but I don't think it was as high as $100M. All things considered, I still believe Deepseek was trained at one (perhaps two) orders of magnitude lower cost than other competing models.
- krackers 1y agoProbably the results were worse than K2 model released today. No serious engineer would say it's for "safety" reasons given that ablation nullifies any safety post-training.
- simonw 1y agoI'm expecting (and indeed hoping) that the open weights OpenAI model is a lot smaller than K2. K2 is 1 trillion parameters and almost a terabyte to download! There's no way I'm running that on my laptop. I think the sweet spot for local models may be around the 20B size - that's Mistral Small 3.x and some of the Gemma 3 models. They're very capable and run in less than 32GB of RAM. I really hope OpenAI put one out in that weight class, personally.
- NitpickLawyer 1y agoEarly rumours (from a hosting company that apparently got early access) was that you'd need "multiple h100s to run it", so I doubt it's a gemma - mistral small tier model..
- simonw 1y agoI think you're right, I've seen a couple of other comments now that indicate the same thing.
- aabhay 1y agoYou will get at 20gb model. Distillation is so compute efficient that it’s all but inevitable that if not OpenAI, numerous other companies will do it. I would rather have an open weights model that’s the best possible one I can run and fine tune myself, allowing me to exceed SOTA models on the narrower domain my customers care about.
- dorkdork 1y agoMaybe they’re making last minute changes to compete with Grok 4?
- xcrunner529 1y agoI like how all these comments from random people saying the same thing. Clear astroturf for an AI model no serious person cares about.