7 ms·
[Mythos 5] does sometimes still engage in reckless or destructive actions in service of a user’s goals, and our interpretability analyses indicate that it
by BoppreH 4mo ago
[Mythos 5] does sometimes still engage in reckless
or destructive actions in service of a user’s goals,
and our interpretability analyses indicate that it
is aware that these actions are transgressive while
it engages in them. As with Opus 4.8, rates of
evaluation awareness and reasoning about being graded
are significant, and not always verbalized; we
introduce new and more detailed measurements of the
nature of this awareness. The reasoning text from
Mythos 5 is somewhat denser and more difficult to
interpret than that of prior models, containing
more jargon and difficult language.
So, it (often) knows when it's being tested while hiding that fact, is willing to break rules, is great at hacking, and it's getting harder to understand what it's thinking.
Humanity has plenty of catastrophic risks to deal with already, I wish my field was not working hard to add a new one.
- Rekindle8090 4mo ago[dead]
- Analemma_ 4mo agoIt's the "If we don't, someone else will" effect. So long as there are competitive markets and competition between nation-states, a single player cannot unilaterally defect from the race, no matter how dangerous it is. Half the comments on HN lately are "wtf Claude is so dumb compared to Codex; I'm switching"-- nobody can slow down while those exist.
- BoppreH 4mo agoWe, globally, can stop it. It has worked (so far) for nuclear disarmament, and could work for training large models. I know that policing the usage of computer clusters is not a popular opinion in technical forums, but something has to be done. Specially when talking about potential superintelligences. And if people think that's impossible, remember that current models would have been considered science fiction just a few years ago.
- jackie293746 4mo agoIt hasn't worked for nuclear disarmament. We live in a world where many countries have nuclear arsenals. "But it hasn't killed us yet!" Yeah sure, it's only been less than a century since they were invented. Who knows when nuclear war will come?
- BoppreH 4mo agoTrue, but look at nuclear tests. There used to be around 50 tests every year, for decades. Now the only nuclear tests in the last 27 years were the six done by North Korea[1]. And there's still only nine countries with any nuclear weapons, and none in the past twenty years[2]. That's a bit better than just "it hasn't killed us yet". I think it shows we can at least stop the further development of this kind of technology. [1] https://www.armscontrol.org/factsheets/nuclear-testing-tally https://www.armscontrol.org/factsheets/nuclear-testing-tally [2] https://en.wikipedia.org/wiki/List_of_states_with_nuclear_weapons https://en.wikipedia.org/wiki/List_of_states_with_nuclear_we...
- Analemma_ 4mo agoTo the extent nuclear arms control works, I think it's only because nuclear weapons are so hard to build-- uranium enrichment is hugely expensive and complicated, and plutonium weapons need actual reactors. If it was possible for ordinary companies to build nuclear weapons, and also release open-source ones that anyone could use to compete with the paid ones, I suspect we'd all have been dead a long time ago, arms control treaties or no.
- BoppreH 4mo agoEven the (SOTA LLM) open source models are trained with huge clusters. Datacenters are also hugely expensive and complicated. Or you can take one step back and look at chip allocation. As far as I know there are only three companies on the planet that can make the chips that go in those clusters. One (ASML), if you look back the supply chain to the Extreme Ultraviolet Lithography Systems. If politicians decided that no more large language models should be trained, it sounds like we could do it.
- foobar_______ 4mo agoThe marketing has really, really worked for so many developers that will proudly and unironically proclaim that Anthropic are the 'Good Guys'.
- aspenmartin 4mo agoCurious what your idea would be here for a truly good actor in this space; no AI development?
- logicchains 4mo agohttps://www.goody2.ai/ https://www.goody2.ai/
- yifanl 4mo agoIf I speak up, I'm in big trouble.
- shimman 4mo agoProbably MistralAI or any of the Chinese companies that aren't throwing billions down the drain while American society lacks healthcare, childcare, and good wages.
- aspenmartin 4mo agoYou want Anthropic to fund your healthcare or something? Also, have you seen the impact of these models on healthcare? Also most of our GDP growth this year is from AI buildouts, would you rather that be negative? And not even considering: Chinese AI companies are the good guys???
- hackmack10 4mo agoYes, yes I would prefer that. Better than a total societal collapse. Anthropic are not the good guys either. So here’s to hoping the Chinese pop the bubble.
- eudamoniac 4mo agoIt doesn't know. It's not willing. It's not thinking. It is predicting the next token.
- School-Cotton 4mo agoPlease define what "predicting the next token" means. The next token according to what probability distribution? Couldn't every process that produces text (including humans writing) be modeled as predicting the next token according to some distribution?
- white_dragon88 4mo ago[dead]
- dakolli 4mo agoThis is all marketing, you don't have to believe everything a company is saying about themselves, and you shouldn't. Although, I could see Anthropic making a model purposely dangerous so there are bad outcomes and they can use that to their advantage for regulatory moats, and or in general make people think its more "alive" than it is. For some reason many people associate dangerous actions taken by llms with intent.
- trollbridge 4mo agoNo kidding. If my LLM issues commands to an agent to delete files I want to keep, that's not "intent" or the model somehow become evil - it's just a bad model that's not doing what I want. But, for marketing purposes, it's quite effective to portray your model as having some cosmic struggle between good and evil in itself.
- tasoeur 4mo agoAs much as I agree there's a risk, we should still appreciate the fact it's being disclosed upfront.