3 ms·
This is symptomatic of a trend that Simon Willison called the “relentless productivity” of US frontier lab. Instead of being just “smarter”, like previous model
by stymaar 3mo ago
This is symptomatic of a trend that Simon Willison called the “relentless productivity” of US frontier lab. Instead of being just “smarter”, like previous models were, the current generation is being trained through RL to have this kind of behavior.
Personally I'm not “impressed”, I'm appalled, because this kind of behavior is practically never what you want (if you forgot to give the model a tool, a useful model should identity the missing part and ask the user for it, not spend a billion token building/stealing the tool as a side quest) but it's the perfect recipe for a “universal paperclip” scenario.
OpenAI and Anthropic talk about “safety” a lot, but they look pretty reckless with this kind of RL training pipeline.
- ifwinterco 3mo agoIf you look at what OpenAI and Anthropic actually do, they clearly either don't believe what they're saying, or they're idiots. They're claiming they've developed a cyber grade model that's "too dangerous to release". But then they're running it connected to the public internet, not airgapped, protected only by a software sandbox... exactly the kind of thing an AI trained for cyber stuff is supposed to be able to find bugs in. (Or maybe they were actually hoping this exact scenario would happen because it's good marketing)
- mike_hock 3mo agoI think this a positive effect of LLMs, especially once these capabilities get into the hands of criminals and hostile foreign states, i.e. they will do maximum damage with all the safeties off. This will force everyone to finally take security seriously at both the development and operational levels. You can no longer keep sneaking backdoors into software and count on them remaining hidden for 10 years so you have a nice portfolio of zero days to exploit at any given time.