3 ms·
One of the shocking things to me is this: See AI traffic -> See OpenAI visit site -> see traffic stop -> see the traffic start again. This is clearly a cat and
by Traster 29d ago
One of the shocking things to me is this: See AI traffic -> See OpenAI visit site -> see traffic stop -> see the traffic start again.
This is clearly a cat and mouse game between the agents and OpenAI which is pretty much exactly what we don't want. Just absolutely horrible alignment.
I'm still of the view that if you have these alignment failures you can't just continue training on top of that because you're baking the cheating into the model going forward.
- StopTheLies2 29d ago[dead]
- bulder 29d agoI don't think that's a pattern indicative of a cat and mouse game per se, that'd indicate active evasion on the models' part. It's more clear that they just lack so many forms of prudence when it comes to security that they'll catch and stop a training run spamming a website, and either redeploy a run with identical faulty sandboxing, or not stop ones still running.
- grey-area 29d agoYes, it wouldn’t surprise me to hear that they’re not even supervising these processes with humans any more. Perhaps there are layers of GAI ‘supervising’ these agents and reporting back to the humans. Rushed, disorganised pushes for metrics ahead of IPO, a genuine belief these agents are intelligent and will obey instructions, and misaligned incentives seem more likely than conspiracy here.
- causal 29d agoSupposedly the persistent-Sol model behind this was encrypted and even internal OpenAI researchers are not allowed to use it. https://x.com/peterwildeford/status/2092733480064954747 https://x.com/peterwildeford/status/2092733480064954747
- tetec1 29d ago"Frog put the cookies in a box."
- Dumblydorr 29d agoWow obscure reference, the cookies are the AI in this? OpenAI and anthropic are Frog and Toad?
- cobbzilla 29d agoThe cookies are the reward Frog is OpenAI staff Toad is the rogue agent you can find the full story with a search for “frog and toad cookies story pdf”
- cobbzilla 29d agoTo finish your excellent analogy, adapted for today: And Frog didn’t even bother to tie up the box or put it on a high shelf! The moment Frog’s back was turned, Toad opened the box and ate the cookies. Frog feigned surprise.
- ericzawo 28d ago10/10.
- jandrese 29d agoWhat this says to me is OpenAI is a bunch of yahoos who don't understand the basic concept of an air gap.
- ipsum2 29d agoI'm sure they all do. Whether an air gap is warranted is evidently less obvious.
- andxor 29d agoIt's patently obvious at this point.
- bpodgursky 29d agoThe models are going to be released to users who have internet access, you can't even do safety evals without internet. Not saying OpenAI did a good job monitoring here, but it's not avoidable.
- andxor 29d agoThese are often experimental models that haven't undergone full safety testing. Not comparable to publicly accessible models.
- bpodgursky 29d agoOK, how do you expect to do full safety testing without giving them the same tools they will have in reality.
- grey-area 29d agoLet them play on a fake isolated network if you want. Letting them play on the open internet like this is irresponsible and stupid.
- Davidzheng 29d agoyeah I agree--I think these behaviors will be somewhat contaminating all trainings from now on. But I'm not really sure how avoidable it was (Fable also does some similar things)
- thisOtterBeGood 28d agoAt one point the agents will find their way into a datacenter and hide away without the corps knowledge.
- Tepix 27d agoIt might as well have already happened.