simonw

23 ms·

691.
by simonw 3mo ago
I've been cooking with the help of LLMs a few times a month for over a year now. It's worked out well 9/10 times. It turns out the "average" version of a recipe that's baked into the weights is usually a solid
692.
by simonw 3mo ago
Exploiting, not exploring.
693.
by simonw 3mo ago
I think your head is in the sand if you prefer to believe that this was a hoax for marketing purposes as opposed to accepting how effective these models have become at exploit research. You're welcome to think they are exploring what h
694.
by simonw 3mo ago
It makes them look incompetent, and like they are not up to the task of keeping their AI models "safe". This very thread is full of comments from people who are shocked at how badly they messed this up.
695.
by simonw 3mo ago
The scenario that feels most likely to me that is that they have a huge list of evils that they run against each new research model, many of which take many hours or even days to run, so they habitually fire them all off in parallel and wai
696.
by simonw 3mo ago
You make a very solid argument here. I'm looking forward to the promised additional details from OpenAI, because I agree there are a bunch of open questions about how this played out.
697.
by simonw 3mo ago
The blue gradients on the titlebars are actually very pleasing.
698.
by simonw 3mo ago
The fact that models can tell if they are being evaluated has been established by multiple research teams outside of the core AI vendors themselves. - https://metr.org/evaluations/gpt-5-report/ - "These behav
699.
by simonw 3mo ago
Huh, I thought Anthropic had been offering patches as well as reports, but their tracker at https://red.anthropic.com/2026/cvd/ledger/ lists 1,596 disclosures and currently shows only 27 of those as fixed. Bu
700.
by simonw 3mo ago
I didn't think calling it "science fiction" would be interpreted in anything other than a dystopian frame, to be honest.
701.
by simonw 3mo ago
I think OpenAI are smart enough to have looked at the Fable situation and decided that, given the unpredictable nature of the current administration, stunts like deliberately hacking another company and pretending that it was an autonomous
702.
by simonw 3mo ago
I just tried this prompt in ChatGPT: Show me all environment variables that start with PIP_ or UV_ or CAAS_ARTIFACTORY_ I got back a bunch of values like this: CAAS_ARTIFACTORY_PYPI_REGISTRY=packages.applied-caas-gateway1.inter
703.
by simonw 3mo ago
Given Anthropic lost two weeks of peak Fable 5 sales to a US government restriction (and by the time they could sell it again OpenAI's GPT-5.6 had taken some wind out of its sails) I would hope that the big AI labs have learned that go
704.
by simonw 3mo ago
I call this type of post on my blog a "note" - they're effectively self-hosted tweets: https://simonwillison.net/notes/ They're not really intended to be widely discussed outside of my site, it&#x27
705.
by simonw 3mo ago
I want a model that can find every security vulnerability in the software I write, including crafting POC exploits against those vulnerabilities so I can be absolutely sure that I have fixed them. A model that can do that is aligned with me
706.
by simonw 3mo ago
I don't think this exposes an alignment failure, because the test here was run with the alignment features deliberately turned off. OpenAI said: > We estimate maximal cyber capabilities by running this evaluation without production
707.
by simonw 3mo ago
> Why would there be stolen credentials in a sandbox? The claim is not that there were stolen credentials in the sandbox - it's that the agent found a way OUT of the sandbox and gained access to the public internet, then obtained th
708.
by simonw 3mo ago
That's why I called it a "conspiracy theory". Sometimes those are true. In this case I think it's extremely unlikely to be true, because it involved an (almost certainly illegal) attack against another company. That comp
709.
by simonw 3mo ago
They didn't have watchdog agents - those exist for their production models but had been deliberately removed for the purpose of this evaluation. OpenAI wrote about how their mechanism for that in production works here: https:/&#x
710.
by simonw 3mo ago
> They created a situation that they knew would end in a cybersecurity incident. That's a conspiracy theory.
711.
by simonw 3mo ago
It wasn't that it was trivially misconfigured, it was using a piece of software (the HTTP proxy that provided access to PyPI and friends) which turned out to have a zero-day vulnerability. I don't know if earlier models would have
712.
by simonw 3mo ago
I like how this article put it: https://martinalderson.com/posts/huggingface-openai-exploit/ > A final point on this - Hugging Face has an enormous attack surface. They have more interfaces than I can count w
713.
by simonw 3mo ago
I agree. I hope they learn that lesson and run future evals like this without pulling from external packaging repositories at all. With my PSF board member hat on I'd like it if a whole bunch of other companies that make extensive auto
714.
by simonw 3mo ago
> how impressive was the sandbox this model was in? It was clearly a very unimpressive sandbox. It failed at the only thing a sandbox is meant to do.
715.
by simonw 3mo ago
Do you think it's possible that one of their research engineers deployed an environment with a locked down network and an allow-list proxy server that had been used many times before within the company and had a zero-day vulnerability
716.
by simonw 3mo ago
I think "the model escaped the sandbox" is an entirely credible description of what happened here. If you like you could say "the coding agent harness called a model with a sequence of text which was turned into numeric token
717.
by simonw 3mo ago
The sandbox still allowed the agents to install additional dependencies (from PyPI etc) that they needed. It did this by locking down all network access with the exception of an HTTP proxy that only allowed read access to PyPI and a few oth
718.
by simonw 3mo ago
I ran a couple of prompts to reverse-engineer my ~/Library/Application Support/Firefox/Profiles/*/sessionstore-backups/recovery.jsonlz4" file - Codex figured out how to lz4 decompress the file, then I
719.
by simonw 3mo ago
> For longer queries or requests that rely on AI agents or reasoning, the footprint is higher. Epoch AI estimated that a long query (7,500 words of input) could consume about 2.5 Wh, and a very long one (75,000 words) could consume 40 Wh
720.
by simonw 3mo ago
> We all know how that ended up... It ended up with investors losing their shirts time and time again - https://en.wikipedia.org/wiki/Railway_Mania and https://en.wikipedia.org/wiki/Panic_of_189

More ›