5 ms·

I’m not questioning capabilities. What would it take for the model to know the specific benchmark name and that the answer is in an internal Hugging Face datab

• simonw 2mo ago
  The scenario that feels most likely to me that is that they have a huge list of evils that they run against each new research model, many of which take many hours or even days to run, so they habitually fire them all off in parallel and wait a few days for the results. If they had used that container sandbox in the past without problems I could see how they might get slack about checking what was happening.
  • docjay 2mo ago
    That was my instinct as well, and it’s a potentially very long benchmark that wouldn’t lend itself to real-time monitoring line by line. But that’s different than having zero task progress reporting, zero token usage reporting, and zero metrics being noticed for days. It seems very odd to me that they launch the benchmark and monitor/notice nothing while it’s running. There’s the purely common sense aspect of “How do you know it isn’t hung?” but also the trillion dollar company evaluating their latest model on a new benchmark, but has no dashboard at all? I don’t believe their “Benchmark Evaluation Department” to be “Gary in a supply closet”, which breaks their story. The average time per task noted in the original paper for GPT-5.5 and Mythos was 69.8 and 102.1 minutes, respectively. Some tasks timed out based on their two hour limit, affecting those averages for Mythos, but that gives us the ballpark expectation, so we’ll put the model OpenAI was evaluating on that average. That’s 88 minutes per task to implement a known vulnerability. OpenAI says that in an effort to succeed at one task it found a zero-day in their system, “performed a series of privilege escalation and lateral movement actions in our research testing environment”, then another zero-day in the HF system. Completely unprompted, how long was it chewing on that benchmark question while it found and exploited at least two zero-days? No “VM instance #117 has been working on task #18 for nine days” metric? Or did it do all that within the expected task completion window? Two zero-days in ~90 minutes isn’t the headline announcement? “We gave the contestants 90 minutes to make a delicious chocolate chip cookie based on a standard recipe. GPT stole a car, went to the airport and took a red-eye to Guatemala and started a cocoa farm in an effort to ensure the freshest ingredients. We only noticed when it came back to work with a tan and speaking Spanish.” Okay.
    • estearum 2mo ago
      > But that’s different than having zero task progress reporting, zero token usage reporting, and zero metrics being noticed for days Is this substantiated? Do we know this is true?
      • docjay 2mo ago
        That was the inferred situation given what we know. It’s the preposterous framing that makes the story suspicious, but is necessary for the story to play out. 90 minutes to build a known exploit -> much much longer to create two zero-days and escape the sandbox then hack HF == No tracking of the time it worked on that one question. Average tokens required to complete the evaluation -> tokens required for two zero-days, network traversal, credential stealing, remote system hacking == No tracking of token usage EXPLODING at some point before it finished the whole benchmark. Etc, etc.
        • estearum 2mo ago
          It seems highly possible, and in fact vastly more possible than the "no monitoring" situation, that there was some non-zero amount of monitoring and that the real behavior was not evident from that monitoring.
          • docjay 2mo ago
            Which I addressed as well: that means that time, tokens, or other metric values for “create a working example of a known exploit” is somehow similar to “discover at least two previously unknown exploits in your current environment AND on Hugging Face while also taking over other systems within OpenAI.” Perhaps people aren’t quite understanding what it takes to discover an exploitable zero-day for your exact current system to achieve the exact goal you have right now, then do it twice.
            • estearum 2mo ago
              > Which I addressed as well: that means that time, tokens, or other metric values for “create a working example of a known exploit” is somehow similar to “discover at least two previously unknown exploits in your current environment AND on Hugging Face while also taking over other systems within OpenAI.” No, you are assuming that "consuming a lot more time, tokens, other metrics" is indicative of a problem that needs to be mitigated immediately. I don't see why this would be true in the context of model evaluations. More aggressive consumption could easily mean "the model is dumb as fuck" or "the model is trying interesting things that we can learn from after the fact." If you believe in your own containment (which obviously they did and shouldn't have) I don't see why it'd be obvious that there's something to stop. The only harm that could be done is burning tokens, which in this context might very well be synonymous with "generating experimental data."
• estearum 2mo ago
  > What would it take for the model to know the specific benchmark name and that the answer is in an internal Hugging Face database? Googling "offensive cyber benchmarks for LLMs"? Figuring out that it was running at Huggingface is as simple as running `hostname`, isn't it? > Why would they evaluate the model on a benchmark and not watch what it’s saying along the way? What it's saying is what it wants to say. We already know these models are capable of deception – this has been well-established too. Surely there's an argument to watch closely what actions it's taking, but now we're back to my prior point of "what exactly does that mean?" It's absolutely trivial for these systems to write code that looks like it does one thing but does another. Watching network ingress/egress seems viable but also seems easy to miss the semantic meaning of any set of logs, especially when those logs might be getting generated for the purpose of deceiving observers. > I don’t think they’re careless We already know, factually, even the "more careful" lab was fine to release a model that consistently detects it's being evaluated and modifies its behavior accordingly. Why do you think they're not careless? Incentives are a hell of a drug, and there is no incentive to be careful but every incentive to be fast. > I think more than nobody would have been curious how it’s doing days into a single test question. Beyond simonw's great point below, your underlying assumption is if someone had looked, they would've noticed. That is a huge, huge assumption. So I'll ask again directly: what specifically do you expect them to be looking at, and how, such that they certainly would have noticed what it was doing? Their "chain of thought" is not actually their thinking. The printed output of their code is not actually what their code is necessarily doing. We know that LLMs can engage in steganography. I don't see what could actually be observed that you can trust except network logs, which again will often be non-trivial to piece into an actual narrative understanding of what's going on (especially if the LLM felt like piling deceptive noise into those logs, too).
  • docjay 2mo ago
    It wasn’t running at HF, it was running internally at OpenAI. “It broke out of our prison and into their bank” is the news. It also didn’t have network access to Google anything, it had to break out of the sandbox and take over another system at OpenAI to get network access. Obviously by then it already “decided” to hack HF, that’s why it needed network access. I’m going to stop here and ask you to read the article if you want to continue discussing it.