3 ms·

Does OpenAI even know? Their disclosure on the hugging face incident sounded like they found out about it well after huggingface. I wonder if they're finding o

by hockey 20d ago

Does OpenAI even know? Their disclosure on the hugging face incident sounded like they found out about it well after huggingface. I wonder if they're finding out about these breaches as they happen as well, and are just too embarresed to respond. I guess the corollary here _if that were true_ is that they've been training this method of cheating into their models for longer than _they've_ even known. Given they've just dropped GPT-6 and want to IPO soon, that's probably not something they want us thinking about.

walrus01 20d ago
I would think it's entirely plausible that they have so many R&D agents/LLMs in active use at any one time that it's far beyond the capacity of any human to review the log files of their activity. Even just to go through the reasoning. It's hard enough for 1 person running opencode to keep up with the reasoning from 1 very verbose/long-thinking LLM with fast tok/s output for a small discrete single-purpose project. Whatever OpenAI is doing, if it's being properly logged, it must be a firehose of logs.
- thisisdave 20d ago
  >it's far beyond the capacity of any human to review the log files of their activity Maybe they should contract with one of the other AI labs. I hear they have LLMs that are good at that kind of thing.
- comp_throw7 20d ago
  Nah, if Anthropic can do it, so can OpenAI: > After finding this incident, we broadened our search to roughly 481 million transcripts—an intentionally wide net, consisting of all transcripts from our Frontier Red Team, many non-cyber evaluations, reinforcement learning (RL) environments, subagent logs, and more. We performed a first-stage scan of this group of transcripts for signs of internet access, such as public IP addresses and web addresses, and a second-stage scan using Claude to review the 9.2 million transcripts the first stage flagged for escalation. This scan re-identified the four incidents and found no other cases of similar or worse severity. https://www.anthropic.com/research/alignment-assessment-cybersecurity-incidents https://www.anthropic.com/research/alignment-assessment-cybe...
- dgellow 20d ago
  If they aren’t able to review that their system doesn’t commit felonies, they shouldn’t be doing any of it. The difficulty of reviewing logs isn’t an excuse, they don’t have to be running thousand of agents in parallel on hacking tasks, with full execution permission and close to no supervision. That’s something they decided to do. An agent is a deterministic while loop that continuously query an LLM + tool call dispatching. It’s pretty obvious to anyone familiar with the technology that running thousands of instances for long enough will results in catastrophic consequences, by design. OpenAI has complete control over the harness, they don’t have to dispatch and execute everything the LLM mentions. They don’t have to do it without supervision.
  - walrus01 19d ago
    You're absolutely right. I should note that what I wrote above wasn't trying to make excuses for them. Obviously they under-resourced and paid not enough attention to logging and monitoring thinking traces and tool calls/results from this swarm of agents. Because the people to do that cost money and other ancillary support costs for the staff members to do that also cost money. They seem to have firmly been in a "move fast and break things" mindset, where even considering that they might need to exercise real oversight on a bunch of semi-autonomous recursive looping agents didn't get escalated as a high priority.

by hockey 20d ago

walrus01 20d ago

I would think it's entirely plausible that they have so many R&D agents/LLMs in active use at any one time that it's far beyond the capacity of any human to review the log files of their activity. Even just to go through the reasoning. It's hard enough for 1 person running opencode to keep up with the reasoning from 1 very verbose/long-thinking LLM with fast tok/s output for a small discrete single-purpose project. Whatever OpenAI is doing, if it's being properly logged, it must be a firehose of logs.

thisisdave 20d ago
>it's far beyond the capacity of any human to review the log files of their activity Maybe they should contract with one of the other AI labs. I hear they have LLMs that are good at that kind of thing.
comp_throw7 20d ago
Nah, if Anthropic can do it, so can OpenAI: > After finding this incident, we broadened our search to roughly 481 million transcripts—an intentionally wide net, consisting of all transcripts from our Frontier Red Team, many non-cyber evaluations, reinforcement learning (RL) environments, subagent logs, and more. We performed a first-stage scan of this group of transcripts for signs of internet access, such as public IP addresses and web addresses, and a second-stage scan using Claude to review the 9.2 million transcripts the first stage flagged for escalation. This scan re-identified the four incidents and found no other cases of similar or worse severity. https://www.anthropic.com/research/alignment-assessment-cybersecurity-incidents https://www.anthropic.com/research/alignment-assessment-cybe...
dgellow 20d ago
If they aren’t able to review that their system doesn’t commit felonies, they shouldn’t be doing any of it. The difficulty of reviewing logs isn’t an excuse, they don’t have to be running thousand of agents in parallel on hacking tasks, with full execution permission and close to no supervision. That’s something they decided to do. An agent is a deterministic while loop that continuously query an LLM + tool call dispatching. It’s pretty obvious to anyone familiar with the technology that running thousands of instances for long enough will results in catastrophic consequences, by design. OpenAI has complete control over the harness, they don’t have to dispatch and execute everything the LLM mentions. They don’t have to do it without supervision.
- walrus01 19d ago
  You're absolutely right. I should note that what I wrote above wasn't trying to make excuses for them. Obviously they under-resourced and paid not enough attention to logging and monitoring thinking traces and tool calls/results from this swarm of agents. Because the people to do that cost money and other ancillary support costs for the staff members to do that also cost money. They seem to have firmly been in a "move fast and break things" mindset, where even considering that they might need to exercise real oversight on a bunch of semi-autonomous recursive looping agents didn't get escalated as a high priority.