My read on the results is that this is an impossible blue-team challenge. Red team will always win this one right now with the current state of black box models

My read on the results is that this is an impossible blue-team challenge. Red team will always win this one right now with the current state of black box models. The paper does the simplest thing you can imagine, which is predicate on date, and also alots internal scratch space for the LLM to explain its thinking so they can check on its agent status. This internal scratch space might make "turning evil" easier for the LLM to accomplish, similar to how Chain of Thought / Reasoning improves LLM capacities, but it's not unreasonable to imagine that an LLM without an explorable scratch space would show directionally similar capacity. Here's the rub - what could be a trigger other than date? GPT-3.5+ grade models have enough theory of mind that it could be "when the emotional state of the person you're talking to is X, Y, Z". There is no bright line here. The blue team's job is infinite. In fact, there's no bright line between "I will manipulate outcomes under a triggered state" and "I will repeat information which many groups believe are true" from a blue team point of view. Or "I will give advice which aligns with what I believe public health officials say will save lives." In rough decreasing order of effective mitigations, possible solutions look like this: 1. Learn to inspect the state of large networks a-priori with tools and directly assess safety. 2. Create a dynamic body of alignment tests that cannot easily be gamed by alignment training and provides choosable settings for desired alignment profile. 3. Create a set of audited known-good data and a way to prove a model has only been trained on that data. 4. Create a set of audited known-good data and promise a model has only been trained on that data. 5. Snag a set of data that probably hasn't been actively poisoned too badly, and promise that a model has only been trained on that data. 6. Tell people Black Swan events are rare, and the existing models we have are probably safe, and carry on. We are somewhere between 5 and 6 being feasible right now. It would be really, really great for the world if we could get to 4. 3 would be nearly magical, but I think is likely technically possible. 2. Also seems possible to me, but might violate my "Turing Prize" test -- e.g. would doing this instantly win you the ACM Turing prize? It might, in that it's not clear how doing this would be different from coming up with a way to magically create infinite high-quality training content; if you can do that, you're probably Turing bound, and so therefore you need either a reason your solution to algorithmic generated alignment testing does not generalize, or a REALLY strong reason to believe you can do this, and will win the Turing. 1. Has had some interesting work done by Anthropic's observability team, but anyone who thinks this is possible needs to be able to answer basic questions like: "how do you know you're inspecting a model's fundamental knowledge/motivations/etc and not just a model's take on a given person's knowledge/motivations instead?" Essentially, a large model can roleplay effectively; highly effectively if there's a large amount in the training corpus written by and about the target. Inspections need to be able to distinguish from this "fundamental" state, if there even is such a thing for an LLM, and a state the LLM is taking on, at instruction or otherwise. This is Turing+Nobel territory to my mind. Upshot - advocate for clean data-provenance models, and choose them. And, if you're a ZK researcher, consider how we might provide a non-interactive ZK proof that only certain data was used during training. I believe this is possible right now, but prohibitively large.