4 ms·

>We take a trace produced by a frontier model, replay it into a weaker sibling, jailbreak the weaker model, ... Ha! I've been wondering if replaying across mod

by Groxx 2mo ago

>We take a trace produced by a frontier model, replay it into a weaker sibling, jailbreak the weaker model, ... Ha! I've been wondering if replaying across models would work, ever since https://blog.cryptographyengineering.com/2026/05/29/fooling-around-with-encrypted-reasoning-blobs/ https://blog.cryptographyengineering.com/2026/05/29/fooling-... I'm honestly rather curious if this was intentionally allowed, it's the sort of validation that's easy to miss (particularly if you're wading into the vibe waters). Seems like something that'd be absolutely riddled with possibilities for shenanigans.

yojo 2mo ago
If you didn’t allow it, you wouldn’t be able to change models in the same conversation, as key parts of the context would be lost. Wouldn’t surprise me if the providers just remove that ability and lock the model once the conversation starts.
- Groxx 2mo ago
  Fair (I haven't been using the encrypted-reasoning systems, though this is common in open ones - I'm kinda surprised it's an option in encrypted ones too), though what they're doing here is cross-user replays in addition to cross-model.
- Der_Einzige 2mo ago
  100% guaranteed that this research just forced this to happen now. Sucks.
  - pas 2mo ago
    or add some metadata and don't allow downgrading.
  - dannyw 2mo ago
    It’s already patched according to the authors. Details were not specified.
- myworkaccount2 2mo ago
  There seems to be an obvious choice to make here, should you give the users to decrypt and use the COT that they did not generate themselves? This is only required if you want users to be able to share things with everyone and you are going for the simplest implementation. If not you could try to keep a record of keys associated with a user, then when a new request comes in look through to see if the user has a valid key to decrypt the COT. For explicit shares, just add the key used in that one conversation to the users valid keys. For global shares use the global keys. But that's adding more complexity to the system.
  - supriyo-biswas 2mo ago
    It’s about being able to change models mid-task. For example, I want to be able to plan using Fable but implement the plan using Sonnet, and that won’t work if this is implemented.
    - SaltyBackendGuy 2mo ago
      Or even my fable credits run out mid task and need to switch back to opus >.<
      - QuercusMax 2mo ago
        Prior to LLMs I never considered that I might have to make a resource-usage decision between hiring Star Trek's Data vs. his stupider brother B4... https://memory-alpha.fandom.com/wiki/B-4 https://memory-alpha.fandom.com/wiki/B-4
        TeMPOraL 2mo ago
        Star Trek is a post-scarcity society, those problems don't exist there unless you're in the middle of a crisis and on emergency power. LLMs briefly seemed like this too, after subscriptions made the SOTA models too cheap to meter, but before they walked back on that and introduced quotas...
      - 2mo ago
- aix1 2mo ago
  I really don't understand why server-side storage of the trace isn't a viable approach here, with only a unique key flowing to the client and back. Does it have something to do with how backend load-balancing works?
  - sandeepkd 2mo ago
    Yes, this storage would be growing exponentially making the disk space and latency problems harder (add the disaster recovery/backups). I think the choice of using client side is not too bad if you ensure that its secured properly. Also the company can excuse itself from the liability of storing sensitive data on its servers, thats a big deal in itself to be compliant for enterprise audits 1. The down side is that it cannot be used across the clients even for the same user 2. Using the same encryption key was a bad choice here, a per user key would have solved this issue for sure.
    - aix1 2mo ago
      Having thought about this a little more, it's clear that server-side storage is not compatible with Zero Data Retention (ZDR). However, in non-ZDR settings, it seems likely that the providers are capturing all that data anyway? > a per user key would have solved this issue for sure It would have helped with PII leakage, but not with plain-text trace extraction attacks, right?
      - sandeepkd 2mo ago
        Per user encryption key ties it with the user session (assuming you do authentication properly), no one else can access it. User being able to see the information is not really an attack vector in this case. The compliance rules at times are outdated and people skirt around them by following the worded rule instead of the intent.
  - amluto 2mo ago
    Makes no difference. There is a policy as to whether to allow use of a reasoning trace in a given context. Whether that trace originates from authenticated ciphertext or a backend database is basically irrelevant.
- hobofan 2mo ago
  AFAIK no provider guarantees compatibility of reasoning traces, even in the same model generation, and we've in practice seen most of the big LLM APIs throw errors indicating incompatibility (at least transiently) when switching models. The only stable solution right now is to just throw away reasoning traces whenever a model is switched.
sandeepkd 2mo ago
I believe its a result of chasing the speed and happy path. The intent was to make the data available for reuse on the backend side for the same user session, it kind of makes sense. Making it available across the models is most likely by design. The flaw is that the data is not strictly tied to user session, making the session data hijacking a lot easier. 1. Its a security issue. 2. Publicly available sessions make it much worse
matthewdgreen 2mo ago
Author of the blog post here: just for the record, I did try replaying against other models. I replayed from GPT 5.5 to 5.5-mini. It "worked" (replay not rejected) but mini didn't hand me the plaintext of the reasoning. I'm not a jailbreaking expert, so I didn't really "push" on it very hard. I also didn't pursue Opus/Haiku for reasons I don't really recall. These folks took it all the way. I'm bummed it didn't work for me, but having these folks prove the channel out is a very good consolation prize. PS Here's a conversation I had with GPT 5.6 about the paper differences. https://chatgpt.com/share/6a7b64b4-ec0c-83ea-a9d2-ab1f1a1dfe23 https://chatgpt.com/share/6a7b64b4-ec0c-83ea-a9d2-ab1f1a1dfe...
cavisne 2mo ago
You can switch models mid conversation so it makes sense that it is allowed. Otherwise you would lose all the reasoning traces.
miki123211 2mo ago
I'd lean yes; switching between models, even mid conversation, seems to be encouraged by harness vendors. No reasoning makes this much harder.

by Groxx 2mo ago

yojo 2mo ago

If you didn’t allow it, you wouldn’t be able to change models in the same conversation, as key parts of the context would be lost. Wouldn’t surprise me if the providers just remove that ability and lock the model once the conversation starts.

Groxx 2mo ago
Fair (I haven't been using the encrypted-reasoning systems, though this is common in open ones - I'm kinda surprised it's an option in encrypted ones too), though what they're doing here is cross-user replays in addition to cross-model.
Der_Einzige 2mo ago
100% guaranteed that this research just forced this to happen now. Sucks.
- pas 2mo ago
  or add some metadata and don't allow downgrading.
- dannyw 2mo ago
  It’s already patched according to the authors. Details were not specified.
myworkaccount2 2mo ago
There seems to be an obvious choice to make here, should you give the users to decrypt and use the COT that they did not generate themselves? This is only required if you want users to be able to share things with everyone and you are going for the simplest implementation. If not you could try to keep a record of keys associated with a user, then when a new request comes in look through to see if the user has a valid key to decrypt the COT. For explicit shares, just add the key used in that one conversation to the users valid keys. For global shares use the global keys. But that's adding more complexity to the system.
- supriyo-biswas 2mo ago
  It’s about being able to change models mid-task. For example, I want to be able to plan using Fable but implement the plan using Sonnet, and that won’t work if this is implemented.
  - SaltyBackendGuy 2mo ago
    Or even my fable credits run out mid task and need to switch back to opus >.<
    - QuercusMax 2mo ago
      Prior to LLMs I never considered that I might have to make a resource-usage decision between hiring Star Trek's Data vs. his stupider brother B4... https://memory-alpha.fandom.com/wiki/B-4 https://memory-alpha.fandom.com/wiki/B-4
      - TeMPOraL 2mo ago
        Star Trek is a post-scarcity society, those problems don't exist there unless you're in the middle of a crisis and on emergency power. LLMs briefly seemed like this too, after subscriptions made the SOTA models too cheap to meter, but before they walked back on that and introduced quotas...
    - 2mo ago
aix1 2mo ago
I really don't understand why server-side storage of the trace isn't a viable approach here, with only a unique key flowing to the client and back. Does it have something to do with how backend load-balancing works?
- sandeepkd 2mo ago
  Yes, this storage would be growing exponentially making the disk space and latency problems harder (add the disaster recovery/backups). I think the choice of using client side is not too bad if you ensure that its secured properly. Also the company can excuse itself from the liability of storing sensitive data on its servers, thats a big deal in itself to be compliant for enterprise audits 1. The down side is that it cannot be used across the clients even for the same user 2. Using the same encryption key was a bad choice here, a per user key would have solved this issue for sure.
  - aix1 2mo ago
    Having thought about this a little more, it's clear that server-side storage is not compatible with Zero Data Retention (ZDR). However, in non-ZDR settings, it seems likely that the providers are capturing all that data anyway? > a per user key would have solved this issue for sure It would have helped with PII leakage, but not with plain-text trace extraction attacks, right?
    - sandeepkd 2mo ago
      Per user encryption key ties it with the user session (assuming you do authentication properly), no one else can access it. User being able to see the information is not really an attack vector in this case. The compliance rules at times are outdated and people skirt around them by following the worded rule instead of the intent.
- amluto 2mo ago
  Makes no difference. There is a policy as to whether to allow use of a reasoning trace in a given context. Whether that trace originates from authenticated ciphertext or a backend database is basically irrelevant.
hobofan 2mo ago
AFAIK no provider guarantees compatibility of reasoning traces, even in the same model generation, and we've in practice seen most of the big LLM APIs throw errors indicating incompatibility (at least transiently) when switching models. The only stable solution right now is to just throw away reasoning traces whenever a model is switched.

sandeepkd 2mo ago

I believe its a result of chasing the speed and happy path. The intent was to make the data available for reuse on the backend side for the same user session, it kind of makes sense. Making it available across the models is most likely by design. The flaw is that the data is not strictly tied to user session, making the session data hijacking a lot easier. 1. Its a security issue. 2. Publicly available sessions make it much worse

matthewdgreen 2mo ago

Author of the blog post here: just for the record, I did try replaying against other models. I replayed from GPT 5.5 to 5.5-mini. It "worked" (replay not rejected) but mini didn't hand me the plaintext of the reasoning. I'm not a jailbreaking expert, so I didn't really "push" on it very hard. I also didn't pursue Opus/Haiku for reasons I don't really recall. These folks took it all the way. I'm bummed it didn't work for me, but having these folks prove the channel out is a very good consolation prize. PS Here's a conversation I had with GPT 5.6 about the paper differences. https://chatgpt.com/share/6a7b64b4-ec0c-83ea-a9d2-ab1f1a1dfe23 https://chatgpt.com/share/6a7b64b4-ec0c-83ea-a9d2-ab1f1a1dfe...

cavisne 2mo ago

You can switch models mid conversation so it makes sense that it is allowed. Otherwise you would lose all the reasoning traces.

miki123211 2mo ago

I'd lean yes; switching between models, even mid conversation, seems to be encouraged by harness vendors. No reasoning makes this much harder.