3 ms·

Using a throwaway account for obvious reasons, but I’m very involved in this space using LLMs from multiple providers. I’m aware of at least two instances in wh

Using a throwaway account for obvious reasons, but I’m very involved in this space using LLMs from multiple providers. I’m aware of at least two instances in which the intermediate infrastructure “swapped” responses, once impacting Claude models and once impacting GPT models, from two different providers. One gave us a proper postmortem in which their API gateway was incorrectly handling HTTP 100 status codes, putting them into an error state where there was effectively an off by one error - you would receive the response to the prompt that came in before yours and would pay it forward (your response would go to the next caller). The other instance never had root cause explained to us, and we were just told to trust it wouldn’t happen again. Both of these are from $1T+ companies. ZDR wasn’t compromised in these cases since it was responses being swapped in flight. I wouldn’t be surprised if this is a similar issue - it’s not that data is being retained, it’s just not being safely isolated in intermediate infrastructure.

theplumber 3mo ago
[flagged]
- minhaz23 3mo ago
  Curious why you feel that way about Dario?
  - solenoid0937 3mo ago
    HN thinks the safety crowd is dumb, and has never seriously engaged with the AI safety space. HN doesn't believe superintelligence will be a thing; while the AI safety crowd believes they are building it. So the decisionmaking of the safety crowd is incomprehensible to HN.
    - DrewADesign 3mo ago
      Reductionist. Many of us think they’re all dumb.
    - pseudony 3mo ago
      [flagged]
      - SubiculumCode 3mo ago
        There is no reason for you to make personal attacks like that. Not on HN. Moreover, your take on Dario is over simplistic, and undersells the extent to which Anthropic takes seriously safety. It's not lip service, there are real dollars and attention spent on alignment at Anthropic.
      - solenoid0937 3mo ago
        It's clear you haven't engaged with the subject matter beyond the typical "internet-forum cynic" mindset. Both companies were founded on the basis of AI Safety. - There are tons of great safety people doing real work at OpenAI. Releases are held back, models are evaluated, etc. - Anthropic goes even further - constrained themselves with a PBC/LTBT structure, treat safety even more rigorously, and notably delayed the release of Mythos (literally the opposite of what you alleged) and continue to hold their two red lines despite threats from the gov. You should actually talk to some of the people at these labs. Nearly everyone working at these places genuinely believe AGI/ASI is actually happening, so they do take safety seriously. To imply these companies don't care about safety is typical internet-brand nihilism/cynicism that helps you feel smart while being literally the opposite of the truth.
        cyral 3mo ago
        To add to this, they should look at the Fable system card. It's 317 pages and it's clear how serious they are taking AI safety.
        superb_dev 3mo ago
        Page count is not a measure of how seriously something is being taken when you can easily generate pages and pages of slop
        3mo ago
    - rvba 3mo ago
      What is the AI safety crowd exactly? Dont we have a thread here how the model allegedly leaks responses what is "normal" safety? (Not "agi will become skynet" safety - what is mostly a rehash of terminator 2 story)
      - nozzlegear 3mo ago
        According to their comment history, the person you're replying to believes that AGI, ASI, "superintelligence" and all of that sci-fi terminator what-have-you is not only possible, but literally inevitable. They're not worried about normal safety, they're worried about Skynet and the T-1000. American AI labs – Anthropic in particular – are apparently playing the role of Sara and/or John Connor in this story.
    - root_axis 3mo ago
      > the AI safety crowd believes they are building it Stated without a hint of irony.
  - politician 3mo ago
    Dario quit OpenAI to hype the AI apocalypse for quick cash and attention. Then, he walked right into an obvious crisis with the Pentagon by continuing to try to play both sides of the AGI doom story that even his own AI would've pointed out. Then, after being labelled a supply chain risk, he starts a new roadshow with the newest most dangerous AI model that definitely cannot be released to the public and its safer little brother Fable. A move that gets both his premier models shut down globally once the same government that labelled them a supply chain risk learns that Fable isn't actually safe from jailbreaks. Just prior to his planned IPO. Dario might not be a literal idiot, but he might strongly benefit from training a model to do strategic thinking for Anthropic.
    - throwatdem12311 3mo ago
      All of these things have people frothing at the mouth to give up all their data to Anthropic to use their models and to buy in when the IPO eventually happens. Seems to me Dario is actually a genius. These are all things that I would to make people believe that my “basically the same as the other guy” product is ackshually best thing ever for real. Trust me bro. The entire bubble is hype and fear mongering. The technical merits of the products are completely irrelevant at this point. Dario is doing exactly what someone that understands this would do and they are winning.
pocksuppet 3mo ago
This attack is called "HTTP desync" or "request smuggling". It's often done intentionally by a client to try and spy on other clients' responses. Every time you multiplex requests from multiple clients onto one upstream connection, you are probably vulnerable to this, because (despite its superficial simplicity) HTTP is just too complex to reliably match the requests and responses to upstream. For example a desync can be triggered in some systems by having more than one Content-Length header, by mixing Content-Length with chunked encoding, or by passing an HTTP/2 header called Content-Length that doesn't match the actual content length. Here's a DEF CON talk (6 years ago) on this topic: https://www.youtube.com/watch?v=w-eJM2Pc0KI https://www.youtube.com/watch?v=w-eJM2Pc0KI The same attack has been applied to SMTP by messing up the line endings surrounding the end-of-message delimiter, where it's called SMTP smuggling. It may also apply to other protocols.
- markasoftware 3mo ago
  Very true, this was likely an attack. Worth noting that mr kettle has done a defcon talk nearly every year on some variant of this attack, the most recent one titled "HTTP/1.1 must die" because he rightfully believes that switching to the binary headers of http/2 (specifically in reverse proxy connections to upstream servers) is the only way to systematically prevent these.
  - albinowax_ 3mo ago
    I’ll be back next month with a load of fresh vectors in “Can AI Do Novel Security Research? Meet the HTTP Terminator” https://portswigger.net/research/talks?talkId=36 https://portswigger.net/research/talks?talkId=36 Maybe my last presentation on the topic! Possibly.
    - bostik 3mo ago
      Or as the Risky Business guys crystallise it: "James Kettle breaks the internet. Again."
    - pocksuppet 3mo ago
tejusarora 3mo ago
Woah. Sounds plausible. However, wouldn’t that still be an implicit violation of ZDR since now the response is possibly egressed out of the enterprise network? So if I were working with PHI, the response egress is a potential violation of HIPAA even though claude didn’t retain anything — but the whole Point was to comply with HIPAA. Thoughts?
rsync 3mo ago
Actually, it’s not obvious why you’re using a throwaway account… Every emergent behavior from these actors - whose claim to positive moral values is barely plausible - should be reported, discussed, dissected and critiqued early and often.
- DANmode 3mo ago
  Your points don’t reference each other. Yes, the discussion should be had constantly. But: Should this person potentially have their life messed up because they pointed out the emperor has no clothes?
- zymhan 3mo ago
  >should be reported, discussed, dissected and critiqued early and often. And why does anonymity detract from any of that?
- throwaway260704 3mo ago
  I like being gainfully employed, and the best position for me to push for genuine AI safety is within an influential company in the space.
nightpool 3mo ago
I hope you reconsider your use of a throwaway account for this sort of comment—This sort of feedback and experience is very valuable and if the culture of the company you're in or the companies your working with discourage this sort of feedback, it's only going to create chilling effect that prevents more people from coming forward
- throwaway260704 3mo ago
  Would prefer to stick to a throwaway, sorry. It’s not that such feedback is discouraged by my company, it’s been very much escalated. However the response to these incidents had legal repercussions and my replies here aren’t subject to attorney-client privilege. While whistleblower protections (at least used to) safeguard against government persecution, I work for a private company and don’t want my livelihood risked. I did what I could to escalate through official channels.

theplumber 3mo ago

[flagged]

minhaz23 3mo ago
Curious why you feel that way about Dario?
- solenoid0937 3mo ago
  HN thinks the safety crowd is dumb, and has never seriously engaged with the AI safety space. HN doesn't believe superintelligence will be a thing; while the AI safety crowd believes they are building it. So the decisionmaking of the safety crowd is incomprehensible to HN.
  - DrewADesign 3mo ago
    Reductionist. Many of us think they’re all dumb.
  - pseudony 3mo ago
    [flagged]
    - SubiculumCode 3mo ago
      There is no reason for you to make personal attacks like that. Not on HN. Moreover, your take on Dario is over simplistic, and undersells the extent to which Anthropic takes seriously safety. It's not lip service, there are real dollars and attention spent on alignment at Anthropic.
    - solenoid0937 3mo ago
      It's clear you haven't engaged with the subject matter beyond the typical "internet-forum cynic" mindset. Both companies were founded on the basis of AI Safety. - There are tons of great safety people doing real work at OpenAI. Releases are held back, models are evaluated, etc. - Anthropic goes even further - constrained themselves with a PBC/LTBT structure, treat safety even more rigorously, and notably delayed the release of Mythos (literally the opposite of what you alleged) and continue to hold their two red lines despite threats from the gov. You should actually talk to some of the people at these labs. Nearly everyone working at these places genuinely believe AGI/ASI is actually happening, so they do take safety seriously. To imply these companies don't care about safety is typical internet-brand nihilism/cynicism that helps you feel smart while being literally the opposite of the truth.
      - cyral 3mo ago
        To add to this, they should look at the Fable system card. It's 317 pages and it's clear how serious they are taking AI safety.
        superb_dev 3mo ago
        Page count is not a measure of how seriously something is being taken when you can easily generate pages and pages of slop
        3mo ago
  - rvba 3mo ago
    What is the AI safety crowd exactly? Dont we have a thread here how the model allegedly leaks responses what is "normal" safety? (Not "agi will become skynet" safety - what is mostly a rehash of terminator 2 story)
    - nozzlegear 3mo ago
      According to their comment history, the person you're replying to believes that AGI, ASI, "superintelligence" and all of that sci-fi terminator what-have-you is not only possible, but literally inevitable. They're not worried about normal safety, they're worried about Skynet and the T-1000. American AI labs – Anthropic in particular – are apparently playing the role of Sara and/or John Connor in this story.
  - root_axis 3mo ago
    > the AI safety crowd believes they are building it Stated without a hint of irony.
- politician 3mo ago
  Dario quit OpenAI to hype the AI apocalypse for quick cash and attention. Then, he walked right into an obvious crisis with the Pentagon by continuing to try to play both sides of the AGI doom story that even his own AI would've pointed out. Then, after being labelled a supply chain risk, he starts a new roadshow with the newest most dangerous AI model that definitely cannot be released to the public and its safer little brother Fable. A move that gets both his premier models shut down globally once the same government that labelled them a supply chain risk learns that Fable isn't actually safe from jailbreaks. Just prior to his planned IPO. Dario might not be a literal idiot, but he might strongly benefit from training a model to do strategic thinking for Anthropic.
  - throwatdem12311 3mo ago
    All of these things have people frothing at the mouth to give up all their data to Anthropic to use their models and to buy in when the IPO eventually happens. Seems to me Dario is actually a genius. These are all things that I would to make people believe that my “basically the same as the other guy” product is ackshually best thing ever for real. Trust me bro. The entire bubble is hype and fear mongering. The technical merits of the products are completely irrelevant at this point. Dario is doing exactly what someone that understands this would do and they are winning.

pocksuppet 3mo ago

This attack is called "HTTP desync" or "request smuggling". It's often done intentionally by a client to try and spy on other clients' responses. Every time you multiplex requests from multiple clients onto one upstream connection, you are probably vulnerable to this, because (despite its superficial simplicity) HTTP is just too complex to reliably match the requests and responses to upstream. For example a desync can be triggered in some systems by having more than one Content-Length header, by mixing Content-Length with chunked encoding, or by passing an HTTP/2 header called Content-Length that doesn't match the actual content length. Here's a DEF CON talk (6 years ago) on this topic: https://www.youtube.com/watch?v=w-eJM2Pc0KI https://www.youtube.com/watch?v=w-eJM2Pc0KI The same attack has been applied to SMTP by messing up the line endings surrounding the end-of-message delimiter, where it's called SMTP smuggling. It may also apply to other protocols.

markasoftware 3mo ago
Very true, this was likely an attack. Worth noting that mr kettle has done a defcon talk nearly every year on some variant of this attack, the most recent one titled "HTTP/1.1 must die" because he rightfully believes that switching to the binary headers of http/2 (specifically in reverse proxy connections to upstream servers) is the only way to systematically prevent these.
- albinowax_ 3mo ago
  I’ll be back next month with a load of fresh vectors in “Can AI Do Novel Security Research? Meet the HTTP Terminator” https://portswigger.net/research/talks?talkId=36 https://portswigger.net/research/talks?talkId=36 Maybe my last presentation on the topic! Possibly.
  - bostik 3mo ago
    Or as the Risky Business guys crystallise it: "James Kettle breaks the internet. Again."
  - pocksuppet 3mo ago

tejusarora 3mo ago

Woah. Sounds plausible. However, wouldn’t that still be an implicit violation of ZDR since now the response is possibly egressed out of the enterprise network? So if I were working with PHI, the response egress is a potential violation of HIPAA even though claude didn’t retain anything — but the whole Point was to comply with HIPAA. Thoughts?

rsync 3mo ago

Actually, it’s not obvious why you’re using a throwaway account… Every emergent behavior from these actors - whose claim to positive moral values is barely plausible - should be reported, discussed, dissected and critiqued early and often.

DANmode 3mo ago
Your points don’t reference each other. Yes, the discussion should be had constantly. But: Should this person potentially have their life messed up because they pointed out the emperor has no clothes?
zymhan 3mo ago
>should be reported, discussed, dissected and critiqued early and often. And why does anonymity detract from any of that?
throwaway260704 3mo ago
I like being gainfully employed, and the best position for me to push for genuine AI safety is within an influential company in the space.

nightpool 3mo ago

I hope you reconsider your use of a throwaway account for this sort of comment—This sort of feedback and experience is very valuable and if the culture of the company you're in or the companies your working with discourage this sort of feedback, it's only going to create chilling effect that prevents more people from coming forward

throwaway260704 3mo ago
Would prefer to stick to a throwaway, sorry. It’s not that such feedback is discouraged by my company, it’s been very much escalated. However the response to these incidents had legal repercussions and my replies here aren’t subject to attorney-client privilege. While whistleblower protections (at least used to) safeguard against government persecution, I work for a private company and don’t want my livelihood risked. I did what I could to escalate through official channels.