6 ms·

It's insane the AI has been provided the tooling to send emails to arbitrary addresses like that. Like, getting it to send a 2FA code at a user's request is one

by hbn 5mo ago

It's insane the AI has been provided the tooling to send emails to arbitrary addresses like that. Like, getting it to send a 2FA code at a user's request is one thing. But it should only be able to "hit a button" to send a 2FA email to the address attached to the account, all run with hand-written code. It shouldn't have access to the 2FA code itself, or the message subject, or body, or the recipient address, etc. Why did they give it any of that?!

AlienRobot 5mo ago
The harness is vibe-coded.
footydude 5mo ago
> But it should only be able to "hit a button" to send a 2FA email to the address attached to the account, all run with hand-written code. Genuine question...why would that need to be hand-written? It makes absolute sense as a general statement and is kinda crazy that this wasn't a built-in limitation, but I'm not quite sure why the code for that bit must be hand-written (provided the code functionally does what you describe).
- andrewstuart2 5mo ago
  Maybe not hand-written, but definitely static, and at least human-reviewed/tested to only allow sending to previously-validated email addresses.
  - daheza 5mo ago
    Right, as in, does not accept an email as a parameter. If its anything like my company they are turning out "agents" super fast and just hooking them up to internal APIs usually via a light MCP wrapper. Since MCP doesn't have any security or auth built in, and internal APIs usually are light on security you have issues like this.
- mediaman 5mo ago
  I think he likely means "code that is hand-reviewed" and not directly controlled by the agent. He's probably meaning to differentiate it against the in-process agent writing the code. It doesn't matter too much if that fixed code was written by an LLM under guidance and review of the SWE, outside the agent.
  - Barbing 5mo ago
    Agreed, “literally written by hand” didn’t cross my mind. Not by keyboard or pen.
  - footydude 5mo ago
    Ahh ok - that's fair enough - hand-reviewed/not controlled by the agent seems a sensible approach (wasn't sure if it was instructive of a complete distrust of AI generated code)
dpark 5mo ago
This exploit has essentially nothing to do with AI and everything to do with a terribly designed account recovery flow. This exact same flow could have been (and may have been; I don’t know how much the chatbot here actually does) statically coded.
- deleted 5mo ago
  [deleted]
- aidenn0 5mo ago
  My impression is that AI didn't replace static code in this place; it replaced a person, who (hopefully) would have been suspicious about sending an account recovery code for e.g. "obamawhitehouse" to e.g. "bscurtu.alfamm.ro@gmail.com"
  - soerxpso 5mo ago
    You're giving a lot of credit to the human alternative, especially considering that the attacker only needs to find one lazy human.
    - kelipso 5mo ago
      Come on, this attack vector would have been flagged by at least one person and you won’t then have multiple accounts hacked because of it. AI reacts fairly predictably to a single attack vector and don’t learn unless it gets flagged and then taught.
      - devmor 5mo ago
        And even if a human didn’t catch it in one case, they will frequently. Giving AI access to the same tools humans use without any oversight mechanism just amplifies the harm and carelessness possible by one person.
    - afavour 5mo ago
      Still makes this exponentially worse, no? It works every time and it's automated so scales up as quickly as you're able to request it.
nashashmi 5mo ago
Some Jr engineer got tired of handling stupid support requests and automated the job with an agent. That’s how. Assigning Jr engineers for security support is ridiculous partly because young people don’t understand how critical security is sometimes. And partly because they don’t value privacy as much.
- kennywinker 5mo ago
  Very generous of you to blame the screw up of one of the largest companies in the world on a jr engineer. I’ve been a jr engineer at a large company. I had the power to implement absolutely jack shit on my own. I deeply doubt the security flow for account recovery in meta ai account security was a single jr engineer. What i think is actually going on is basically a soft form of ai psychosis. Senior engineer gets ai to code ai account recovery feature, that same or a different engineer asks ai to review the feature, and then it gets pushed to prod. Move fast, break things. The ai coded it, the ai reviewed it - the people trusted the ai because it sounds confidently right. Just like how the ai doesn’t know if you should walk or drive to the car wash, the ai doesn’t understand exploits like this one.
- parable 5mo ago
  As a "young person" (under 30), my thoughts: There's a minority of us that do genuinely care, possibly more than most - so hiring someone from this minority would be helpful - but the vast majority of my peers don't care about privacy nor security. They often take this defeatist mindset of "my data is already out there, why should I care?", or prefer convenience over security. For example, "why should I switch to Signal if I have a public Instagram profile?" or "I can't remember all those passwords! I just use one for everything." As for your comment about junior engineers, see kennywinker's reply to this thread - I share the same thoughts.
- acdha 5mo ago
  If a single junior engineer can do this, it’s an even bigger indictment of Facebook’s senior management than this exploit. A well-designed system doesn’t rely on individuals never making mistakes and if our hypothetical junior developer can make critical security policy changes without oversight, that should be a C-level job loss event. If our goal isn’t to make excuses for the top of the org chart, a more likely explanation is that senior management is heavily incentivizing shipping AI features and this went out as a high-impact change reviewed in a rush, probably by AI.
plagiarist 5mo ago
This exploit is my new gold standard for trivially avoidable security failures. Someone has finally beaten Gitlab's password reset emails to attacker-provided addresses.
brianmcnulty 5mo ago
I do a lot of bug bounty research on Meta and Instagram, and some of the bugs I find look extremely simple like this but have some slightly complicated reason for why they occur. Maybe not this one, but I do have a guess as to what might have actually happened. Based on what I've seen so far, Meta AI Support Assistant (they call it "MAISA") had tool calls that a) start an email verification to any specific email, phone number, or the contact points linked to an account and b) allow generating a password reset link for an account based on an email verification attempt. I don't think it had any access to the actual codes themselves, but rather think a handle or ID for an email verification attempt (along with the user provided verification code based on user input) was provided to the "generate reset password link" tool call, and the tool call failed to properly validate the actual email used in that attempt belonged to the account allowing the ATO. The tool call for MAISA to generate a password reset link should have failed with an email verification attempt that corresponds to an email not linked to the account (and I believe I even tested this at one point on Facebook and encountered an error that successfully prevented it), but I suspect they tried making a change to this tool call for Instagram where slightly older, recently unlinked emails could be used to recover an account that got hijacked by an attacker, which added the need to allow emails not currently linked to the account to be used and set to the user's primary email. I also suspect that the MAISA tool call change called a wrong API or something that unintentionally allowed any email verification attempt that was successful to be used, but the engineers did not add a sufficiently thorough e2e test case to test the tool call against unrelated email verification attempts being provided to the tool call. This is the part I think should be focused on the most. Tool calls for agents that have their output potentially influenced by an attacker should be treated like external APIs that anyone can reach, and they should be tested as such. This is all obviously a guess, doesn't take into account the many signals they use to determine if an account recovery attempt is valid, and could be very inaccurate, but it's the closest to what I (someone who deals with Meta security a lot) think could have allowed this to happen.
- muglug 5mo ago
  Seems like the most plausible explanation. OTOH it feels like this is the sort of thing that might have been discovered/mitigated more quickly had there been a human in the loop.
guestbest 5mo ago
One would have to assume that this was by design.
drtz 5mo ago
Yeah it's bad, but AI isn't required for this type of thing to work. My anecdotal experience is my Facebook account was compromised several years ago after TOTP 2FA was disabled. Didn't exactly give me a warm fuzzy about Facebook security policies at the time, and this new attack just reaffirms that.
taco_emoji 5mo ago
This reeks of vibe coding. "Make it so the AI agent can help with password resets" and then zero human vetting of the change.
- chuckadams 5mo ago
  The human vetting was that it was cheaper. Someone probably got promoted for it.
- mannanj 5mo ago
  And zero accountability too. No one will be found and detected.
smrtinsert 5mo ago
It's stuff like this that honestly makes it very hard for me to take anyone working at Meta seriously. How much communication had to happen to enable this feature? It really casts doubt across the organization at multiple levels, don't tell me a single engineer caused this.
- queenkjuul 5mo ago
  I can't take Meta seriously, period.
mjmsmith 5mo ago
If this exploit has nothing to do with AI, why haven't we heard about it succeeding before? I find it hard to believe it's never been tried.
pif 5mo ago
> Why did they give it any of that?! Because they are idiots. You need to be a freaking idiit to trust AI.
dannyw 4mo ago
No big tech company hands write code by default now. You’d get PIP’d very quickly when your manager notices your PRs don’t have AI as a co-committer.

by hbn 5mo ago

AlienRobot 5mo ago

The harness is vibe-coded.

footydude 5mo ago

> But it should only be able to "hit a button" to send a 2FA email to the address attached to the account, all run with hand-written code. Genuine question...why would that need to be hand-written? It makes absolute sense as a general statement and is kinda crazy that this wasn't a built-in limitation, but I'm not quite sure why the code for that bit must be hand-written (provided the code functionally does what you describe).

andrewstuart2 5mo ago
Maybe not hand-written, but definitely static, and at least human-reviewed/tested to only allow sending to previously-validated email addresses.
- daheza 5mo ago
  Right, as in, does not accept an email as a parameter. If its anything like my company they are turning out "agents" super fast and just hooking them up to internal APIs usually via a light MCP wrapper. Since MCP doesn't have any security or auth built in, and internal APIs usually are light on security you have issues like this.
mediaman 5mo ago
I think he likely means "code that is hand-reviewed" and not directly controlled by the agent. He's probably meaning to differentiate it against the in-process agent writing the code. It doesn't matter too much if that fixed code was written by an LLM under guidance and review of the SWE, outside the agent.
- Barbing 5mo ago
  Agreed, “literally written by hand” didn’t cross my mind. Not by keyboard or pen.
- footydude 5mo ago
  Ahh ok - that's fair enough - hand-reviewed/not controlled by the agent seems a sensible approach (wasn't sure if it was instructive of a complete distrust of AI generated code)

dpark 5mo ago

This exploit has essentially nothing to do with AI and everything to do with a terribly designed account recovery flow. This exact same flow could have been (and may have been; I don’t know how much the chatbot here actually does) statically coded.

deleted 5mo ago
[deleted]
aidenn0 5mo ago
My impression is that AI didn't replace static code in this place; it replaced a person, who (hopefully) would have been suspicious about sending an account recovery code for e.g. "obamawhitehouse" to e.g. "bscurtu.alfamm.ro@gmail.com"
- soerxpso 5mo ago
  You're giving a lot of credit to the human alternative, especially considering that the attacker only needs to find one lazy human.
  - kelipso 5mo ago
    Come on, this attack vector would have been flagged by at least one person and you won’t then have multiple accounts hacked because of it. AI reacts fairly predictably to a single attack vector and don’t learn unless it gets flagged and then taught.
    - devmor 5mo ago
      And even if a human didn’t catch it in one case, they will frequently. Giving AI access to the same tools humans use without any oversight mechanism just amplifies the harm and carelessness possible by one person.
  - afavour 5mo ago
    Still makes this exponentially worse, no? It works every time and it's automated so scales up as quickly as you're able to request it.

nashashmi 5mo ago

Some Jr engineer got tired of handling stupid support requests and automated the job with an agent. That’s how. Assigning Jr engineers for security support is ridiculous partly because young people don’t understand how critical security is sometimes. And partly because they don’t value privacy as much.

kennywinker 5mo ago
Very generous of you to blame the screw up of one of the largest companies in the world on a jr engineer. I’ve been a jr engineer at a large company. I had the power to implement absolutely jack shit on my own. I deeply doubt the security flow for account recovery in meta ai account security was a single jr engineer. What i think is actually going on is basically a soft form of ai psychosis. Senior engineer gets ai to code ai account recovery feature, that same or a different engineer asks ai to review the feature, and then it gets pushed to prod. Move fast, break things. The ai coded it, the ai reviewed it - the people trusted the ai because it sounds confidently right. Just like how the ai doesn’t know if you should walk or drive to the car wash, the ai doesn’t understand exploits like this one.
parable 5mo ago
As a "young person" (under 30), my thoughts: There's a minority of us that do genuinely care, possibly more than most - so hiring someone from this minority would be helpful - but the vast majority of my peers don't care about privacy nor security. They often take this defeatist mindset of "my data is already out there, why should I care?", or prefer convenience over security. For example, "why should I switch to Signal if I have a public Instagram profile?" or "I can't remember all those passwords! I just use one for everything." As for your comment about junior engineers, see kennywinker's reply to this thread - I share the same thoughts.
acdha 5mo ago
If a single junior engineer can do this, it’s an even bigger indictment of Facebook’s senior management than this exploit. A well-designed system doesn’t rely on individuals never making mistakes and if our hypothetical junior developer can make critical security policy changes without oversight, that should be a C-level job loss event. If our goal isn’t to make excuses for the top of the org chart, a more likely explanation is that senior management is heavily incentivizing shipping AI features and this went out as a high-impact change reviewed in a rush, probably by AI.

plagiarist 5mo ago

This exploit is my new gold standard for trivially avoidable security failures. Someone has finally beaten Gitlab's password reset emails to attacker-provided addresses.

brianmcnulty 5mo ago

I do a lot of bug bounty research on Meta and Instagram, and some of the bugs I find look extremely simple like this but have some slightly complicated reason for why they occur. Maybe not this one, but I do have a guess as to what might have actually happened. Based on what I've seen so far, Meta AI Support Assistant (they call it "MAISA") had tool calls that a) start an email verification to any specific email, phone number, or the contact points linked to an account and b) allow generating a password reset link for an account based on an email verification attempt. I don't think it had any access to the actual codes themselves, but rather think a handle or ID for an email verification attempt (along with the user provided verification code based on user input) was provided to the "generate reset password link" tool call, and the tool call failed to properly validate the actual email used in that attempt belonged to the account allowing the ATO. The tool call for MAISA to generate a password reset link should have failed with an email verification attempt that corresponds to an email not linked to the account (and I believe I even tested this at one point on Facebook and encountered an error that successfully prevented it), but I suspect they tried making a change to this tool call for Instagram where slightly older, recently unlinked emails could be used to recover an account that got hijacked by an attacker, which added the need to allow emails not currently linked to the account to be used and set to the user's primary email. I also suspect that the MAISA tool call change called a wrong API or something that unintentionally allowed any email verification attempt that was successful to be used, but the engineers did not add a sufficiently thorough e2e test case to test the tool call against unrelated email verification attempts being provided to the tool call. This is the part I think should be focused on the most. Tool calls for agents that have their output potentially influenced by an attacker should be treated like external APIs that anyone can reach, and they should be tested as such. This is all obviously a guess, doesn't take into account the many signals they use to determine if an account recovery attempt is valid, and could be very inaccurate, but it's the closest to what I (someone who deals with Meta security a lot) think could have allowed this to happen.

muglug 5mo ago
Seems like the most plausible explanation. OTOH it feels like this is the sort of thing that might have been discovered/mitigated more quickly had there been a human in the loop.

guestbest 5mo ago

One would have to assume that this was by design.

drtz 5mo ago

Yeah it's bad, but AI isn't required for this type of thing to work. My anecdotal experience is my Facebook account was compromised several years ago after TOTP 2FA was disabled. Didn't exactly give me a warm fuzzy about Facebook security policies at the time, and this new attack just reaffirms that.

taco_emoji 5mo ago

This reeks of vibe coding. "Make it so the AI agent can help with password resets" and then zero human vetting of the change.

chuckadams 5mo ago
The human vetting was that it was cheaper. Someone probably got promoted for it.
mannanj 5mo ago
And zero accountability too. No one will be found and detected.

smrtinsert 5mo ago

It's stuff like this that honestly makes it very hard for me to take anyone working at Meta seriously. How much communication had to happen to enable this feature? It really casts doubt across the organization at multiple levels, don't tell me a single engineer caused this.

queenkjuul 5mo ago
I can't take Meta seriously, period.

mjmsmith 5mo ago

If this exploit has nothing to do with AI, why haven't we heard about it succeeding before? I find it hard to believe it's never been tried.

pif 5mo ago

> Why did they give it any of that?! Because they are idiots. You need to be a freaking idiit to trust AI.

dannyw 4mo ago

No big tech company hands write code by default now. You’d get PIP’d very quickly when your manager notices your PRs don’t have AI as a co-committer.