5 ms·

This is like 80% of security vulnerability reports we receive at my current job Long convoluted ways of saying "if you authorize X to do Y and attackers take X

This is like 80% of security vulnerability reports we receive at my current job Long convoluted ways of saying "if you authorize X to do Y and attackers take X, they can then do Y"

tough 1y ago
Long convoluted ways of saying users don't know shit and will click any random links
grg0 1y ago
Sounds like confused deputy and is what capability-based systems solve. X should not be allowed to do Y, but only what the user was allowed to do in the first place (X is only as capable as the user, not more.)
Aurornis 1y ago
We had a bug bounty program manager who didn’t screen reports before sending them to each team as urgent tickets. 80% of the tickets were exactly like you said: “If the attacker could get X, then they can also do Y” where “getting X” was often equivalent to getting root on the system. Getting root was left as an exercise to the reader.
- stzsch 1y ago
  Or as Raymond Chen likes to put it: "It rather involved being on the other side of this airtight hatchway". https://devblogs.microsoft.com/oldnewthing/20060508-22/?p=31283 https://devblogs.microsoft.com/oldnewthing/20060508-22/?p=31... (actually a hitchhiker's guide to the galaxy quote, but I digress)
  - LelouBil 1y ago
    I'm 100% gonna be reusing the "unlocking windows from the inside" analogy
- monkeyelite 1y ago
  Security teams themselves make these reports all the time. Internal tools do not have the same vulnerabilities as systems which operate on external data.
tom1337 1y ago
Yea - I honestly don't get why a random commenter on your GitHub Repo should be able to run arbitrary prompts on a LLM which the whole "attack" seems to be based on?
- kiitos 1y ago
  Random commenters on your GitHub repo aren't able to run arbitrary prompts on your LLM. But if you yourself run a prompt on your LLM, which explicitly says to fetch random commenters' comments from your GitHub repo, and then run the body of those comments without validation, and then take the results of that execution and submit it as the body of a new PR on your GitHub repo, then, yeah, that's a different thing.
  - rafram 1y ago
    > if you yourself run a prompt on your LLM, which explicitly says to fetch random commenters' comments from your GitHub repo, and then run the body of those comments without validation, and then take the results of that execution and submit it as the body of a new PR on your GitHub repo Read the article more carefully. The repo owner only has to ask the LLM to “take a look at the issues.” They’re not asking it to “run” anything or create a new PR - that’s all the attacker’s prompt injection.
    - detaro 1y ago
      Doesn't seem that clear cut? "Look at these issues and address them" sounds to me like it could easily trigger PR creation, especially since the injected prompt does not specify it, but only suggests how to edit the code. I.e. I'd assume a normal issue would also trigger PR creation with that prompt.
    - kiitos 1y ago
      The repo owner needs to set up and run the GitHub MCP server with a token that has access to their public and private repos, set up and configure an LLM with access to that MCP server, and then ask that LLM to "take a look at my public issues _and address them_".
zer00eyz 1y ago
In many cases I would argue that these ARE bugs. Were talking about githubs token system here... by the time you have generated the 10th one of these and its expiring or you lost them along the way and re-generated them your just smashing all the buttons to get through it as fast and as thoughtlessly as possible. If you make people change their passwords often, and give them stupid requirements they write it down on a post it and stick it on their monitor. When you make your permissions system, or any system onerous the quality of the input declines to the minimal of effort/engagement. Usability bugs are still bugs... it's part of the full stack that product, designers and developers are responsible for.
- TeMPOraL 1y ago
  This. People adopting security aspect often tend to forget to account for all the additional complexity they implement user-side. More insidiously though, they also fail to understand the fundamental mismatch between the behavior they're expecting, vs. how the real world operates. Passwords are treated as means of identification. The implied expectation is that they stick to one person and one person only. "Passwords are like panties - change them often and never share them", as the saying goes. Except that flies in the face of how humans normally do things in groups. Sharing and delegation are the norm. Trust is managed socially and physically. It's perfectly normal and common to give keys to your house to a neighbor or even a stranger if situation demands it. It's perfectly normal to send a relative to the post office with a failed-delivery note in your name, to pick your mail up for you; the post office may technically not be allowed to give your mail to a third party, but it's normal and common practice, so they do anyway. Similarly, no matter what the banks say, it's perfectly normal to give your credit or debit card to someone else, e.g. to your kid or spouse to shop groceries for you - so hardly any store actually bothers checking the name or signature on the card. And so on, and so on. Even in the office, there's a constant need to have someone else access a computing system for you. Delegating stuff on the fly is how humans self-organize. Suppressing that is throwing sand into gears of society. Passwords make sharing/delegating hard by default, but people defeat that by writing them down. Which leads the IT/security side to try and make it harder for people to share their passwords, through technical and behavioral means. All this is an attempt to force passwords to become personal identifiers. But then, they have to also allow for some delegation, which they want to control (internalizing the trust management), and from there we get all kinds of complex insanity of modern security; juggling tightly-scoped tokens is just one small example of it. I don't claim to have a solution for it. I just strongly feel we've arrived at our current patterns through piling hacks after hacks, trying to herd users back to the barn, with no good idea why they're running away. Now that we've mapped the problem space and identified a lot of relevant concepts (e.g. authN vs authZ, identity vs. role, delegation, user agents, etc.), maybe it's time for some smart folks to figure out a better theoretical framework for credentials and access, that's designed for real-world use patterns - not like State/Corporate sees it, but like real people do. At the very least, understanding that would help security-minded people what extra costs their newest operational or technological lock incurs on users, and why they keep defeating it in "stupid" ways.

This is like 80% of security vulnerability reports we receive at my current job Long convoluted ways of saying "if you authorize X to do Y and attackers take X, they can then do Y"

tough 1y ago

Long convoluted ways of saying users don't know shit and will click any random links

grg0 1y ago

Sounds like confused deputy and is what capability-based systems solve. X should not be allowed to do Y, but only what the user was allowed to do in the first place (X is only as capable as the user, not more.)

Aurornis 1y ago

We had a bug bounty program manager who didn’t screen reports before sending them to each team as urgent tickets. 80% of the tickets were exactly like you said: “If the attacker could get X, then they can also do Y” where “getting X” was often equivalent to getting root on the system. Getting root was left as an exercise to the reader.

stzsch 1y ago
Or as Raymond Chen likes to put it: "It rather involved being on the other side of this airtight hatchway". https://devblogs.microsoft.com/oldnewthing/20060508-22/?p=31283 https://devblogs.microsoft.com/oldnewthing/20060508-22/?p=31... (actually a hitchhiker's guide to the galaxy quote, but I digress)
- LelouBil 1y ago
  I'm 100% gonna be reusing the "unlocking windows from the inside" analogy
monkeyelite 1y ago
Security teams themselves make these reports all the time. Internal tools do not have the same vulnerabilities as systems which operate on external data.

tom1337 1y ago

Yea - I honestly don't get why a random commenter on your GitHub Repo should be able to run arbitrary prompts on a LLM which the whole "attack" seems to be based on?

kiitos 1y ago
Random commenters on your GitHub repo aren't able to run arbitrary prompts on your LLM. But if you yourself run a prompt on your LLM, which explicitly says to fetch random commenters' comments from your GitHub repo, and then run the body of those comments without validation, and then take the results of that execution and submit it as the body of a new PR on your GitHub repo, then, yeah, that's a different thing.
- rafram 1y ago
  > if you yourself run a prompt on your LLM, which explicitly says to fetch random commenters' comments from your GitHub repo, and then run the body of those comments without validation, and then take the results of that execution and submit it as the body of a new PR on your GitHub repo Read the article more carefully. The repo owner only has to ask the LLM to “take a look at the issues.” They’re not asking it to “run” anything or create a new PR - that’s all the attacker’s prompt injection.
  - detaro 1y ago
    Doesn't seem that clear cut? "Look at these issues and address them" sounds to me like it could easily trigger PR creation, especially since the injected prompt does not specify it, but only suggests how to edit the code. I.e. I'd assume a normal issue would also trigger PR creation with that prompt.
  - kiitos 1y ago
    The repo owner needs to set up and run the GitHub MCP server with a token that has access to their public and private repos, set up and configure an LLM with access to that MCP server, and then ask that LLM to "take a look at my public issues _and address them_".

zer00eyz 1y ago

In many cases I would argue that these ARE bugs. Were talking about githubs token system here... by the time you have generated the 10th one of these and its expiring or you lost them along the way and re-generated them your just smashing all the buttons to get through it as fast and as thoughtlessly as possible. If you make people change their passwords often, and give them stupid requirements they write it down on a post it and stick it on their monitor. When you make your permissions system, or any system onerous the quality of the input declines to the minimal of effort/engagement. Usability bugs are still bugs... it's part of the full stack that product, designers and developers are responsible for.

TeMPOraL 1y ago
This. People adopting security aspect often tend to forget to account for all the additional complexity they implement user-side. More insidiously though, they also fail to understand the fundamental mismatch between the behavior they're expecting, vs. how the real world operates. Passwords are treated as means of identification. The implied expectation is that they stick to one person and one person only. "Passwords are like panties - change them often and never share them", as the saying goes. Except that flies in the face of how humans normally do things in groups. Sharing and delegation are the norm. Trust is managed socially and physically. It's perfectly normal and common to give keys to your house to a neighbor or even a stranger if situation demands it. It's perfectly normal to send a relative to the post office with a failed-delivery note in your name, to pick your mail up for you; the post office may technically not be allowed to give your mail to a third party, but it's normal and common practice, so they do anyway. Similarly, no matter what the banks say, it's perfectly normal to give your credit or debit card to someone else, e.g. to your kid or spouse to shop groceries for you - so hardly any store actually bothers checking the name or signature on the card. And so on, and so on. Even in the office, there's a constant need to have someone else access a computing system for you. Delegating stuff on the fly is how humans self-organize. Suppressing that is throwing sand into gears of society. Passwords make sharing/delegating hard by default, but people defeat that by writing them down. Which leads the IT/security side to try and make it harder for people to share their passwords, through technical and behavioral means. All this is an attempt to force passwords to become personal identifiers. But then, they have to also allow for some delegation, which they want to control (internalizing the trust management), and from there we get all kinds of complex insanity of modern security; juggling tightly-scoped tokens is just one small example of it. I don't claim to have a solution for it. I just strongly feel we've arrived at our current patterns through piling hacks after hacks, trying to herd users back to the barn, with no good idea why they're running away. Now that we've mapped the problem space and identified a lot of relevant concepts (e.g. authN vs authZ, identity vs. role, delegation, user agents, etc.), maybe it's time for some smart folks to figure out a better theoretical framework for credentials and access, that's designed for real-world use patterns - not like State/Corporate sees it, but like real people do. At the very least, understanding that would help security-minded people what extra costs their newest operational or technological lock incurs on users, and why they keep defeating it in "stupid" ways.