6 ms·
I guess I don't really get the attack. The idea seems to be that if you give your Claude an access token, despite what you tell it that it's for, Claude can be
by losvedir 1y ago
I guess I don't really get the attack. The idea seems to be that if you give your Claude an access token, despite what you tell it that it's for, Claude can be convinced to use it for anything that it's authorized for.
I think that's probably something anybody using these tools should always think. When you give a credential to an LLM, consider that it can do up to whatever that credential is allowed to do, especially if you auto-allow the LLM to make tool use calls!
But GitHub has fine-grained access tokens, so you can generate one scoped to just the repo that you're working with, and which can only access the resources it needs to. So if you use a credential like that, then the LLM can only be tricked so far. This attack wouldn't work in that case. The attack relies on the LLM having global access to your GitHub account, which is a dangerous credential to generate anyway, let alone give to Claude!
- lbeurerkellner 1y agoI agree, one of the issues are tokens with too broad permission sets. However, at the same time, people want general agents which do not have to be unlocked on a repository-by-repository basis. That's why they give them tokens with those access permissions, trusting the LLM blindly. Your caution is wise, however, in my experience, large parts of the eco-system do not follow such practices. The report is an educational resource, raising awareness that indeed, LLMs can be hijacked to do anything if they have the tokens, and access to untrusted data. The solution: To dynamically restrict what your agent can and cannot do with that token. That's precisely the approach we've been working on for a while now [1]. [1] https://explorer.invariantlabs.ai/docs/guardrails/ https://explorer.invariantlabs.ai/docs/guardrails/
- idontwantthis 1y agoWe all want to not have to code permissions properly, but we live in a society.
- flakeoil 1y agoHow about using LLMs to help us configure the access permissions and guardrails? /s I think I have to go full offline soon.
- spacebanana7 1y agoThis is interesting to expanding upon. Conceivably, prompt injection could be leveraged to make LLMs give bad advice. Almost like social engineering.
- TeMPOraL 1y agoProblem is, the mental model of what user wants to do almost never aligns with whatever security model the vendor actually implemented. Broadly-scoped access at least makes it easy on the user; anything I'd like to do will fit as a superset of "read all" or "read/write all". The fine-grained access forces people to solve a tough riddle, that may actually not have a solution. E.g. I don't believe there's a token configuration in GitHub that corresponds to "I want to allow pushing to and pulling from my repos, but only my repos, and not those of any of the organizations I want to; in fact, I want to be sure you can't even enumerate those organizations by that token". If there is one, I'd be happy to learn - I can't figure out how to make it out of checkboxes GitHub gives me, and honestly, when I need to mint a token, solving riddles like this is the last thing I need. Getting LLMs to translate what user wants to do into correct configuration might be the simplest solution that's fully general.
- ljm 1y agoIf you look at Github's fine-grained token permissions then I can totally imagine someone looking at the 20-30 separate scopes and thinking "fuck this" while they back out and make a non-expiring classic token with access to everything. It's one of those things where a token creation wizard would come in really handy.
- robertlagrant 1y agoI agree, but that is the permissions boundary, not the LLM. Saying "ooh it's hard so things are fuzzy" just perpetuates the idea that you can create all-powerful API keys.
- sam-cop-vimes 1y agoThis has happened to me. Can't find the exact combination of scopes required for the job to be done so you end up with the "f this" scenario you mentioned. And it is a constant source of background worry.
- ahmeni 1y agoDon't forget the also fun classic "what you want to do is not possible with scoped tokens so enjoy your PAT". I think we're now at year 3 of PATs being technically deprecated but still absolutely required in some use cases.
- arccy 1y agogithub's fine grained scopes aren't even that good, you still have to grant super broad permissions to do specific things, especially when it comes to orgs
- weego 1y agoI've definitely done this, but it's in a class of "the problem is between the keyboard and chair" 'exploits' that shouldn't be pinned on a particular tech or company.
- ljm 1y ago
- shawabawa3 1y agoThis is like 80% of security vulnerability reports we receive at my current job Long convoluted ways of saying "if you authorize X to do Y and attackers take X, they can then do Y"
- tough 1y agoLong convoluted ways of saying users don't know shit and will click any random links
- grg0 1y agoSounds like confused deputy and is what capability-based systems solve. X should not be allowed to do Y, but only what the user was allowed to do in the first place (X is only as capable as the user, not more.)
- Aurornis 1y agoWe had a bug bounty program manager who didn’t screen reports before sending them to each team as urgent tickets. 80% of the tickets were exactly like you said: “If the attacker could get X, then they can also do Y” where “getting X” was often equivalent to getting root on the system. Getting root was left as an exercise to the reader.
- stzsch 1y agoOr as Raymond Chen likes to put it: "It rather involved being on the other side of this airtight hatchway". https://devblogs.microsoft.com/oldnewthing/20060508-22/?p=31283 https://devblogs.microsoft.com/oldnewthing/20060508-22/?p=31... (actually a hitchhiker's guide to the galaxy quote, but I digress)
- LelouBil 1y agoI'm 100% gonna be reusing the "unlocking windows from the inside" analogy
- monkeyelite 1y agoSecurity teams themselves make these reports all the time. Internal tools do not have the same vulnerabilities as systems which operate on external data.
- deleted 1y ago[deleted]
- worldsayshi 1y agoYes, if you let the chatbot face users you have to assume that the chatbot will be used for anything it is allowed to do. It's a convenience layer op top of your api. It's not an api itself. Clearly?
- Abishek_Muthian 1y agoThis is applicable to those deployment services like Railway which require access to all the GitHub repositories even though we need to deploy only a single project. In that regard Netlify respects access to just the repository we want to deploy. GitHub shouldn't approve the apps which don't respect the access controls.
- hoppp 1y agoThey exploit the fact the llm will do anything it can to anyone. These tools cant exist securely as long as the llm doesn't reach at least the level of intelligence of a bug that can make decisions about access control and knows the concept of lying and bad intent
- om8 1y agoEven human level intelligence (whatever that means) is not enough. Social engineering works fine on our meat brains, it will most probably work on llms for foreseeable non-weird non-2027-takeoff-timeline future. Based on “bug level of intelligence”, I (perhaps wrongly) infer that you don’t believe in possibility of a takeoff. In case it is even semi-accurate, I think llms can be secure, but, perhaps, humanity will be able to interact with such secure system for not so long time
- hoppp 1y agoI believe it takes off. I just think a bug is the lowest lifeform that can differentiate between friend or foe. so that's why I wrote that but it could be a fish or whatever But I do think we need a different paradigm to get to actual intelligence as an LLM is still not it.
- dodslaser 1y agoYes they can. If the token you give the LLM isn't permitted to access private repos you can lie all you want, it still can't access private repos. Of course you shouldn't give an app/action/whatever a token with too lax permissions. Especially not a user facing one. That's not in any way unique to tools based on LLMs.
- om8 1y agoI thing you are just arguing about words, not about meanings. I’d call what you are referring to “secure llm infrastructure ”, not “secure llm”. But the thing is that we both agree about what’s going on, just with different words
- 1y ago
- miki123211 1y agoThe issue here (which is almost always the case with prompt injection attacks) is that an LLM has access to attacker-controlled data, sensitive information, and a data exfiltration capability. THe "cardinal rule of agent design" should be that an LLM can have access to at most two of these during one session. To avoid security issues, agents should be designed in a way that ensures this. For example, any agent that accesses an issue created by an untrusted party should be considered "poisoned" by attacker-controlled data. If it then accesses any private information, its internet access capability should be severely restricted or disabled altogether until context is cleared. In this model, you don't need per-repo tokens. As long as the "cardinal rule" is followed, no security issue is possible. Sadly, it seems like MCP doesn't provide the tools needed to ensure this.
- tmpz22 1y agoGenuine question - can we even make a convincing argument for security over convenience to two generations of programmers who grew up on corporate breach after corporate breach with just about zero tangible economic or legal consequences to the parties at fault? Presidential pardons for about a million a pop [1]? What’s the cassus belli to this younger crop of executives that will be leading the next generation of AI startups? [1]: https://www.cnbc.com/2025/03/28/trump-pardons-nikola-trevor-milton-ceo-securities-fraud-electric-vehicle.html https://www.cnbc.com/2025/03/28/trump-pardons-nikola-trevor-...
- eGQjxkKF6fif 1y agoAs ethical hackers and for the love of technology, yes we can make a convincing argument for security over convenience. Don't look too much in to it I say; there will always be people convincing talent to do and make things and disregard security and protocol. Those younger flocks of execs will have been mentored and answer to others. Their fiduciary duty is to share-holders and the business' bottom line. Us, as technology enthusiasts should design, create, and launch things with security in mind. Don't focus on the tomfoolery and corruption, focus on the love for the craft. Just my opinion
- cwsx 1y ago
- babyshake 1y agoIt's just that agentic AI introduces the possibility of old school social engineering.
- p1necone 1y agoI've noticed this as a trend with new technology. People seem to forget the most basic things as if they don't apply because the context is new and special and different. Nope, you don't magically get to ignore basic security practices just because you're using some new shiny piece of tech. See also: the cryptocurrency space rediscovering financial fraud and scams from centuries ago because they didn't think their new shiny tech needed to take any lessons from what came before them.
- bloppe 1y agoWell you're not giving the access token to Claude directly. The token is private to the MCP server and Claude uses the server's API, so the server could (should) take measures to prevent things like this from happening. It could notify the user whenever the model tries to write to a public repo and ask for confirmation, for instance. Probably the only bulletproof measure is to have a completely separate model for each private repo that can only write to its designated private repo, but there are a lot of layers of security one could apply with various tradeoffs
- guluarte 1y agoAnother type of attack waiting to happen is a malicious prompt in a url where an attacker could make the model do a curl request to post sensitive information