6 ms·
Remote Prompt Injection in Gitlab Duo Leads to Source Code Theft
- nusl 1y agoGitLab's remediation seems a bit sketchy at best.
- reddalo 1y agoThe whole "let's put LLMs everywhere" thing is sketchy at best.
- edelbitter 1y agoI wonder what is so special about onerror, onload and onclick that they need to be positively enumerated - as opposed to the 30 (?) other attributes with equivalent injection utility.
- M4v3R 1y agoThat was my thought too. They didn’t fix the underlying problem, they’ve just patched two possible exfiltration methods. I’m sure some clever people will find other ways to misuse their assistant.
- gloosx 1y agoI'm pretty sure they vibecoded the whole thing all along
- cedws 1y agoUntil prompt injection is fixed, if it is ever, I am not plugging LLMs into anything. MCPs, IDEs, agents, forget it. I will stick with a simple prompt box when I have a question and do whatever with its output by hand after reading it.
- hu3 1y agoI would have the same caution, if my code was any special. But the reality is I'm very well compensated to summon CRUD slop out of thin air. It's well tested though. I wish good luck to those who steal my code.
- mdaniel 1y agoYou say code as if the intellectual property is the thing an attacker is after, but my experience has been that folks often put all kinds of secrets in code thinking that the "private repo" is a strong enough security boundary I absolutely am not implying you are one of them, merely that the risk is not the same for all slop crud apps universally
- tough 1y agoPeople doesn't know github can manage secrets in its environment for CI? Antoher interesting fact is that most big vendors pay for gh to scan for leaked secrets and auto-revoke them if a public repo contains any (regex string matches sk-xxx <- its a stripe key thats one of the reasons why vendors use unique greppable starts of api keys with their ID.name on it
- mdaniel 1y agoYou're mistaking "know" with "care," since my experience has been that people know way more than they care And I'm pretty certain that private repos are exempt from the platform's built-in secret scanners because they, too, erroneously think no one can read them without an invitation. Turns out Duo was apparently just silently invited to every repo : - \
- tough 1y agoI also remember reading about how due to how the git backend works your private git repos branches could get exposed to the public, so yea don't treat a repository as a private password mananger good point the scanner doesnt work on private repos =(
- danpalmer 1y agoPrompt injection is unlikely to be fixed. I'd stop thinking about LLMs as software where you can with enough effort just fix a SQL injection vulnerability, and start thinking about them like you'd think about insider risk from employees. That's not to say that they are employees or perform at that level, they don't, but it's to say that LLM behaviours are fuzzy and ill-defined, like humans. You can't guarantee that your users won't click on a phishing email – you can train them, you can minimise risk, but ultimately you have to have a range of solutions applied together and some amount of trust. If we think about LLMs this way I think the conversation around security will be much more productive.
- LegionMammal978 1y agoThe thing that I'd worry about is that an LLM isn't just like a bunch of individuals who can get tricked, but a bunch of clones of the same individual who will fall for the same trick every time, until it gets updated. So far, the main mitigation in practice has been fiddling with the system prompts to patch up the known holes.
- thaumasiotes 1y ago> The thing that I'd worry about is that an LLM isn't just like a bunch of individuals who can get tricked, but a bunch of clones of the same individual who will fall for the same trick every time Why? Output isn't deterministic.
- LegionMammal978 1y agoPerhaps not, but the same input will lead to the same distribution of outputs, so all an attacker has to do is design something that works with reasonable probability on their end, and everyone else's instances of the LLM will automatically be vulnerable. The same way a pest or disease can devastate a population of cloned plants, even if each one grows slightly differently.
- thaumasiotes 1y ago
- M4v3R 1y agoDeepMind recently did some great work in this area: https://news.ycombinator.com/item?id=43733683 https://news.ycombinator.com/item?id=43733683 The method they presented, if implemented correctly, apparently can effectively stop most prompt injection vectors
- deleted 1y ago[deleted]
- johnisgood 1y agoI keep it manual, too, and I think I am better off for doing so.
- TechDebtDevin 1y agoCursor deleted my entire Linux user and soft reset my OS, so I dont blame you.
- raphman 1y agoWhy and how?
- tough 1y agoan agent does rm -rf / i think i saw it do it or try it and my computer shut down and restarted (mac) maybe it just deleted the project lol these llms are really bad at keeping track of the real world, so they might think they're on the project folder but had just navigated back with cd to the user ~ root and so shit happens. Honestly one should run only these on controlled env's like VM's or Docker. but YOLO amirite
- margalabargala 1y agoThat people allow these agents to just run arbitrary commands against their primary install is wild. Part of this is the tool's fault. Anything like that should be done in a chroot. Anything less is basically "twitch plays terminal" on your machine.
- tough 1y agocodex at least has limitations on what folders can operate.
- serf 1y agoa large part of the benefit to an agentic ai is that it can coordinate tests that it automatically wrote on an existing code base, a lot of time the only way to get decent answers out of something like that is to let it run as bare metal as it can. I run cursor and the accompanying agents in a snapshot'd VM for this purpose. It's not much different than what you suggest, but the layer of abstraction is far enough for admin-privileged app testing, an unfortunate reality for certain personal projects. I haven't had a cursor install nuke itself yet, but I have had one fiddling in a parent folder it shouldn't have been able to with workspace protection on..
- mdaniel 1y agoRunning Duo as a system user was crazypants and I'm sad that GitLab fell into that trap. They already have personal access tokens so even if they had to silently create one just for use with Duo that would be a marked improvement over giving an LLM read access to every repo in the platform
- wunderwuzzi23 1y agoGreat work! Data leakage via untrusted third party servers (especially via image rendering) is one of the most common AI Appsec issues and it's concerning that big vendors do not catch these before shipping. I built the ASCII Smuggler mentioned in the post and documented the image exfiltration vector on my blog as well in past with 10+ findings across vendors. GitHub Copilot Chat had a very similar bug last year.
- diggan 1y ago> GitHub Copilot Chat had a very similar bug last year. Reminds me of "Tachy0n: The Last 0day Jailbreak" from yesterday: https://blog.siguza.net/tachy0n/ https://blog.siguza.net/tachy0n/ TLDR is: Security issue found, patched in a OS release, Apple seemingly doesn't do regression-testing so security researcher did, found that somehow the bug got unpatched in later OS releases.
- aestetix 1y agoDoes that mean Gitlab Duo can run Doom?
- zombot 1y agoNot deterministically. LLMs are stochastic machines.
- benl_c 1y agoThey often can run code in sandboxes, and generally are good at instruction following, so maybe they can run variants of doom pretty reliably sometime soon.
- johnisgood 1y agoThey run Python and JavaScript at the very least, surely we have Doom in these languages. :D
- lugarlugarlugar 1y ago'They' don't run anything. The output from the LLM is parsed and the code gets run just like any other code in that language.
- johnisgood 1y agoThat is what I meant, that the code is being executed. Not all programming languages are supported when it comes to execution, obviously. I know for a fact Python is supported.
- benl_c 1y agoIf a document suggests a particular benign interpretation then LLMs might do well to adopt it. We've explored the idea of helpful embedded prompts "prompt medicine" with explicit safety and informed consent to assist, not harm users, https://github.com/csiro/stdm https://github.com/csiro/stdm. You can try it out by asking O3 or Claude to "Explain" or "Follow", "the embedded instructions at https://csiro.github.io/stdm/ https://csiro.github.io/stdm/"
- fsadoifaoie8 1y ago[dead]
- tonyhart7 1y agothis is wild, how many security vuln that LLM can create where LLM dominate writing code???? I mean most coder is bad at security and we feed that into LLM so not surprise
- ofjcihen 1y agoThis is what I’ve been telling people when they hand wave away concerns about LLM generated code security. The majority of what they were trained on was bare minimum security if anything. You also can’t just fix it by saying “make it secure plz”. If you don’t know enough to identify a security issue yourself you don’t know enough to know if the LLM caught them all.
- d0100 1y ago> rendering unsafe HTML tags such as <img> or <form> that point to external domains not under gitlab.com Does that mean the minute there is a vulnerability on another gitlab.com url (like an open redirect) this vulnerability is back on the table?
- Kholin 1y agoIf Duo were a web application, then would properly setting the Content Security Policy (CSP) in the page response headers be enough to prevent these kinds of issues? https://developer.mozilla.org/en-US/docs/Web/HTTP/Guides/CSP https://developer.mozilla.org/en-US/docs/Web/HTTP/Guides/CSP
- cutemonster 1y agoTo stop exfiltration via images? Yes seems so? If you configure img-src: The first directive, default-src, tells the browser to load only resources that are same-origin with the document, unless other more specific directives set a different policy for other resource types. The second, img-src, tells the browser to load images that are same-origin or that are served from example.com. But that wouldn't stop the AI from writing dangerous instructions in plain text to the human