8 ms·
New Vulnerability in GitHub Copilot, Cursor: Hackers Can Weaponize Code Agents
- handfuloflight 2y agoThe cunning aspect of human ingenuity will never cease to amaze me.
- almery 2y agoYes, but are you sure a human invented this attack?
- bryanrasmussen 2y agoIs there any LLM that includes in its training sets a large number of previous hacks? I bet there probably is but we don't know about it, and damn, now I suddenly got another moneymaking idea I don't have time to work on!
- handfuloflight 2y agoMaybe, maybe not, but right now humans are the ones who are exploiting it.
- ekzy 2y agoNot saying that you are, but reading this as if a AI bot wrote that comment gives me the chills
- tsimionescu 2y agoThe most concerning part of the attack here seems to be the ability to hide arbitrary text in a simple text file using Unicode tricks such that GitHub doesn't actually show this text at all, per the authors. Couple this with the ability of LLMs to "execute" any instruction in the input set, regardless of such a weird encoding, and you've got a recipe for attacks. However, I wouldn't put any fault here on the AIs themselves. It's the fact that you can hide data in a plain text file that is the root of the issue - the whole attack goes away once you fix that part.
- deleted 2y ago[deleted]
- NitpickLawyer 2y ago> the whole attack goes away once you fix that part. While true, I think the main issue here, and the most impactful is that LLMs currently use a single channel for both "data" and "control". We've seen this before on modems (ath0++ attacks via ping packet payloads) and other tech stacks. Until we find a way to fix that, such attacks will always be possible, invisible text or not.
- tsimionescu 2y agoI don't think that's an accurate way to look at how LLMs work, there is no possible separation between data and control given the fundamental nature of LLMs. LLMs are essentially a plain text execution engine. Their fundamental design is to take arbitrary human language as input, and produce output that matches that input in some way. I think the most accurate way to look at them from a traditional security model perspective is as a script engine that can execute arbitrary text data. So, just like there is no realsitics hope of securely executing an attacker-controllers bash script, there is no realistic way to provide attacker controlled input to an LLM and still trust the output. In this sense, I completely agree with Google and Microsoft's decision for these discolosures: a bug report of the form "if I sneak a malicious prompt, the LLM returns a malicious answer" is as useless as a bug report in Bash that says that if you find a way to feed a malicious shell script to bash, it will execute it and produce malicious results. So, the real problem is if people are not treating LLM control files as arbitrary scripts, or if tools don't help you detect attempts at inserting malicious content in said scripts. After all, I can also control your code base if you let me insert malicious instructions in your Makefile.
- valenterry 2y agoJust like with humans. And there will be no fix, there can only be guards.
- jerf 2y ago
- throwaway290 2y agoNext thing, LLMs that review code! Next next thing, poisoning LLMs that review code! Galaxy brain: just put all the effort from developing those LLMs into writing better code
- GenshoTikamura 2y agoMan I wish I could upvote you more. Most humans are never able to tell the wrong turn in real time until it's too late
- mrmattyboy 2y ago> effectively turning the developer's most trusted assistant into an unwitting accomplice "Most trusted assistant" - that made me chuckle. The assistant that hallucinates packages, avoides null-pointer checks and forgets details that I've asked it.. yes, my most trusted assistant :D :D
- bastardoperator 2y agoMy favorite is when it hallucinates documentation and api endpoints.
- pona-a 2y agoThis kind of nonsense prose has "AI" written all over it. In either case, be it if your writing was AI generated/edited or if you put so little thought into it, it reads as such, doesn't show give its author any favor.
- mrmattyboy 2y agoAre you talking about my comment or the article? :eyes:
- Joker_vD 2y agoWell, "trusted" in the strict CompSec sense: "a trusted system is one whose failure would break a security policy (if a policy exists that the system is trusted to enforce)".
- gyesxnuibh 2y agoWell my most trusted assistant would be the kernel by that definition
- jeffbee 2y agoI wonder which understands the effect of null-pointer checks in a compiled C program better: the state-of-the-art generative model or the median C programmer.
- mock-possum 2y agoSorry, but isn’t this a bit ridiculous? Who just allows the AI to add code without reviewing it? And who just allows that code to be merged into a main branch without reviewing the PR? They start out talking about how scary and pernicious this is, and then it turns out to be… adding a script tag to an html file? Come on, as if you wouldn’t spot that immediately? What I’m actually curious about now is - if I saw that, and I asked the LLM why it added the JavaScript file, what would it tell me? Would I be able to deduce the hidden instructions in the rules file?
- ohgr 2y agoOh man don’t even go there. It does happen. AI generated code will get to production if you don’t pay people to give a fuck about it or hire people who don’t give a fuck.
- rvnx 2y agoIt will also go in production because this is the most efficient way to produce code today
- ohgr 2y agoIt depends somewhat on how tolerant your customers are of shite. Literally all I’ve seen is stuff that I wouldn’t ship in a million years because of the potential reputational damage to our business. And I get told a lot by people who really have no idea what they are doing clearly that it’s actually good.
- GenshoTikamura 2y agoThe most efficient way per whom, AI stakeholders and top managers?
- Gerardo1 2y agoOnly if you don't examine that proposition at all. You still have to review AI generated code, and with a higher level of attention than you do most code reviews for your peer developers. That requires someone who understands programming, software design, etc. You still have to test the code. Even if AI generates perfect code, you still need some kind of QA shop. Basically you're paying for the same people to do similar work to what they do now, but now you also paying for an enterprise license to your LLM provider of choice.
- tobyhinloopen 2y agoStop hijacking scrolling. Why would you do that? What developer thought this was a good idea?
- bryanrasmussen 2y agothe LLM.
- richrichardsson 2y agoThe scrolling I didn't find too off putting, but that floating nav bar is beyond awful; I had to Inspect -> Delete Element to be able to read the article.
- guappa 2y agoI think the main issue is that designers and web "developers" do not use their own crap.
- DougBTX 2y agoFrom the article: > A 2024 GitHub survey found that nearly all enterprise developers (97%) are using Generative AI coding tools. These tools have rapidly evolved from experimental novelties to mission-critical development infrastructure, with teams across the globe relying on them daily to accelerate coding tasks. That seemed high, what the actual report says: > More than 97% of respondents reported having used AI coding tools at work at some point, a finding consistent across all four countries. However, a smaller percentage said their companies actively encourage AI tool adoption or allow the use of AI tools, varying by region. The U.S. leads with 88% of respondents indicating at least some company support for AI use, while Germany is lowest at 59%. This highlights an opportunity for organizations to better support their developers’ interest in AI tools, considering local regulations. Fun that the survey uses the stats to say that companies should support increasing usage, while the article uses it to try and show near-total usage already.
- rvnx 2y agoIn some way, we reached 100% of developers, and now usage is expanding, as non-developers can now develop applications.
- _heimdall 2y agoWouldn't that then make those people developers? The total pool of developers would grow, the percentage couldn't go above 100%.
- rvnx 2y agoProbably. There is a similar question: if you ask ChatGPT / Midjourney to generate a drawing, are you an artist ? (to me yes, which would mean that AI "vibe coders" are actual developers in their own way)
- dimitri-vs 2y agoIf my 5 yo daughter draws a square with a triangle on top is she an architect?
- MadsRC 2y agoWhen this was released I thought that perhaps we could mitigate it by having the tooling only load “rules” if they were signed. But thinking on it a bit more, from the LLMs perspective there’s no difference between the rule files and the source files. The hidden instructions might as well be in the source files… Using code signing on the rule files would be security theater. As mentioned by another comms ter, the solution could be to find a way to separate the command and data channels. The LLM only operates on a single channel, that being input of tokens.
- namaria 2y ago> As mentioned by another comms ter, the solution could be to find a way to separate the command and data channels. The LLM only operates on a single channel, that being input of tokens. I think the issue is deeper than that. None of the inputs to an LLM should be considered as command. It incidentally gives you output compatible with the language in what is phrased by people as commands. But the fact that it's all just data to the LLM and that it works by taking data and returning plausible continuations of that data is the root cause of the issue. The output is not determined by the input, it is only statistically linked. Anything built on the premise that it is possible to give commands to LLMs or to use it's output as commands is fundamentally flawed and bears security risks. No amount of 'guardrails' or 'mitigations' can address this fundamental fact.
- TeMPOraL 2y ago> As mentioned by another comms ter, the solution could be to find a way to separate the command and data channels. The LLM only operates on a single channel, that being input of tokens. It's not possible, period. Lack of it is the very thing that makes LLMs general-purpose tools and able to handle natural language so well. Command/data channel separation doesn't exist in the real world, humans don't have it either. Even limiting ourselves to conversations, which parts are commands and which are data is not clear (and doesn't really make sense) - most of them are both to some degree, and that degree changes with situational context. There's no way to have a model capable of reading between lines and inferring what you mean but only when you like it, not without time travel.
- yair99dd 2y agoReminds me of this wild paper https://boingboing.net/2025/02/26/emergent-misalignment-ai-trained-to-write-insecure-code-also-became-a-misanthropic-nazi.html https://boingboing.net/2025/02/26/emergent-misalignment-ai-t...
- gregwebs 2y agoIs there a proactive way to defend against invisible Unicode attacks?
- Tepix 2y agoFiltering them?
- DrNosferatu 2y agoFor some piece of mind, we can perform the search: OUTPUT=$(find .cursor/rules/ -name '*.mdc' -print0 2>/dev/null | xargs -0 perl -wnE ' BEGIN { $re = qr/\x{200D}|\x{200C}|\x{200B}|\x{202A}|\x{202B}|\x{202C}|\x{202D}|\x{202E}|\x{2066}|\x{2067}|\x{2068}|\x{2069}/ } print "$ARGV:$.:$_" if /$re/ ' 2>/dev/null) FILES_FOUND=$(find .cursor/rules/ -name '*.mdc' -print 2>/dev/null) if [[ -z "$FILES_FOUND" ]]; then echo "Error: No .mdc files found in the directory." elif [[ -z "$OUTPUT" ]]; then echo "No suspicious Unicode characters found." else echo "Found suspicious characters:" echo "$OUTPUT" fi - Can this be improved?
- Joker_vD 2y agoNow, my toy programming languages all share the same "ensureCharLegal" function in their lexers that's called on every single character in the input (including those inside the literal strings) that filters out all those characters, plus all control characters (except the LF), and also something else that I can't remember right now... some weird space-like characters, I think? Nothing really stops the non-toy programming and configuration languages from adopting the same approach except from the fact that someone has to think about it and then implement it.
- Cthulhu_ 2y agoHere's a Github Action / workflow that says it'll do something similar: https://tech.michaelaltfield.net/2021/11/22/bidi-unicode-github-defense/ https://tech.michaelaltfield.net/2021/11/22/bidi-unicode-git... I'd say it's good practice to configure github or whatever tool you use to scan for hidden unicode files, ideally they are rendered very visibly in the diff tool.
- anthk 2y agoYou can just use Perl for the whole script instead of Bash.
- deleted 2y ago[deleted]
- lukaslalinsky 2y agoI'm quite happy with spreading a little bit of scare about AI coding. People should not treat the output as code, only as a very approximate suggestion. And if people don't learn, and we will see a lot more shitty code in production, programmers who can actually read and write code will be even more expensive.
- GenshoTikamura 2y agoThere is an equal unit of trouble per each unit of "progress"
- Oras 2y agoThis is a vulnerability in the same sense as someone committing a secret key in the front end. And for enterprise, they have many tools to scan vulnerability and malicious code before going to production.
- AutoAPI 2y agoRecent discussion: Smuggling arbitrary data through an emoji https://news.ycombinator.com/item?id=43023508 https://news.ycombinator.com/item?id=43023508
- fjni 2y agoBoth GitHub and Cursor’s response seems a bit lazy. Technically they may be correct in their assertion that it’s the user’s responsibility. But practically isn’t part of their product offering a safe coding environment? Invisible Unicode instruction doesn’t seem like a reasonable feature to support, it seems like a security vulnerability that should be addressed.
- markussss 2y agoThis page has horrible scrolling. I really don't understand why anybody creates this kind of scroll. Are they not using what they create?
- AlienRobot 2y agoI don't think they create it, they just use some template that comes with it.
- nuxi 2y agoAnd then they don't ever open the page, right?
- zx0r1 2y ago[flagged]
- budmichstelk 2y ago[dead]
- TZubiri 2y agoMay god forgive me, but I'm rooting for the hackers on this one. Job security you know?
- jdthedisciple 2y agosimple solution: preprocess any input to agents by restricting them to a set of visible characters / filtering out suspicious ones
- cess11 2y agoNasty characters should be rather common in your test cases.
- stevenwliao 2y agoNot sure about internationalization but at least for English, constraining to ASCII characters seems like a simple solution.
- nektro 2y agohijacked scrollbar. cardinal sin.
- t_believ-er873 2y agoRecently, I've seen a lot of information on the internet on how attackers use AI to spread malware, like jailbreak vulnerabilities that allow attackers to modify the tool's behavior. Here is the good article also on the topic: https://gitprotect.io/blog/how-attackers-use-ai-to-spread-malware-on-github/ https://gitprotect.io/blog/how-attackers-use-ai-to-spread-ma...