5 ms·

It’s obviously fundamentally unsafe when Google, OpenAI and Anthropic haven’t released the same feature and instead use a locked down VM with no cookies to brow

• cma 1y ago
  I think if you let claude code go wild with auto approval something similar could happen, since it can search the web and has the potential for prompt injection in what it reads there. Even without auto approval on reading and modifying files, if you aren't running it in a sandbox it could write code that then modifies your browser files the next time you do something like run your unit tests that it made, if you aren't reviewing every change carefully.
  • veganmosfet 1y ago
    I tried this on Gemini CLI and it worked, just add some magic vibes ;-)
  • darepublic 1y ago
    I really don't get why you would use a coding agent in yolo mode. I use the llm code gen in chunks at least glancing over it each time I add something. Why the hell would you have an approach of AI take the wheel
    • ec109685 1y ago
      It still keeps you in the loop, but doesn’t ask to run shell commands, etc.
      • jameshart 1y ago
        That seems like a bad default. VSCode’s agent mode requires approval for shell commands every time by default, with a whitelisting capability (which is itself risky, because hiding shell commands in args to an executable is quite doable). Are people running agents under their own user identity without supervising the commands they run?
        • cma 1y ago
          The default is ask for approval with option to whitelist certain commands.
    • threecheese 1y ago
      It depends on what you are using it for; I use CC for producing code that’s run elsewhere, but have also found it’s useful for producing code and commands behind day to day sysadmin/maintenance tasks. I don’t actually allow it to YOLO in this case (I have a few brain cells left), but the fact that it’s excellent at using bash suggests there are some terminal-based computer use tasks it could be useful for, or some set of useful tasks that might be considered harmful on your laptop but much less so in a virtual machine or container.
    • szundi 1y ago
      [dead]
    • cma 1y ago
      If you are only glancing over it and not doing a detailed review I think you could get hit with a prompt injection in the way I mentioned, with it writing something into the code that then when you run tests or the app ends up doing the action, which could be spinning up another claude code instance with approval off or turning off safety hooks etc.
• snet0 1y ago
  > Instead they believe model alignment, trying to understand when a user is doing a dangerous task, etc. will be enough. Maybe I have a fundamental misunderstanding, but I feel like hoping that model alignment and in-model guardrails are statistical preventions, ie you'll reduce the odds to some number of zeroes preceeding the 1. These things should literally never be able to happen, though. It's a fools errand to hope that you'll get to a model where there is no value in the input space that maps to <bad thing you really don't want>. Even if you "stack" models, having a safety-check model act on the output of your larger model, you're still just multiplying odds.
  • zeta0134 1y ago
    The sortof fun thing is that this happens with human safety teams too. The Swiss Cheese model is generally used to understand how the failures can line up to cause disaster to punch right through the guardrails: https://medium.com/backchannel/how-technology-led-a-hospital-to-give-a-patient-38-times-his-dosage-ded7b3688558 https://medium.com/backchannel/how-technology-led-a-hospital... It's better to close the hole entirely by making dangerous actions actually impossible, but often (even with computers) there's some wiggle room. For example, if we reduce the agent's permissions, then we haven't eliminated the possibility of those permissions being exploited, merely required some sort of privilege escalation to remove the block. If we give the agent an approved list of actions, then we may still have the possibility of unintended and unsafe interactions between those actions, or some way an attacker could add an unsafe action to the list. And so on, and so forth. In the case of an AI model, just like with humans, the security model really should not assume that the model will not "make mistakes." It has a random number generator built right in. It will, just like the user, occasionally do dumb things, misunderstand policies, and break rules. Those risks have to be factored in if one is to use the things at all.
    • oskarkk 1y ago
      Thank you for that link, that was a great read.
    • recursivecaveat 1y ago
      Humans are dramatically stronger than LLMs. An LLM is like a human you can memory wipe and try to phish hundreds of times a second until you find a script that works. I agree with what you're saying, but it's important to frame an LLM is not like a security guard who will occasionally let a former employee in because they recognize them. They can be attacked pretty relentlessly and once they're open they're wide open.
• skaul 1y ago
  (I lead privacy at Brave and am one of the authors) > Instead they believe model alignment, trying to understand when a user is doing a dangerous task, etc. will be enough. No, we never claimed or believe that those will be enough. Those are just easy things that browser vendors should be doing, and would have prevented this simple attack. These are necessary, not sufficient.
  • cowboylowrez 1y ago
    what you're saying is that the described step, "model alignment" is necessary even though it will fail a percentage of the time. whenever I see something that is "necessary" but doesn't have like a dozen 9's for reliability against failure or something well lets make that not necessary then. whadya say?
    • skaul 1y ago
      That's not how defense-in-depth works. If a security mitigation catches 90% of the "easy" attacks, that's worth doing, especially when trying to give users an extremely powerful capability. It just shouldn't be the only security measure you're taking.
      • cowboylowrez 1y ago
        sure sure, except llms. I mean its valid and all bringing up tried and true maxims that we all should know regarding software, but whens the last time the ssl guys were happy with a fix that "has a chance of working, but a chance of not working." defense in depth is to prevent one layer failure from getting to the next, you know, exploit chains etc. Failure in a layer is a failure, not statistically expected behavior. we fix bugs. what we need to do is treat llms as COMPLETELY UNTRUSTED user input as has been pointed out here and elsewhere time and again. you reply to me like I need to be lectured, so consider me a dumb student in your security class. what am I missing here?
        • jrflowers 1y ago
          > what am I missing here? Yeah the tone of that response seems unnecessarily smug. “I’m working on removing your front door and I’m designing a really good ‘no trespassing’ sign. Only a simpleton would question my reasoning on this issue”
• ryanjshaw 1y ago
  Maybe the article was updated but right now it says “The browser should isolate agentic browsing from regular browsing”
  • skaul 1y ago
    That was in the blog from the starting, and it's also the most important mitigation we identified immediately when starting to think about building agentic AI into the browser. Isolating agentic browsing while still enabling important use-cases (which is why users want to use agentic browsing in the first place) is the hard part, which is presumably why many browsers are just rolling out agentic capabilities in regular browsing.
    • waterproof 1y ago
      Isn't there a situation where the agentic browser, acting correctly on behalf of the user, needs to send Bitcoin or buy plane tickets? Isn't that flexibility kind of the whole point of the system? If so, I don't see what you get by distinguishing between agentic and no agentic browsing. Bad actors will now be working to scam users' LLMs rather than the users themselves. You can use more LLMs to monitor the LLMs and try and protect them, but it's turtles all the way down. The difference: when someone loses their $$$, they're not a fool for falling for some Nigerian Prince wire scam themselves, they're just a fool for using your browser. Or am I missing something?
      • skaul 1y ago
        You're right that if the user logs into a sensitive website, the "isolated browsing" mitigation stops helping. We don't want the user to accidentally end up in that state though. Separately, I can also imagine use-cases for agentic browsing where the user doesn't have to be logged into sensitive websites. Summarizing Hacker News front page, for one.
  • ec109685 1y ago
    That was my point about dropping privileges. It can still be exploited if the summary contains a link to an image that the attacker can control via text on the page that the LLM sees. It’s just a lot of Swiss cheese. That said, it’s definitely the best approach listed. And turns that exploit into an XSS attack on reddit.com, which is still bad.
• ngcazz 1y ago
  > Instead they believe model alignment, trying to understand when a user is doing a dangerous task, etc. will be enough. In other words: motivated reasoning.
• ivape 1y ago
  A smart performant local model will be the equivalent of having good anti-virus and firewall software. It will be the only thing between you and wrong prompts being sent every which way from which app. We’re probably three or four years away from the hardware necessary for this (NPUs in every computer).
  • ec109685 1y ago
    A local LLM wouldn’t have helped at all here.
    • ivape 1y ago
      You can’t imagine a MITM LLM that sits between you and the world?
      • QuadmasterXLII 1y ago
        Local llms can get offline searched for vulnerabilities using gradient based attacks. they will always be very easy to prompt inject.
      • solid_fuel 1y ago
        I can't imagine how such a thing would _help_, it seems like it would just be another injection target.
• petralithic 1y ago
  > It’s interesting that in Brave’s post describing this exploit, they didn’t reach the fundamental conclusion this is a bad idea "It is difficult to get a man to understand something, when his salary depends on his not understanding it." - Upton Sinclair
  • jazzyjackson 1y ago
    "If there's a steady paycheck in it, I'll believe anything you say." -Winston Zeddemore