5 ms·

Until prompt injection is fixed, if it is ever, I am not plugging LLMs into anything. MCPs, IDEs, agents, forget it. I will stick with a simple prompt box when

by cedws 1y ago

Until prompt injection is fixed, if it is ever, I am not plugging LLMs into anything. MCPs, IDEs, agents, forget it. I will stick with a simple prompt box when I have a question and do whatever with its output by hand after reading it.

hu3 1y ago
I would have the same caution, if my code was any special. But the reality is I'm very well compensated to summon CRUD slop out of thin air. It's well tested though. I wish good luck to those who steal my code.
- mdaniel 1y ago
  You say code as if the intellectual property is the thing an attacker is after, but my experience has been that folks often put all kinds of secrets in code thinking that the "private repo" is a strong enough security boundary I absolutely am not implying you are one of them, merely that the risk is not the same for all slop crud apps universally
  - tough 1y ago
    People doesn't know github can manage secrets in its environment for CI? Antoher interesting fact is that most big vendors pay for gh to scan for leaked secrets and auto-revoke them if a public repo contains any (regex string matches sk-xxx <- its a stripe key thats one of the reasons why vendors use unique greppable starts of api keys with their ID.name on it
    - mdaniel 1y ago
      You're mistaking "know" with "care," since my experience has been that people know way more than they care And I'm pretty certain that private repos are exempt from the platform's built-in secret scanners because they, too, erroneously think no one can read them without an invitation. Turns out Duo was apparently just silently invited to every repo : - \
      - tough 1y ago
        I also remember reading about how due to how the git backend works your private git repos branches could get exposed to the public, so yea don't treat a repository as a private password mananger good point the scanner doesnt work on private repos =(
danpalmer 1y ago
Prompt injection is unlikely to be fixed. I'd stop thinking about LLMs as software where you can with enough effort just fix a SQL injection vulnerability, and start thinking about them like you'd think about insider risk from employees. That's not to say that they are employees or perform at that level, they don't, but it's to say that LLM behaviours are fuzzy and ill-defined, like humans. You can't guarantee that your users won't click on a phishing email – you can train them, you can minimise risk, but ultimately you have to have a range of solutions applied together and some amount of trust. If we think about LLMs this way I think the conversation around security will be much more productive.
- LegionMammal978 1y ago
  The thing that I'd worry about is that an LLM isn't just like a bunch of individuals who can get tricked, but a bunch of clones of the same individual who will fall for the same trick every time, until it gets updated. So far, the main mitigation in practice has been fiddling with the system prompts to patch up the known holes.
  - thaumasiotes 1y ago
    > The thing that I'd worry about is that an LLM isn't just like a bunch of individuals who can get tricked, but a bunch of clones of the same individual who will fall for the same trick every time Why? Output isn't deterministic.
    - LegionMammal978 1y ago
      Perhaps not, but the same input will lead to the same distribution of outputs, so all an attacker has to do is design something that works with reasonable probability on their end, and everyone else's instances of the LLM will automatically be vulnerable. The same way a pest or disease can devastate a population of cloned plants, even if each one grows slightly differently.
      - thaumasiotes 1y ago
M4v3R 1y ago
DeepMind recently did some great work in this area: https://news.ycombinator.com/item?id=43733683 https://news.ycombinator.com/item?id=43733683 The method they presented, if implemented correctly, apparently can effectively stop most prompt injection vectors
- deleted 1y ago
  [deleted]
johnisgood 1y ago
I keep it manual, too, and I think I am better off for doing so.
TechDebtDevin 1y ago
Cursor deleted my entire Linux user and soft reset my OS, so I dont blame you.
- raphman 1y ago
  Why and how?
  - tough 1y ago
    an agent does rm -rf / i think i saw it do it or try it and my computer shut down and restarted (mac) maybe it just deleted the project lol these llms are really bad at keeping track of the real world, so they might think they're on the project folder but had just navigated back with cd to the user ~ root and so shit happens. Honestly one should run only these on controlled env's like VM's or Docker. but YOLO amirite
    - margalabargala 1y ago
      That people allow these agents to just run arbitrary commands against their primary install is wild. Part of this is the tool's fault. Anything like that should be done in a chroot. Anything less is basically "twitch plays terminal" on your machine.
      - tough 1y ago
        codex at least has limitations on what folders can operate.
      - serf 1y ago
        a large part of the benefit to an agentic ai is that it can coordinate tests that it automatically wrote on an existing code base, a lot of time the only way to get decent answers out of something like that is to let it run as bare metal as it can. I run cursor and the accompanying agents in a snapshot'd VM for this purpose. It's not much different than what you suggest, but the layer of abstraction is far enough for admin-privileged app testing, an unfortunate reality for certain personal projects. I haven't had a cursor install nuke itself yet, but I have had one fiddling in a parent folder it shouldn't have been able to with workspace protection on..

by cedws 1y ago

hu3 1y ago

I would have the same caution, if my code was any special. But the reality is I'm very well compensated to summon CRUD slop out of thin air. It's well tested though. I wish good luck to those who steal my code.

mdaniel 1y ago
You say code as if the intellectual property is the thing an attacker is after, but my experience has been that folks often put all kinds of secrets in code thinking that the "private repo" is a strong enough security boundary I absolutely am not implying you are one of them, merely that the risk is not the same for all slop crud apps universally
- tough 1y ago
  People doesn't know github can manage secrets in its environment for CI? Antoher interesting fact is that most big vendors pay for gh to scan for leaked secrets and auto-revoke them if a public repo contains any (regex string matches sk-xxx <- its a stripe key thats one of the reasons why vendors use unique greppable starts of api keys with their ID.name on it
  - mdaniel 1y ago
    You're mistaking "know" with "care," since my experience has been that people know way more than they care And I'm pretty certain that private repos are exempt from the platform's built-in secret scanners because they, too, erroneously think no one can read them without an invitation. Turns out Duo was apparently just silently invited to every repo : - \
    - tough 1y ago
      I also remember reading about how due to how the git backend works your private git repos branches could get exposed to the public, so yea don't treat a repository as a private password mananger good point the scanner doesnt work on private repos =(

danpalmer 1y ago

Prompt injection is unlikely to be fixed. I'd stop thinking about LLMs as software where you can with enough effort just fix a SQL injection vulnerability, and start thinking about them like you'd think about insider risk from employees. That's not to say that they are employees or perform at that level, they don't, but it's to say that LLM behaviours are fuzzy and ill-defined, like humans. You can't guarantee that your users won't click on a phishing email – you can train them, you can minimise risk, but ultimately you have to have a range of solutions applied together and some amount of trust. If we think about LLMs this way I think the conversation around security will be much more productive.

LegionMammal978 1y ago
The thing that I'd worry about is that an LLM isn't just like a bunch of individuals who can get tricked, but a bunch of clones of the same individual who will fall for the same trick every time, until it gets updated. So far, the main mitigation in practice has been fiddling with the system prompts to patch up the known holes.
- thaumasiotes 1y ago
  > The thing that I'd worry about is that an LLM isn't just like a bunch of individuals who can get tricked, but a bunch of clones of the same individual who will fall for the same trick every time Why? Output isn't deterministic.
  - LegionMammal978 1y ago
    Perhaps not, but the same input will lead to the same distribution of outputs, so all an attacker has to do is design something that works with reasonable probability on their end, and everyone else's instances of the LLM will automatically be vulnerable. The same way a pest or disease can devastate a population of cloned plants, even if each one grows slightly differently.
    - thaumasiotes 1y ago

M4v3R 1y ago

DeepMind recently did some great work in this area: https://news.ycombinator.com/item?id=43733683 https://news.ycombinator.com/item?id=43733683 The method they presented, if implemented correctly, apparently can effectively stop most prompt injection vectors

deleted 1y ago
[deleted]

johnisgood 1y ago

I keep it manual, too, and I think I am better off for doing so.

TechDebtDevin 1y ago

Cursor deleted my entire Linux user and soft reset my OS, so I dont blame you.

raphman 1y ago
Why and how?
- tough 1y ago
  an agent does rm -rf / i think i saw it do it or try it and my computer shut down and restarted (mac) maybe it just deleted the project lol these llms are really bad at keeping track of the real world, so they might think they're on the project folder but had just navigated back with cd to the user ~ root and so shit happens. Honestly one should run only these on controlled env's like VM's or Docker. but YOLO amirite
  - margalabargala 1y ago
    That people allow these agents to just run arbitrary commands against their primary install is wild. Part of this is the tool's fault. Anything like that should be done in a chroot. Anything less is basically "twitch plays terminal" on your machine.
    - tough 1y ago
      codex at least has limitations on what folders can operate.
    - serf 1y ago
      a large part of the benefit to an agentic ai is that it can coordinate tests that it automatically wrote on an existing code base, a lot of time the only way to get decent answers out of something like that is to let it run as bare metal as it can. I run cursor and the accompanying agents in a snapshot'd VM for this purpose. It's not much different than what you suggest, but the layer of abstraction is far enough for admin-privileged app testing, an unfortunate reality for certain personal projects. I haven't had a cursor install nuke itself yet, but I have had one fiddling in a parent folder it shouldn't have been able to with workspace protection on..