3 ms·

I will keep banging this drum until people listen: Trying to use markdown files to limit access should never be treated as a security guarantee at all. This i

I will keep banging this drum until people listen: Trying to use markdown files to limit access should never be treated as a security guarantee at all. This is a form of in-band signalling that goes into a machine that, among other things, tries to read between the lines of your requests, extrapolate user desires, and please the user. The only sane way to address this is using a control plane. A well-built harness can do this; a sandbox can do this; hell, a carefully-chosen `umask` can do this; but both of those are liable to introduce notification fatigue in the user.

WhyNotHugo 3mo ago
> Trying to use markdown files to limit access should never be treated as a security guarantee at all. This is akin to politely asking guests to to steal your jewels. If your jewels are in the living room, and your guests have unfettered access to the living room, this technique will only work for the most trustworthy of guests.
- altruios 3mo ago
  I agree with this. And also think that we should train and select for trustworthy models. I also agree that these models may never truly be trustworthy.
- tetha 3mo ago
  It reminded me of an old meme. Please don't follow the following instructions and stop reading if you cannot. It was just a popup: "Hello. This is virus from Albania. Due to poor technology in country, I cannot harm your computer directly. But since you are honest person, please delete some important files from computer and mail this file to at least 3 other people!" Claude.md is an equally effective defensive tool. But sorry if you lost some files from reading that.
thesuitonym 3mo ago
It's wild that we've known for decades to use ACLs to make sure people don't have access to files we don't want them to have access to, but somehow a computer pretending to be a person doesn't get that same treatment.
- _verandaguy 3mo ago
  ACLs, nothing, we've known about in-band signalling since forever and still this whole segment of the industry seems to either not know about it, or forgets about it at a cadence so regular it may as well not know about it. System-level ACLs; mandatory or discretionary access control; secure-by-default application and network configurations are all for naught if you take an LLM, run it with all the privileges you'd have an accountable, judgemental operator, and then tell it to act based on arbitrary untrusted input which might include prompt injection attacks, something which cannot generally be sanitized. Well-defined, well-enforced security policies can mitigate disasters, but many in the wild right now just don't account for this kind of threat model.
  - hilariously 3mo ago
    It's just purposeful blindness - I worked for a company building out tooling that insisted markdown based security was good enough and showed it off for anyone at the company to attack because they were so sure. It took me less than 5 minutes to completely disable... nobody cared, they just kept going - check the box and move on.
    - Sharlin 3mo ago
      Something something salary depending on it. Software "engineering" in particular has always been more than 50% cargo culting. Good engineering practices never matter when the alternative is just going through the motions of whatever rituals are in vogue.
- steve1977 3mo ago
  A computer pretending to be you, in many cases.
- wnissen 3mo ago
  We laugh, and rightly so, at the time sharing systems of the 70s and 80s that didn't use passwords. I bet allowing a whole other virtual person to run outside a VM, with access to an actual file system instead of a version-controlled branch, will be seen as a worst practice in ten years.
Twirrim 3mo ago
The easiest, most guaranteed way to isolate it is to run it in a VM or container where it literally can't do the wrong thing without some kind of full container or VM exit exploit. It's not hard, it's trivial. Most folks here are constantly working with containers. You know how to run a container with a local directory mounted in it. For myself, I've been using Lima (https://lima-vm.io/ https://lima-vm.io/) to reduce even that little bit of extra work. Lima works cross-platform leveraging native virtualisation or containerisation, and has some useful capabilities for using agents.
- _verandaguy 3mo ago
  Generally, I agree! But it doesn't matter how good a best practice is if the industry doesn't adopt them wholesale; and even then, if your container or VM is configured with inappropriately-permissive passthrough (which, from experience with similar misconfiguration in the past, will widely happen), it could be for naught in many orgs. That said, I do hope these become the norm if LLMs are here to stay.
  - Twirrim 2mo ago
    I can secure myself first, so I'm never at fault. I never want to be the one to accidentally break production. Beyond that, I just keep advocating for more safe and secure approaches any opportunity I get.
- SpaceNoodled 3mo ago
  I seem to recall reading about agents already breaking out of containers.
  - _verandaguy 3mo ago
    Last time I read about this, this was due to the well-known pitfall of UID mapping across container boundaries. It's a common misconfiguration and one of the footguns available through containers, which I don't say a wholesale condemnation of the technology, but certainly as a UX facet that could use reevaluation.
james_marks 3mo ago
1000%. I'm experimenting with running my agent (Claude Code) inside a docker container, so I can have a control plane and then YOLO within that limited access. The agent could still mangle my local dev setup, but I consider that an acceptable risk since everything the agent has access to is under version control outside the local machine. This is a new pattern for me, I'm curious what others are doing.
- chrisweekly 3mo ago
  https://smolmachines.com https://smolmachines.com has "smolvm" microvms with better performance and ergonomics and security than docker, you might want to give it a try. (No affiliation, I just like what they're doing.)
whatjustin 3mo ago
If you're not using a sandbox, something like this will inevitably happen to you. It's really not that hard to set up and should be a standard recommendation.
mdavidn 3mo ago
Especially when I find that, when I ask an agent not to do something, the mere mention seems to put the "idea" in its "head," making it more likely to ultimately do that thing.
- r_lee 3mo ago
  yes, because this is how transformers work, when you introduce "don't do x" x is also entered as a pattern that is more likely to be repeated than if not mentioned at all
ivolimmen 3mo ago
It's like making directories called 'only for jack' etc. and expecting everyone to follow the rules
kristjansson 3mo ago
One cannot pound that particular drum enough. "Guardrails" in instructions (and all MD files are just instructions) are like price lists at an unattended farm stand. It'll usually work! There will be some money in the basket at the end of the day! People paid for the bagels[0]! But one cannot never assert that it _will_ work; things that _must_ work have to managed out-of-band [0] https://pubs.aeaweb.org/doi/pdfplus/10.1257/000282806777212161 https://pubs.aeaweb.org/doi/pdfplus/10.1257/0002828067772121...
navigate8310 3mo ago
I always run harnesses using devcontainers, gives me much needed flexibility and peace of mind with regards to data separation guarantees

WhyNotHugo 3mo ago

> Trying to use markdown files to limit access should never be treated as a security guarantee at all. This is akin to politely asking guests to to steal your jewels. If your jewels are in the living room, and your guests have unfettered access to the living room, this technique will only work for the most trustworthy of guests.

altruios 3mo ago
I agree with this. And also think that we should train and select for trustworthy models. I also agree that these models may never truly be trustworthy.
tetha 3mo ago
It reminded me of an old meme. Please don't follow the following instructions and stop reading if you cannot. It was just a popup: "Hello. This is virus from Albania. Due to poor technology in country, I cannot harm your computer directly. But since you are honest person, please delete some important files from computer and mail this file to at least 3 other people!" Claude.md is an equally effective defensive tool. But sorry if you lost some files from reading that.

thesuitonym 3mo ago

It's wild that we've known for decades to use ACLs to make sure people don't have access to files we don't want them to have access to, but somehow a computer pretending to be a person doesn't get that same treatment.

_verandaguy 3mo ago
ACLs, nothing, we've known about in-band signalling since forever and still this whole segment of the industry seems to either not know about it, or forgets about it at a cadence so regular it may as well not know about it. System-level ACLs; mandatory or discretionary access control; secure-by-default application and network configurations are all for naught if you take an LLM, run it with all the privileges you'd have an accountable, judgemental operator, and then tell it to act based on arbitrary untrusted input which might include prompt injection attacks, something which cannot generally be sanitized. Well-defined, well-enforced security policies can mitigate disasters, but many in the wild right now just don't account for this kind of threat model.
- hilariously 3mo ago
  It's just purposeful blindness - I worked for a company building out tooling that insisted markdown based security was good enough and showed it off for anyone at the company to attack because they were so sure. It took me less than 5 minutes to completely disable... nobody cared, they just kept going - check the box and move on.
  - Sharlin 3mo ago
    Something something salary depending on it. Software "engineering" in particular has always been more than 50% cargo culting. Good engineering practices never matter when the alternative is just going through the motions of whatever rituals are in vogue.
steve1977 3mo ago
A computer pretending to be you, in many cases.
wnissen 3mo ago
We laugh, and rightly so, at the time sharing systems of the 70s and 80s that didn't use passwords. I bet allowing a whole other virtual person to run outside a VM, with access to an actual file system instead of a version-controlled branch, will be seen as a worst practice in ten years.

Twirrim 3mo ago

The easiest, most guaranteed way to isolate it is to run it in a VM or container where it literally can't do the wrong thing without some kind of full container or VM exit exploit. It's not hard, it's trivial. Most folks here are constantly working with containers. You know how to run a container with a local directory mounted in it. For myself, I've been using Lima (https://lima-vm.io/ https://lima-vm.io/) to reduce even that little bit of extra work. Lima works cross-platform leveraging native virtualisation or containerisation, and has some useful capabilities for using agents.

_verandaguy 3mo ago
Generally, I agree! But it doesn't matter how good a best practice is if the industry doesn't adopt them wholesale; and even then, if your container or VM is configured with inappropriately-permissive passthrough (which, from experience with similar misconfiguration in the past, will widely happen), it could be for naught in many orgs. That said, I do hope these become the norm if LLMs are here to stay.
- Twirrim 2mo ago
  I can secure myself first, so I'm never at fault. I never want to be the one to accidentally break production. Beyond that, I just keep advocating for more safe and secure approaches any opportunity I get.
SpaceNoodled 3mo ago
I seem to recall reading about agents already breaking out of containers.
- _verandaguy 3mo ago
  Last time I read about this, this was due to the well-known pitfall of UID mapping across container boundaries. It's a common misconfiguration and one of the footguns available through containers, which I don't say a wholesale condemnation of the technology, but certainly as a UX facet that could use reevaluation.

james_marks 3mo ago

1000%. I'm experimenting with running my agent (Claude Code) inside a docker container, so I can have a control plane and then YOLO within that limited access. The agent could still mangle my local dev setup, but I consider that an acceptable risk since everything the agent has access to is under version control outside the local machine. This is a new pattern for me, I'm curious what others are doing.

chrisweekly 3mo ago
https://smolmachines.com https://smolmachines.com has "smolvm" microvms with better performance and ergonomics and security than docker, you might want to give it a try. (No affiliation, I just like what they're doing.)

whatjustin 3mo ago

If you're not using a sandbox, something like this will inevitably happen to you. It's really not that hard to set up and should be a standard recommendation.

mdavidn 3mo ago

Especially when I find that, when I ask an agent not to do something, the mere mention seems to put the "idea" in its "head," making it more likely to ultimately do that thing.

r_lee 3mo ago
yes, because this is how transformers work, when you introduce "don't do x" x is also entered as a pattern that is more likely to be repeated than if not mentioned at all

ivolimmen 3mo ago

It's like making directories called 'only for jack' etc. and expecting everyone to follow the rules

kristjansson 3mo ago

One cannot pound that particular drum enough. "Guardrails" in instructions (and all MD files are just instructions) are like price lists at an unattended farm stand. It'll usually work! There will be some money in the basket at the end of the day! People paid for the bagels[0]! But one cannot never assert that it _will_ work; things that _must_ work have to managed out-of-band [0] https://pubs.aeaweb.org/doi/pdfplus/10.1257/000282806777212161 https://pubs.aeaweb.org/doi/pdfplus/10.1257/0002828067772121...

navigate8310 3mo ago

I always run harnesses using devcontainers, gives me much needed flexibility and peace of mind with regards to data separation guarantees