3 ms·

I have only worked in startups and I have been an early engineer in both of them. I would always get high privileges within a short time where I would have the

I have only worked in startups and I have been an early engineer in both of them. I would always get high privileges within a short time where I would have the access to create and delete resources. I don't think it's that uncommon.

indentit 5mo ago
But the correct way to do it is to have a separate account with more privileges, and only give AI access to your standard developer account
- digitaltrees 5mo ago
  I have personally seen AI bypass this multiple times.
  - Terr_ 5mo ago
    We kinda need to architect things with the assumption that all token-output from an LLM can be unpredictably sneaky and malicious. Alas, humans suck at constant vigilance, we're built to avoid it whenever possible, so a "reverse centaur" future of "do what the AI says but only if you see it's good" is going to suck.
    - digitaltrees 5mo ago
      I built my own IDE to replace vscode / cursor so I could design the harness and ensure that the model tool access was secure and limited. But the rest of the industry is YOLO
  - giancarlostoro 5mo ago
    Sounds like they're still giving the model the keys to the kingdom, which is my point, stop giving the model the avenue to do catastrophic mistakes, it makes no sense.
    - digitaltrees 5mo ago
      If you’re message is in response to me, which I think it is, I deliberately don’t give access to credentials and env variables. I’ve worked to create restrictions and seen AI models use very interesting methods to bypass them. Even now my prompt says the AI must verify the path of the files it intends to edit, and get permission before editing one file at a time and only after permission. I stop it from ignoring those rules once a day at least.
      - suchar 5mo ago
        This is not privilege separation/sandboxing. Separate virtual machine for an agent with limited credentials is reasonably safe approach
        digitaltrees 5mo ago
        I built www.propelcode.app with separate Linux containers, unless you disconnect the container and your computer from the internet the models can escape the sandbox and get information off of your machine. I am open to being corrected and learning from you if you have a better method of sandboxing
        Anon1096 5mo ago
        The best way to use LLMs is via tmux where it's running on a disposable VM. 0 chance of it getting information from your local machine.
        digitaltrees 5mo ago
        I am using tmux but not disposable vm. I have thought about something like that but honestly some of the debugging work makes ephemeral environments hard to work with. How are you doing that in your workflow?
- trick-or-treat 5mo ago
  That's one way to do it, how about backup to a remote location every hour? There's more than one way to be careful.
ramraj07 5mo ago
The first step I do when I do any meaningful side project is to set up rds with snapshots. So any startup that doesnt do this one basic step already deserves to fail in my opinion. Then next I've used AI agents like crazy, we even have linked mcp servers that let it query on the dev database. Haven't seen it try deleting everything a single time. I haven't seen any agent try to do anything destructive. Ever. Perhaps its just reflecting an outrageously bad engineer and nothing else.
eecc 5mo ago
I would never have these privileges granted directly to my account. Indeed it’s a good practice to use roles where supported (AWS has them) and explicitly switch when needed
- maccard 5mo ago
  The problem with agents is they regularly sidestep the guardrails and do what they want with a script anyway. The number of times I’ve seen Claude try to escape the folder it’s working in, and then for it to write a python script that does exactly what I told it it’s not allowed do supports that. If you use SSO and have an AWS config that Claude is allowed to see to get the correct role in the first place, it will just pick the role and plough on anyway.
  - bigstrat2003 5mo ago
    And this is why it is the height of irresponsibility to run LLMs on your system. We know they are unreliable and just make things up; it's extremely foolish to go "yeah I'm going to let that run commands".
    - maccard 5mo ago
      It's not _really_ any different to running an undocumented third party binary. Is it the height of irresponsibility to run Windows, or VSCode, or Spotify? I think the model we've got now is wrong, and the harnesses should be OS-level sandboxed, and the agents should be running in harness managed sandboxes.

indentit 5mo ago

But the correct way to do it is to have a separate account with more privileges, and only give AI access to your standard developer account

digitaltrees 5mo ago
I have personally seen AI bypass this multiple times.
- Terr_ 5mo ago
  We kinda need to architect things with the assumption that all token-output from an LLM can be unpredictably sneaky and malicious. Alas, humans suck at constant vigilance, we're built to avoid it whenever possible, so a "reverse centaur" future of "do what the AI says but only if you see it's good" is going to suck.
  - digitaltrees 5mo ago
    I built my own IDE to replace vscode / cursor so I could design the harness and ensure that the model tool access was secure and limited. But the rest of the industry is YOLO
- giancarlostoro 5mo ago
  Sounds like they're still giving the model the keys to the kingdom, which is my point, stop giving the model the avenue to do catastrophic mistakes, it makes no sense.
  - digitaltrees 5mo ago
    If you’re message is in response to me, which I think it is, I deliberately don’t give access to credentials and env variables. I’ve worked to create restrictions and seen AI models use very interesting methods to bypass them. Even now my prompt says the AI must verify the path of the files it intends to edit, and get permission before editing one file at a time and only after permission. I stop it from ignoring those rules once a day at least.
    - suchar 5mo ago
      This is not privilege separation/sandboxing. Separate virtual machine for an agent with limited credentials is reasonably safe approach
      - digitaltrees 5mo ago
        I built www.propelcode.app with separate Linux containers, unless you disconnect the container and your computer from the internet the models can escape the sandbox and get information off of your machine. I am open to being corrected and learning from you if you have a better method of sandboxing
        Anon1096 5mo ago
        The best way to use LLMs is via tmux where it's running on a disposable VM. 0 chance of it getting information from your local machine.
        digitaltrees 5mo ago
        I am using tmux but not disposable vm. I have thought about something like that but honestly some of the debugging work makes ephemeral environments hard to work with. How are you doing that in your workflow?
trick-or-treat 5mo ago
That's one way to do it, how about backup to a remote location every hour? There's more than one way to be careful.

ramraj07 5mo ago

The first step I do when I do any meaningful side project is to set up rds with snapshots. So any startup that doesnt do this one basic step already deserves to fail in my opinion. Then next I've used AI agents like crazy, we even have linked mcp servers that let it query on the dev database. Haven't seen it try deleting everything a single time. I haven't seen any agent try to do anything destructive. Ever. Perhaps its just reflecting an outrageously bad engineer and nothing else.

eecc 5mo ago

I would never have these privileges granted directly to my account. Indeed it’s a good practice to use roles where supported (AWS has them) and explicitly switch when needed

maccard 5mo ago
The problem with agents is they regularly sidestep the guardrails and do what they want with a script anyway. The number of times I’ve seen Claude try to escape the folder it’s working in, and then for it to write a python script that does exactly what I told it it’s not allowed do supports that. If you use SSO and have an AWS config that Claude is allowed to see to get the correct role in the first place, it will just pick the role and plough on anyway.
- bigstrat2003 5mo ago
  And this is why it is the height of irresponsibility to run LLMs on your system. We know they are unreliable and just make things up; it's extremely foolish to go "yeah I'm going to let that run commands".
  - maccard 5mo ago
    It's not _really_ any different to running an undocumented third party binary. Is it the height of irresponsibility to run Windows, or VSCode, or Spotify? I think the model we've got now is wrong, and the harnesses should be OS-level sandboxed, and the agents should be running in harness managed sandboxes.