5 ms·

I agree with the sentiment but I think for normie agents to take off in the way that you expect, you're going to have to grant them with full access. But, by gr

I agree with the sentiment but I think for normie agents to take off in the way that you expect, you're going to have to grant them with full access. But, by granting agents full access, you immediately turn the computer into an extremely adversarial device insofar as txt files become credible threat vectors. For all the benefits that agents offer, they can be asymmetrically harmful. This is not a solved issue. That hurts growth. I don't disagree with your general points, though.

cjbarber 6mo ago
> For all the benefits that agents offer, they can be asymmetrically harmful. This is not a solved issue. Strongly agreed. I saw a few people running these things with looser permissions than I do. e.g. one non-technical friend using claude cli, no sandbox, so I set them up with a sandbox etc. And the people who were using Cowork already were mostly blind approving all requests without reading what it was asking. The more powerful, the more dangerous, and vice versa.
- TeMPOraL 6mo ago
  > I saw a few people running these things with looser permissions than I do. e.g. one non-technical friend using claude cli, no sandbox, so I set them up with a sandbox etc. People have different levels of safety-consciousness, but also different tolerances and threat models. For example, I would hesitate running a Mythos-level model in YOLO mode with full control over my computer, but right now, for personal stuff, even figuring out WTF are sandboxes in Claude Code / Gemini CLI, much less setting them up, is too much hassle. What's the worst it can do without me noticing? Format the drive and upload some private data into pastebin? Much as I hate cloud and the proliferation of 2FA in every service, that alone means it can't actually do more to me than waste few hours of my life, as I reimage my desktop and restore OneDrive (in case of destructive changes that got synced up). These models are not yet good enough to empty my bank account in few minutes I'm not looking; everything else they can do quickly is reversible or inconsequential. Now, I do look at things closely when working with agentic AI tools. But my threat model is limited to worrying about those few hours of my life. `rm -rf / --no-preserve-root` is an annoyance, not a danger. (I accept that different contexts give different threat modeling. I would be more worried if I were doing businessy business stuff with all kinds of secret sauces, or was processing PII of my employer's customers, or lived in a country where it's easy to have all your money stolen if your CC number or SSN gets posted online.)
planb 6mo ago
How many of these threat vectors are just theoretical? Don’t use skills from random sources (just like don’t execute files from unknown sources). Don’t paste from untrusted sites (don’t click links on untrusted sites). Maybe there are fake documentation sites that the agent will search and have a prompt injected - but I haven’t heard of a single case where that happened. For now, the benefits outweigh the risk so much that I am willing to take it - and I think I have an almost complete knowledge of all the attack vectors.
- postalcoder 6mo ago
  i think you lack creativity. you could create a site that targets a very narrow niche, say an upper income school district. build some credibility, get highly ranked on google due to niche. post lunch menus with hidden embedded text. the attack surface is so wide idk where to start.
  - planb 6mo ago
    Why would my agent retrieve that lunch menu?
    - thuuuomas 6mo ago
      Because it’s hooked up to a microphone in your kitchen & your kid is arguing with you about what lunch they want & they say “Hey [agent], what day is pizza day at [school]?”
      - planb 6mo ago
        I’m not doing that. That would be like giving my child shell access to my system.
    - boxedemp 6mo ago
      Funny joke, But for real, obviously we all know people use agents to pick restaurants and that's a legit vector. I agree it's not the biggest surface, but it's worth knowing imdo
avaer 6mo ago
> for normie agents to take off in the way that you expect, you're going to have to grant them with full access At this point it's a foregone conclusion this is what users will choose. It'll be like (lack of) privacy on the internet caused by the ad industrial complex, but much worse and much more invasive. The threats are real, but it's just a product opportunity to these companies. OpenAI and friends will sell the poison (insecure computing) and the antidote (Mythos et all) and eat from both ends. Anyone trying to stay safe will be on the gradient to a Stallmanesque monastic computing existence. I don't want this, I just think it's going down that route.
- intended 6mo ago
  There was a recent Stanford study which showed that AI enthusiasts and experts and the normies had very different sentiment when it came to AI. I think most people are going to say they dont want it. I mean, why would anyone want a tool that can screw up their bank account? What benefit does it gain them? Theres lots of cases of great highly useful LLM tools, but the moment they scale up you get slammed by the risks that stick out all along the long tail of outcomes.
  - ryandrake 6mo ago
    I agree, in general we are going to find that ultimately most employee end users don't want it. Assuming it actually makes you more productive. I mean, who the hell wants to be 10X more productive without a commensurate 10X compensation increase? You're just giving away that value to your employer. On the other hand, entrepreneurs and managers are going to want it for their employees (and force it on them) for the above reason.
    - yes_man 6mo ago
      If everyone becomes 10x more productive it won’t mean the companies cash flow 10x’s. Where value is loose there is competition, so in theory everyone should win. Unless nobody else can compete to capture that loose 10x value, in which case congratulations, you are now a unicorn. Of course in reality in the short term what happens is companies lay off people to increase margins. Times will be tough for workers, and equity keeps gravitating towards those who already had it.
canarias_mate 6mo ago
[flagged]
Anvoker 6mo ago
What about setting environments for normies that mitigate this problem? I don't know that you can do it on Windows, but Linux offers various tools for isolation where you can give full rights to an LLM and still be safe from certain classes of disaster. Maybe this kind of isolation neuters the benefit you're thinking of, but I do believe some sort of solution could be reached.
- teiferer 6mo ago
  "Isolation" and "full rights" are mutually exclusive, contradictory properties.
jasongi 6mo ago
I cannot reconcile that growth for non-technical users is going to explode, when most utility from agents is via the ability to execute arbitrary code, generally in yolo mode, with the fact that almost all corporate IT departments do not give users the ability to install anything on their machine, let alone arbitrary code. Even developers at many companies are subject to this despite the productivity impacts. The culture of corporate IT would need to change to allow it, and I just don't see it happening.

cjbarber 6mo ago

> For all the benefits that agents offer, they can be asymmetrically harmful. This is not a solved issue. Strongly agreed. I saw a few people running these things with looser permissions than I do. e.g. one non-technical friend using claude cli, no sandbox, so I set them up with a sandbox etc. And the people who were using Cowork already were mostly blind approving all requests without reading what it was asking. The more powerful, the more dangerous, and vice versa.

TeMPOraL 6mo ago
> I saw a few people running these things with looser permissions than I do. e.g. one non-technical friend using claude cli, no sandbox, so I set them up with a sandbox etc. People have different levels of safety-consciousness, but also different tolerances and threat models. For example, I would hesitate running a Mythos-level model in YOLO mode with full control over my computer, but right now, for personal stuff, even figuring out WTF are sandboxes in Claude Code / Gemini CLI, much less setting them up, is too much hassle. What's the worst it can do without me noticing? Format the drive and upload some private data into pastebin? Much as I hate cloud and the proliferation of 2FA in every service, that alone means it can't actually do more to me than waste few hours of my life, as I reimage my desktop and restore OneDrive (in case of destructive changes that got synced up). These models are not yet good enough to empty my bank account in few minutes I'm not looking; everything else they can do quickly is reversible or inconsequential. Now, I do look at things closely when working with agentic AI tools. But my threat model is limited to worrying about those few hours of my life. `rm -rf / --no-preserve-root` is an annoyance, not a danger. (I accept that different contexts give different threat modeling. I would be more worried if I were doing businessy business stuff with all kinds of secret sauces, or was processing PII of my employer's customers, or lived in a country where it's easy to have all your money stolen if your CC number or SSN gets posted online.)

planb 6mo ago

How many of these threat vectors are just theoretical? Don’t use skills from random sources (just like don’t execute files from unknown sources). Don’t paste from untrusted sites (don’t click links on untrusted sites). Maybe there are fake documentation sites that the agent will search and have a prompt injected - but I haven’t heard of a single case where that happened. For now, the benefits outweigh the risk so much that I am willing to take it - and I think I have an almost complete knowledge of all the attack vectors.

postalcoder 6mo ago
i think you lack creativity. you could create a site that targets a very narrow niche, say an upper income school district. build some credibility, get highly ranked on google due to niche. post lunch menus with hidden embedded text. the attack surface is so wide idk where to start.
- planb 6mo ago
  Why would my agent retrieve that lunch menu?
  - thuuuomas 6mo ago
    Because it’s hooked up to a microphone in your kitchen & your kid is arguing with you about what lunch they want & they say “Hey [agent], what day is pizza day at [school]?”
    - planb 6mo ago
      I’m not doing that. That would be like giving my child shell access to my system.
  - boxedemp 6mo ago
    Funny joke, But for real, obviously we all know people use agents to pick restaurants and that's a legit vector. I agree it's not the biggest surface, but it's worth knowing imdo

avaer 6mo ago

> for normie agents to take off in the way that you expect, you're going to have to grant them with full access At this point it's a foregone conclusion this is what users will choose. It'll be like (lack of) privacy on the internet caused by the ad industrial complex, but much worse and much more invasive. The threats are real, but it's just a product opportunity to these companies. OpenAI and friends will sell the poison (insecure computing) and the antidote (Mythos et all) and eat from both ends. Anyone trying to stay safe will be on the gradient to a Stallmanesque monastic computing existence. I don't want this, I just think it's going down that route.

intended 6mo ago
There was a recent Stanford study which showed that AI enthusiasts and experts and the normies had very different sentiment when it came to AI. I think most people are going to say they dont want it. I mean, why would anyone want a tool that can screw up their bank account? What benefit does it gain them? Theres lots of cases of great highly useful LLM tools, but the moment they scale up you get slammed by the risks that stick out all along the long tail of outcomes.
- ryandrake 6mo ago
  I agree, in general we are going to find that ultimately most employee end users don't want it. Assuming it actually makes you more productive. I mean, who the hell wants to be 10X more productive without a commensurate 10X compensation increase? You're just giving away that value to your employer. On the other hand, entrepreneurs and managers are going to want it for their employees (and force it on them) for the above reason.
  - yes_man 6mo ago
    If everyone becomes 10x more productive it won’t mean the companies cash flow 10x’s. Where value is loose there is competition, so in theory everyone should win. Unless nobody else can compete to capture that loose 10x value, in which case congratulations, you are now a unicorn. Of course in reality in the short term what happens is companies lay off people to increase margins. Times will be tough for workers, and equity keeps gravitating towards those who already had it.

canarias_mate 6mo ago

[flagged]

Anvoker 6mo ago

What about setting environments for normies that mitigate this problem? I don't know that you can do it on Windows, but Linux offers various tools for isolation where you can give full rights to an LLM and still be safe from certain classes of disaster. Maybe this kind of isolation neuters the benefit you're thinking of, but I do believe some sort of solution could be reached.

teiferer 6mo ago
"Isolation" and "full rights" are mutually exclusive, contradictory properties.

jasongi 6mo ago

I cannot reconcile that growth for non-technical users is going to explode, when most utility from agents is via the ability to execute arbitrary code, generally in yolo mode, with the fact that almost all corporate IT departments do not give users the ability to install anything on their machine, let alone arbitrary code. Even developers at many companies are subject to this despite the productivity impacts. The culture of corporate IT would need to change to allow it, and I just don't see it happening.