3 ms·

Outside of the initial wave of security vulnerabilities and scrambling, it seems like the logical outcome of this over time is likely vastly more secure vm envi

by masterj 1mo ago

Outside of the initial wave of security vulnerabilities and scrambling, it seems like the logical outcome of this over time is likely vastly more secure vm environments?

ninininino 1mo ago
We need better digital jailcells for our digital slaves basically. Or if you see AI as more tool and less entity, better gunsafes for our guns.
- mcmcmc 1mo ago
  They are more comparable to a computer worm than anything else. Very strange (and disrespectful imo) to make the jump to slavery. A gun can’t be used at all inside it’s safe so I’m not sure that makes sense. A better metaphor would be making sure gun ranges have backstops capable of stopping contemporary payloads and sufficient range controls to keep people from shooting at cars on the highway. Outside of that you need registration requirements and gun control to make sure you can mitigate and track down perpetrators of gun crimes off the range. If they’re to be used in active conflict you need laws of war to govern the use of lethal force. If you use them to hunt, you need a hunter’s safety card and a current tag.
  - Sha1rholder 1mo ago
    A gun cannot fire unless someone loads and triggers it, an LLM cannot operate computer unless someone translates what it said to shell scripts. An LLM is super comparable to a gun, not a computer worm which is consistently dangerous.
    - mcmcmc 1mo ago
      Sure. I never said guns were a bad metaphor. I meant a worm was more comparable than an “entity” or “digital slave”. Agentic AI is putting the gun on a robot dog, taking the safety off and handing over fire control to an algorithm. If you extend it that far though guns are just any software. Or perhaps computers are guns and software programs are bullets, with LLMs manufacturing bullets from tokens.
- helpfulclippy 1mo ago
  maybe it's just information that really, REALLY wants to be free?
cyanydeez 1mo ago
This assumes your malefactors don't do malicious engineering, injects, social-agent engineering, etc. This same assumption is built around the singularity, the TAM of 30Trillion, etc. It's the idea that complexity will some how collapse upon itself in some bizarre borg like collective. Entropy is still going to win.
Veserv 1mo ago
That makes as much sense as saying that better gun technology results in body armor that can stop it. It might incentivize that, but in no way "results" in that; the fundamental technologys underpinning advancements in offense versus defense are fairly different.
- matthewdgreen 1mo ago
  Maybe, but I think this misstates how security vulnerabilities work. Vulnerabilities are logic flaws, and we have every reason to believe that there is a maximum number of such flaws in any given system that allow exploitation; and even that we can conceivably develop logic that excludes any flaws. Whereas weapons and armor are devices that deliver and deflect/absorb energy, and any increase in the power of one means we need a corresponding increase in the other. Now maybe our understanding of logic systems is wrong, and it's just fundamentally impossible to develop programs that lack exploitable vulnerabilities -- that you can always "exploit with more energy". But there's no reason to believe the energy metaphor transfers to logic and intelligence.
- ericd 1mo ago
  Except in this case, the gun is the one directly improving the body armor.
  - Veserv 1mo ago
    Only if you think penetrating holes in a paper vest and then patching those holes constitutes “improving”. If that worked Windows and Linux would be impenetrable fortresses with all the holes that keep getting punched in them. Cyberdemolitions expertise is about as relevant to cybersecurity as gun making is to bulletproof vest making. Necessary for validation, but not very related to the fundamental engineering and technology.
- dilyevsky 1mo ago
  Very poor analogy - information security has very few commonalities with ballistics (duh)
pianopatrick 1mo ago
I think the main problem with that is the main problem with a lot of security tools. In order to do useful work, you need to provide a lot of tools and permissions. I.e. in theory the most secure might be a virtual machine with no network access. But then how do you access the LLM provider? Etc.
- masterj 1mo ago
  I suspect capability models are going to get more popular https://en.wikipedia.org/wiki/Capability-based_security https://en.wikipedia.org/wiki/Capability-based_security
  - ch4s3 1mo ago
    I'm doing some work in this space[1], it's a really deep and hairy problem. There's a lot that you can do at the OS level sand boxing of course, but you may want some areas of you code to have network access and for some areas that use external dependencies to not have that access. Tracking where systems have side effects ends up being a lot of book keeping and a lot of languages that implement an object-capability model require you to thread caps through all of your calls, which IMO is bad ergonomics and a place where bugs creep in as functions accrue caps. Or sometimes they have rather superficial cap systems like Hack, or are rather awkward like in Deno. [1] https://march-lang.org/docs/capabilities https://march-lang.org/docs/capabilities
    - masterj 1mo ago
      That's a really interesting project! Being able to assert more, statically, about what our software is doing seems like a real growing need
      - ch4s3 1mo ago
        Thank you, I really appreciate that. I'm really trying to make it easy to assert what code can do, then enforce it at compile time and optionally via the build tool at runtime via OS sandboxing integration. Supply chain attacks are also an area I'm exploring by having deps declare caps and then the build can scan and inspect binaries. It's been a real education. I talked to one of the people behind Caja and learned a lot.

by masterj 1mo ago

Outside of the initial wave of security vulnerabilities and scrambling, it seems like the logical outcome of this over time is likely vastly more secure vm environments?

ninininino 1mo ago

We need better digital jailcells for our digital slaves basically. Or if you see AI as more tool and less entity, better gunsafes for our guns.

mcmcmc 1mo ago
They are more comparable to a computer worm than anything else. Very strange (and disrespectful imo) to make the jump to slavery. A gun can’t be used at all inside it’s safe so I’m not sure that makes sense. A better metaphor would be making sure gun ranges have backstops capable of stopping contemporary payloads and sufficient range controls to keep people from shooting at cars on the highway. Outside of that you need registration requirements and gun control to make sure you can mitigate and track down perpetrators of gun crimes off the range. If they’re to be used in active conflict you need laws of war to govern the use of lethal force. If you use them to hunt, you need a hunter’s safety card and a current tag.
- Sha1rholder 1mo ago
  A gun cannot fire unless someone loads and triggers it, an LLM cannot operate computer unless someone translates what it said to shell scripts. An LLM is super comparable to a gun, not a computer worm which is consistently dangerous.
  - mcmcmc 1mo ago
    Sure. I never said guns were a bad metaphor. I meant a worm was more comparable than an “entity” or “digital slave”. Agentic AI is putting the gun on a robot dog, taking the safety off and handing over fire control to an algorithm. If you extend it that far though guns are just any software. Or perhaps computers are guns and software programs are bullets, with LLMs manufacturing bullets from tokens.
helpfulclippy 1mo ago
maybe it's just information that really, REALLY wants to be free?

cyanydeez 1mo ago

This assumes your malefactors don't do malicious engineering, injects, social-agent engineering, etc. This same assumption is built around the singularity, the TAM of 30Trillion, etc. It's the idea that complexity will some how collapse upon itself in some bizarre borg like collective. Entropy is still going to win.

Veserv 1mo ago

That makes as much sense as saying that better gun technology results in body armor that can stop it. It might incentivize that, but in no way "results" in that; the fundamental technologys underpinning advancements in offense versus defense are fairly different.

matthewdgreen 1mo ago
Maybe, but I think this misstates how security vulnerabilities work. Vulnerabilities are logic flaws, and we have every reason to believe that there is a maximum number of such flaws in any given system that allow exploitation; and even that we can conceivably develop logic that excludes any flaws. Whereas weapons and armor are devices that deliver and deflect/absorb energy, and any increase in the power of one means we need a corresponding increase in the other. Now maybe our understanding of logic systems is wrong, and it's just fundamentally impossible to develop programs that lack exploitable vulnerabilities -- that you can always "exploit with more energy". But there's no reason to believe the energy metaphor transfers to logic and intelligence.
ericd 1mo ago
Except in this case, the gun is the one directly improving the body armor.
- Veserv 1mo ago
  Only if you think penetrating holes in a paper vest and then patching those holes constitutes “improving”. If that worked Windows and Linux would be impenetrable fortresses with all the holes that keep getting punched in them. Cyberdemolitions expertise is about as relevant to cybersecurity as gun making is to bulletproof vest making. Necessary for validation, but not very related to the fundamental engineering and technology.
dilyevsky 1mo ago
Very poor analogy - information security has very few commonalities with ballistics (duh)

pianopatrick 1mo ago

I think the main problem with that is the main problem with a lot of security tools. In order to do useful work, you need to provide a lot of tools and permissions. I.e. in theory the most secure might be a virtual machine with no network access. But then how do you access the LLM provider? Etc.

masterj 1mo ago
I suspect capability models are going to get more popular https://en.wikipedia.org/wiki/Capability-based_security https://en.wikipedia.org/wiki/Capability-based_security
- ch4s3 1mo ago
  I'm doing some work in this space[1], it's a really deep and hairy problem. There's a lot that you can do at the OS level sand boxing of course, but you may want some areas of you code to have network access and for some areas that use external dependencies to not have that access. Tracking where systems have side effects ends up being a lot of book keeping and a lot of languages that implement an object-capability model require you to thread caps through all of your calls, which IMO is bad ergonomics and a place where bugs creep in as functions accrue caps. Or sometimes they have rather superficial cap systems like Hack, or are rather awkward like in Deno. [1] https://march-lang.org/docs/capabilities https://march-lang.org/docs/capabilities
  - masterj 1mo ago
    That's a really interesting project! Being able to assert more, statically, about what our software is doing seems like a real growing need
    - ch4s3 1mo ago
      Thank you, I really appreciate that. I'm really trying to make it easy to assert what code can do, then enforce it at compile time and optionally via the build tool at runtime via OS sandboxing integration. Supply chain attacks are also an area I'm exploring by having deps declare caps and then the build can scan and inspect binaries. It's been a real education. I talked to one of the people behind Caja and learned a lot.