3 ms·

It’s not black magic, but it is non deterministic. It’s not going to erase security and stability but it will require new skills and reasoning. The current ment

It’s not black magic, but it is non deterministic. It’s not going to erase security and stability but it will require new skills and reasoning. The current mental model of “software will always do X if you prohibit bad actors from getting in” is broken.

Yoric 2y ago
I agree that with the generation of code we're discussing this mental model is broken, but I think it goes a bit beyond determinism. Non-determinism is something that we've always had to deal with. Maybe your user is going to take the USB key before you're done writing your file, maybe your disk is going to fail, the computer is going to run out of battery during a critical operation, or your network request is going to timeout. But this was non-determinism within predictable boundaries. Yes, you may need to deal with corrupted file, an incomplete transaction, etc. but you could fairly easily predict where it could happen and how it could affect the integrity of your system. Now, if you're relying on a GenAI or RAG at runtime for anything other than a end-user interface, you'll need to deal with the possibility that your code might be doing something entirely unrelated than what you're expecting. For instance, even if we assume that your GenAI is properly sandboxed (and I'm not counting on early movers in the industry to ensure anything close to proper sandboxing), you could request one piece of statistics you'd like to display to your user, only to receive something entirely unrelated – and quite possibly something that, by law, you're not allowed to use or display. If we continue on the current trajectory, I suspect that it will take decades before we achieve anything like the necessary experience to write code that works without accident. And if the other trend of attempting to automate away engineering jobs continues, we might end up laying off the only people with the necessary experience to actually see the accidents coming.
- schoen 2y ago
  > But this was non-determinism within predictable boundaries. Yes, you may need to deal with corrupted file, an incomplete transaction, etc. but you could fairly easily predict where it could happen and how it could affect the integrity of your system. There's been lots of interesting computer security research relying on aspects of the physical instantiation of software systems (like side channel attacks where something physical could be measured in order to reveal secret state, or fault injection attacks where an attacker could apply heat or radiation in order to make the CPU or memory violate its specifications occasionally). These attacks fortunately aren't always applicable because there aren't always attackers in a position to carry them out, but where they are applicable, they could be very powerful!
  - Yoric 2y ago
    Yeah, I briefly mentioned those in my previous message :)

Yoric 2y ago

I agree that with the generation of code we're discussing this mental model is broken, but I think it goes a bit beyond determinism. Non-determinism is something that we've always had to deal with. Maybe your user is going to take the USB key before you're done writing your file, maybe your disk is going to fail, the computer is going to run out of battery during a critical operation, or your network request is going to timeout. But this was non-determinism within predictable boundaries. Yes, you may need to deal with corrupted file, an incomplete transaction, etc. but you could fairly easily predict where it could happen and how it could affect the integrity of your system. Now, if you're relying on a GenAI or RAG at runtime for anything other than a end-user interface, you'll need to deal with the possibility that your code might be doing something entirely unrelated than what you're expecting. For instance, even if we assume that your GenAI is properly sandboxed (and I'm not counting on early movers in the industry to ensure anything close to proper sandboxing), you could request one piece of statistics you'd like to display to your user, only to receive something entirely unrelated – and quite possibly something that, by law, you're not allowed to use or display. If we continue on the current trajectory, I suspect that it will take decades before we achieve anything like the necessary experience to write code that works without accident. And if the other trend of attempting to automate away engineering jobs continues, we might end up laying off the only people with the necessary experience to actually see the accidents coming.

schoen 2y ago
> But this was non-determinism within predictable boundaries. Yes, you may need to deal with corrupted file, an incomplete transaction, etc. but you could fairly easily predict where it could happen and how it could affect the integrity of your system. There's been lots of interesting computer security research relying on aspects of the physical instantiation of software systems (like side channel attacks where something physical could be measured in order to reveal secret state, or fault injection attacks where an attacker could apply heat or radiation in order to make the CPU or memory violate its specifications occasionally). These attacks fortunately aren't always applicable because there aren't always attackers in a position to carry them out, but where they are applicable, they could be very powerful!
- Yoric 2y ago
  Yeah, I briefly mentioned those in my previous message :)