4 ms·

That's an interesting difference but I'm not convinced it's valid. Perhaps you can explain what you mean in more detail? Let's say the software is good enough

That's an interesting difference but I'm not convinced it's valid. Perhaps you can explain what you mean in more detail? Let's say the software is good enough if it does the right thing 99.9% of the time it is used. I take it, you're saying that if an AI starts modifying it and only writes correct code in 99.9% of cases (yes, current AI is not even close, but it will improve), that makes it worse because the software might start failing completely. However, if you have proper tests and release management, such obvious flaws will quickly be detected and fixed or rolled back. For most applications that seems pretty much equivalent to what we have now. The other case is software that is completely AI generated and cannot reasonably be modified by humans anymore. In that case, again, you have tests and a sane deployment strategy that mitigates failures to a sufficient degree depending on application. So the only issue is when you start making a completely AI generated software and fail to ever meet requirements, or pass the human written test cases? Users at least will never be impacted by that. Even now, many human-written software projects never get to the stage where they can be used. Is this really a problem, especially if the attempt at AI generation of software is cheap?

rco8786 4y ago
> yes, current AI is not even close This is really my main argument currently... > but it will improve With this being the "if" question. Improve, but improve to a point where we can trust it to do the things with the level of correctness actually required? Unclear so far. GPT3, the state of the art, can't be trusted to answer basic questions correctly (yet). I also don't personally buy into the other notions in these comments that "the future is probabilistic software". I think that's wishful thinking outside of some specific domains and an attempt to bend our actual requirements to meet the capabilities of AI software, rather than the opposite. > pass the human written test cases I'm not super sold on this idea either. It seems reasonably possible that writing the test cases to a level of specification necessary to ensure that correctness we're after is just as much effort as just writing the code. But, time will tell with everything.
- staunton 4y ago
  The way I see it is as expanding the high-level-low-level spectrum of programing tools towards the high end. In very large-scale projects that have many levels of complexity, I would argue "probabilistic software" is already a reality. You don't even try to fix all the bugs or edge cases. You try to minimize the impact of failures while accepting that there will always be failures. Building in logic to fail more gracefully is a big part of that and cost/benefit of such efforts is a probabilistic question, even if it might rarely be framed as such. It's usually much easier (and never harder) to specify what needs to be done than how it should be done. Whether you can trust the result (enough) depends on the application. AI driven development will be applied first wheree errors and failures are least harmful and advance from there. It might take a long time until the degree of correctness improves enough and trust is build around it. After all, some countries' railroad networks still don't use computers but instead have a human map out new routes and schedules on paper to make sure trains don't crash. Nevertheless, the speed of AI development has exceeded (at least my) expectations time and time again in recent years. What might happen is that we eventually get a lot of software that fails more often than now but is a lot cheaper. That would still mean that the new methods are widely adopted. People accept software errors as a part of life already, so even if they get more frequent in less critical applications, we will adapt. What is really dangerous is when the various software components get too fast and complex to understand or control and develop pathological feedback loops in situations that cause real trouble. This kind of thing is a continuum of badness which tops out at "AI taking over and wiping out humanity". Given market incentives driving adoption, which I anticipate to be strong, it's hard to imagine how such risks might be mitigated.
  - rco8786 4y ago
    > You don't even try to fix all the bugs or edge cases. You try to minimize the impact of failures while accepting that there will always be failures Yes but I'll point you back at my original comment about correctness. I've never been on a team that shipped code we knew would do the wrong thing. I ship code with known failure points all the time. But when the code runs to completion, I'm pretty darn sure it's doing the correct thing and we try very, very hard to make sure of that. With AI I am seeing that they can't discern between issues of correctness and issues of failure or availability. It's just like 95% success across all of those spectrums. > What is really dangerous is when the various software components get too fast and complex to understand or control and develop pathological feedback loops in situations that cause real trouble. Yea I agree, that's somewhat scary to think about.

rco8786 4y ago

> yes, current AI is not even close This is really my main argument currently... > but it will improve With this being the "if" question. Improve, but improve to a point where we can trust it to do the things with the level of correctness actually required? Unclear so far. GPT3, the state of the art, can't be trusted to answer basic questions correctly (yet). I also don't personally buy into the other notions in these comments that "the future is probabilistic software". I think that's wishful thinking outside of some specific domains and an attempt to bend our actual requirements to meet the capabilities of AI software, rather than the opposite. > pass the human written test cases I'm not super sold on this idea either. It seems reasonably possible that writing the test cases to a level of specification necessary to ensure that correctness we're after is just as much effort as just writing the code. But, time will tell with everything.

staunton 4y ago
The way I see it is as expanding the high-level-low-level spectrum of programing tools towards the high end. In very large-scale projects that have many levels of complexity, I would argue "probabilistic software" is already a reality. You don't even try to fix all the bugs or edge cases. You try to minimize the impact of failures while accepting that there will always be failures. Building in logic to fail more gracefully is a big part of that and cost/benefit of such efforts is a probabilistic question, even if it might rarely be framed as such. It's usually much easier (and never harder) to specify what needs to be done than how it should be done. Whether you can trust the result (enough) depends on the application. AI driven development will be applied first wheree errors and failures are least harmful and advance from there. It might take a long time until the degree of correctness improves enough and trust is build around it. After all, some countries' railroad networks still don't use computers but instead have a human map out new routes and schedules on paper to make sure trains don't crash. Nevertheless, the speed of AI development has exceeded (at least my) expectations time and time again in recent years. What might happen is that we eventually get a lot of software that fails more often than now but is a lot cheaper. That would still mean that the new methods are widely adopted. People accept software errors as a part of life already, so even if they get more frequent in less critical applications, we will adapt. What is really dangerous is when the various software components get too fast and complex to understand or control and develop pathological feedback loops in situations that cause real trouble. This kind of thing is a continuum of badness which tops out at "AI taking over and wiping out humanity". Given market incentives driving adoption, which I anticipate to be strong, it's hard to imagine how such risks might be mitigated.
- rco8786 4y ago
  > You don't even try to fix all the bugs or edge cases. You try to minimize the impact of failures while accepting that there will always be failures Yes but I'll point you back at my original comment about correctness. I've never been on a team that shipped code we knew would do the wrong thing. I ship code with known failure points all the time. But when the code runs to completion, I'm pretty darn sure it's doing the correct thing and we try very, very hard to make sure of that. With AI I am seeing that they can't discern between issues of correctness and issues of failure or availability. It's just like 95% success across all of those spectrums. > What is really dangerous is when the various software components get too fast and complex to understand or control and develop pathological feedback loops in situations that cause real trouble. Yea I agree, that's somewhat scary to think about.