4 ms·

Agreed. We may imagine ourselves very smart but in my experience most people do not understand probability and statistics well. Is 90% accuracy good enough? Is

Agreed. We may imagine ourselves very smart but in my experience most people do not understand probability and statistics well. Is 90% accuracy good enough? Is 95%? 99%? 99.9%? No matter the answer, you have to tolerate errors. Now your stakeholders have to tolerate errors. Are they going to accept errors just because "Software 2.0" is here and that's what we all have to live with? Nope.

staunton 4y ago
All software is riddled with errors and for most purposes that's fine. Any developer, stakeholder, whatever, who thinks otherwise is living in a parallel universe. It may be possible in the future to have "for all practical purposes flawless" software, which might make sense for select special applications. That would be a new thing though, rather than something we have and could lose due to adopting AI development.
- marginalia_nu 4y ago
  Formal verification of software has been around for a while though.
  - staunton 4y ago
    Which has almost never been used to build software anyone ever used... That might be changing very slowly
- rco8786 4y ago
  > All software is riddled with errors and for most purposes that's fine. This is typically not that true when it comes to correctness. Most software does the correct thing in the eyes of the user, nearly 100% of the time. And when it doesn't, the bug gets fixed and that edge case is corrected for every other user going forward. AI generated software, from what I've seen, has a wide range of errors in correctness (along with all the other errors that you mentioned all software having..which is true). Like it literally just does the wrong thing given what the user is expecting it to do. The path toward iteratively improving and getting it to an acceptable level of correctness for any given application might be there, but so far I have not seen it.
  - staunton 4y ago
    That's an interesting difference but I'm not convinced it's valid. Perhaps you can explain what you mean in more detail? Let's say the software is good enough if it does the right thing 99.9% of the time it is used. I take it, you're saying that if an AI starts modifying it and only writes correct code in 99.9% of cases (yes, current AI is not even close, but it will improve), that makes it worse because the software might start failing completely. However, if you have proper tests and release management, such obvious flaws will quickly be detected and fixed or rolled back. For most applications that seems pretty much equivalent to what we have now. The other case is software that is completely AI generated and cannot reasonably be modified by humans anymore. In that case, again, you have tests and a sane deployment strategy that mitigates failures to a sufficient degree depending on application. So the only issue is when you start making a completely AI generated software and fail to ever meet requirements, or pass the human written test cases? Users at least will never be impacted by that. Even now, many human-written software projects never get to the stage where they can be used. Is this really a problem, especially if the attempt at AI generation of software is cheap?
    - rco8786 4y ago
      > yes, current AI is not even close This is really my main argument currently... > but it will improve With this being the "if" question. Improve, but improve to a point where we can trust it to do the things with the level of correctness actually required? Unclear so far. GPT3, the state of the art, can't be trusted to answer basic questions correctly (yet). I also don't personally buy into the other notions in these comments that "the future is probabilistic software". I think that's wishful thinking outside of some specific domains and an attempt to bend our actual requirements to meet the capabilities of AI software, rather than the opposite. > pass the human written test cases I'm not super sold on this idea either. It seems reasonably possible that writing the test cases to a level of specification necessary to ensure that correctness we're after is just as much effort as just writing the code. But, time will tell with everything.
      - staunton 4y ago
- madeforhnyo 4y ago
  It might be easier for an engineer to fix a bug by changing some lines of text than readjust a neural network, for the time being at least. As pointed out, there are already formal languages that allow formal verification like B [0] notably for like-critical systems. [0] https://en.wikipedia.org/wiki/B-Method https://en.wikipedia.org/wiki/B-Method
- hwayne 4y ago
  There's a difference between errors in UI/reliability/performance/etc and errors in business logic. When there's an error in the business logic, then heads roll.
duckmysick 4y ago
> No matter the answer, you have to tolerate errors. We already do that in manufacturing. Physical parts are imperfect and we design with such variation in mind.
badloginagain 4y ago
The entire field of Service Reliability is dedicated to finding the exact boundaries of acceptable errors, and defining the response function when those boundaries are crossed. I would assume this actually gels very nicely with neural nets since its constantly optimizing for fitness. Hell in theory you could bake in your SLA/SLIs into the models to self correct? Give the model direct feedback that its unfit?

staunton 4y ago

All software is riddled with errors and for most purposes that's fine. Any developer, stakeholder, whatever, who thinks otherwise is living in a parallel universe. It may be possible in the future to have "for all practical purposes flawless" software, which might make sense for select special applications. That would be a new thing though, rather than something we have and could lose due to adopting AI development.

marginalia_nu 4y ago
Formal verification of software has been around for a while though.
- staunton 4y ago
  Which has almost never been used to build software anyone ever used... That might be changing very slowly
rco8786 4y ago
> All software is riddled with errors and for most purposes that's fine. This is typically not that true when it comes to correctness. Most software does the correct thing in the eyes of the user, nearly 100% of the time. And when it doesn't, the bug gets fixed and that edge case is corrected for every other user going forward. AI generated software, from what I've seen, has a wide range of errors in correctness (along with all the other errors that you mentioned all software having..which is true). Like it literally just does the wrong thing given what the user is expecting it to do. The path toward iteratively improving and getting it to an acceptable level of correctness for any given application might be there, but so far I have not seen it.
- staunton 4y ago
  That's an interesting difference but I'm not convinced it's valid. Perhaps you can explain what you mean in more detail? Let's say the software is good enough if it does the right thing 99.9% of the time it is used. I take it, you're saying that if an AI starts modifying it and only writes correct code in 99.9% of cases (yes, current AI is not even close, but it will improve), that makes it worse because the software might start failing completely. However, if you have proper tests and release management, such obvious flaws will quickly be detected and fixed or rolled back. For most applications that seems pretty much equivalent to what we have now. The other case is software that is completely AI generated and cannot reasonably be modified by humans anymore. In that case, again, you have tests and a sane deployment strategy that mitigates failures to a sufficient degree depending on application. So the only issue is when you start making a completely AI generated software and fail to ever meet requirements, or pass the human written test cases? Users at least will never be impacted by that. Even now, many human-written software projects never get to the stage where they can be used. Is this really a problem, especially if the attempt at AI generation of software is cheap?
  - rco8786 4y ago
    > yes, current AI is not even close This is really my main argument currently... > but it will improve With this being the "if" question. Improve, but improve to a point where we can trust it to do the things with the level of correctness actually required? Unclear so far. GPT3, the state of the art, can't be trusted to answer basic questions correctly (yet). I also don't personally buy into the other notions in these comments that "the future is probabilistic software". I think that's wishful thinking outside of some specific domains and an attempt to bend our actual requirements to meet the capabilities of AI software, rather than the opposite. > pass the human written test cases I'm not super sold on this idea either. It seems reasonably possible that writing the test cases to a level of specification necessary to ensure that correctness we're after is just as much effort as just writing the code. But, time will tell with everything.
    - staunton 4y ago
madeforhnyo 4y ago
It might be easier for an engineer to fix a bug by changing some lines of text than readjust a neural network, for the time being at least. As pointed out, there are already formal languages that allow formal verification like B [0] notably for like-critical systems. [0] https://en.wikipedia.org/wiki/B-Method https://en.wikipedia.org/wiki/B-Method
hwayne 4y ago
There's a difference between errors in UI/reliability/performance/etc and errors in business logic. When there's an error in the business logic, then heads roll.

duckmysick 4y ago

> No matter the answer, you have to tolerate errors. We already do that in manufacturing. Physical parts are imperfect and we design with such variation in mind.

badloginagain 4y ago

The entire field of Service Reliability is dedicated to finding the exact boundaries of acceptable errors, and defining the response function when those boundaries are crossed. I would assume this actually gels very nicely with neural nets since its constantly optimizing for fitness. Hell in theory you could bake in your SLA/SLIs into the models to self correct? Give the model direct feedback that its unfit?