4 ms·

My first thought. I'm a licensed Professional Engineer. If I build something and "stamp" it for approval I am legally held responsible for negligence of work.

My first thought. I'm a licensed Professional Engineer. If I build something and "stamp" it for approval I am legally held responsible for negligence of work. https://www.nspe.org/resources/ethics/code-ethics https://www.nspe.org/resources/ethics/code-ethics This is something I've been wondering about the Software Engineering profession. As software becomes more "blackbox" or AI if you will should this profession be required for licensure as well. Especially if the software design is used in public service.

missosoup 7y ago
NASA has lost multiple extremely expensive missions due to software errors despite investing hundreds of millions of dollars and dedicating entire teams to software safety. You want to put a guarantee stamp on something at the risk of going to jail for missing a bounds check at some point in your life? Go ahead, I'll pass. This kind of requirement would kill the industry overnight. Software is not engineering. Software engineering is a hilarious oxymoron, and I say this as an accredited engineer. There is no way to guarantee that software will work 'properly', and 'properly' is only a small subset of 'as intended' which is a small subset of 'as envisioned'. It could be that software never becomes engineering. And until it does, holding it to the same standards as engineering is pointless.
- criddell 7y ago
  > Software is not engineering. It can be. There are ways to prove a program is correct.
  - missosoup 7y ago
    There are ways to prove the program will reach certain outputs within a certain envelope of inputs under a certain set of constraints. This in itself is rather limited. For example, a formal prover would know nothing about something like rowhammer, and would happily accept code that seems to work as intended but also executes a rowhammer attack on the side which completely changes the output. Ignoring that, this definition of correct is entirely divorced from the level where requirements exist, which is as envisioned. An error in requirements or an error in design can still very well yield formally 'correct' code. There's no way around this today, and there's a pretty strong argument that this is an inherent problem to which no solution exists. This is precisely why NASA has lost missions despite being one of the most prolific users of formal provers. They proved the code works as designed, but that says nothing of the design being correct, which says nothing of the intent of the design being correct. Also see: https://en.wikipedia.org/wiki/Instrumental_convergence#Paperclip_maximizer https://en.wikipedia.org/wiki/Instrumental_convergence#Paper...
    - criddell 7y ago
      I think saying software can't be correct because of things like row hammer is like saying a wood structure can't be safe because of fire or termites.
      - missosoup 7y ago
        I think myopically selecting the easiest to dismiss part of what you're replying to, a part that was a deliberately simple example of where provers can and have failed, is not arguing in good faith. The rest of the argument stands even in a hypothetical world where provers are prefect.
        criddell 7y ago
        My argument holds even when you extend it to the rest of what you say. Mechanical and electrical systems can also be built correctly to an specification that's ultimately inadequate.
criddell 7y ago
If being able to kill is part of the design, then you aren't negligent if it works as intended.

missosoup 7y ago

NASA has lost multiple extremely expensive missions due to software errors despite investing hundreds of millions of dollars and dedicating entire teams to software safety. You want to put a guarantee stamp on something at the risk of going to jail for missing a bounds check at some point in your life? Go ahead, I'll pass. This kind of requirement would kill the industry overnight. Software is not engineering. Software engineering is a hilarious oxymoron, and I say this as an accredited engineer. There is no way to guarantee that software will work 'properly', and 'properly' is only a small subset of 'as intended' which is a small subset of 'as envisioned'. It could be that software never becomes engineering. And until it does, holding it to the same standards as engineering is pointless.

criddell 7y ago
> Software is not engineering. It can be. There are ways to prove a program is correct.
- missosoup 7y ago
  There are ways to prove the program will reach certain outputs within a certain envelope of inputs under a certain set of constraints. This in itself is rather limited. For example, a formal prover would know nothing about something like rowhammer, and would happily accept code that seems to work as intended but also executes a rowhammer attack on the side which completely changes the output. Ignoring that, this definition of correct is entirely divorced from the level where requirements exist, which is as envisioned. An error in requirements or an error in design can still very well yield formally 'correct' code. There's no way around this today, and there's a pretty strong argument that this is an inherent problem to which no solution exists. This is precisely why NASA has lost missions despite being one of the most prolific users of formal provers. They proved the code works as designed, but that says nothing of the design being correct, which says nothing of the intent of the design being correct. Also see: https://en.wikipedia.org/wiki/Instrumental_convergence#Paperclip_maximizer https://en.wikipedia.org/wiki/Instrumental_convergence#Paper...
  - criddell 7y ago
    I think saying software can't be correct because of things like row hammer is like saying a wood structure can't be safe because of fire or termites.
    - missosoup 7y ago
      I think myopically selecting the easiest to dismiss part of what you're replying to, a part that was a deliberately simple example of where provers can and have failed, is not arguing in good faith. The rest of the argument stands even in a hypothetical world where provers are prefect.
      - criddell 7y ago
        My argument holds even when you extend it to the rest of what you say. Mechanical and electrical systems can also be built correctly to an specification that's ultimately inadequate.

criddell 7y ago

If being able to kill is part of the design, then you aren't negligent if it works as intended.