6 ms·

External audits are already required for finances. Safety critical systems are even more important and should be subject to even more scrutiny as a result.

InclinedPlane 11y ago
Absolutely. But finances are pretty standardized, software is vastly more complex. Audits are a good idea, but it's an incredibly hard problem.
- tluyben2 11y ago
  Maybe parent meant the software in finances, because that also requires external audits to some extend.
- DrJokepu 11y ago
  I think a better approach would be requiring that developers (and their managers and testers etc.) working on software that could kill or injure people if it malfunctioned have some sort of a professional license, that would be granted and revoked similarly to how medical and engineering licenses are granted and revoked.
  - st0p 11y ago
    I'm not opposing this idea, but I'm not sure it would have helped in the VW case. There were some people (engineers? Managers?) who were cheating and they knew that what they were doing was wrong. I don't believe a license would have changed that.
    - michaelt 11y ago
      If a developer is asked to do something obviously wrong they might not feel they can refuse, because they can be replaced with someone willing to do it. If an architect is asked to design a bridge that isn't safe they can refuse, secure in the knowledge they can't be replaced with someone willing to do it, as no licensed architect will knowingly design an unsafe bridge. Of course, a licensing scheme would probably have a bunch of disadvantages.
    - ehvatum 11y ago
      Perhaps the angle is that this would constitute ethical turpitude sure to cause loss of license and ejection from one's specialty.
    - mtbcoder 11y ago
      Perhaps the threat of having their license pulled, thereby nullifying potential future employment might have caused them to think twice about wilfully cheating emissions controls?
      - thomnottom 11y ago
        While the FDA may not be a great regulatory group, if someone at a pharmaceutical were found to cheat like this they could potentially be barred from working in the industry again. This works in some cases, at least in theory.
    - acdha 11y ago
      Other people have raised the question of how well the prospect of losing a license would act as a deterrent. One other aspect which might be even stronger would be if the professional organization had a role not unlike a union in protecting its members’ professional decisions. Imagine if you worked at VW and your boss told you to make a change which affected safety, emissions, etc. – how different might your reaction be if you know that if you refused or reported it to the appropriate regulators and there were repercussions the Bitpackers Guild could provide legal representation and expert witnesses for you, stage a strike where no licensed engineer would work for an irresponsible company, or simply ensure a lot of publicity? Suddenly it's not “go lean on Sally until she gives the engineering sign-off. She can't afford to quit until her kid's out of college” but “do we want a team of professional engineers to hold a press conference saying we're cutting corners over our experts' judgement?” There are certainly potential downsides but … anyone who drives a car, uses medical equipment, etc. might reasonably conclude they're worth it, particularly if the system was structured to focus on transparency and due process rather than the pathology some unions are prone to where members are always defended even when they're in the wrong.
  - redcalx 11y ago
    > some sort of a professional license Sure, as per the construction industry. Or perhaps simply the threat of being prosecuted for manslaughter or bodily harm, etc?
  - cuckcuckspruce 11y ago
    While I don't agree with the license requirement the least we could require is publication of the source code and validation by an industry body made up for subject matter experts for safety critical code.
- w_t_payne 11y ago
  Even the possibility of an external source code, revision history, requirements etc... audit would change working practices dramatically ... particularly if there were legal penalties against developers found responsible for introducing bugs.
  - logfromblammo 11y ago
    There's no way in Hell that I will consent to be held responsible for the output if I do not have full control over the inputs. If I am an employee of the company, and someone else is telling me what to do for my job, and particularly if they are telling me how to do my job, they must necessarily share responsibility for anything that I do pursuant to obeying those instructions. And threat of retribution leads to stupid practices: public void CoverYourAss() { try { int x = 0; } catch { throw; } } This is a simplified example of a real-world coding standard. At one of my former workplaces, everything had to be wrapped in a try-catch block, including statements that would only ever generate run-time exceptions, like out-of-memory exceptions. It didn't matter if you re-threw the exception you just caught. You just had to make sure the try-catch was there. In every function. Or you're fired. I am not making this up. If the software ever crashed to desktop for any reason, including a bad memory module in the computer running it, or someone nuking parts of the filesystem while it was running, or even a bullet striking the motherboard, someone was getting blamed for it on the development team, and fired. As it would be a witch hunt anyway, the inquisition squad would obviously look at the code written by those most threatening to them, or least popular, or both, before anyone else, and seize upon any irregularity to lay blame. You'd better believe I was sending out resumes the day I found out about that. I can only imagine how bad it would be if the penalty was to be fired plus arrested and/or sued.
    - JoBrad 11y ago
      But if there were a standard set of industry-specific tests that the program had to comply with, it's not like it would just be on you.
      - 11y ago
- rayiner 11y ago
  That's true, and it isn't hard problem. But note that audits are also a hard problem. Auditing teams don't go through and reconcile every transaction. They conduct spot checks of sample transactions and scrutinize controls, and aggressively follow up when any failure of controls is observed. I think a lot of those concepts could be applied to code audits.
simonh 11y ago
The thing to bear in mind about banks is that nowadays they are really computerized service companies that happen to specialize in finance. Competent tech isn't just a competitive edge, it's fundamental to being able to operate at all.
- SideburnsOfDoom 11y ago
  > about banks ... competent tech isn't just a competitive edge, it's fundamental to being able to operate at all. Automobiles are going along that curve too. Maybe this whole VW scandal is an indication of this change? edit and the Toyota firmware scandal in 2013 has similar implications: http://www.latimes.com/business/autos/la-fi-hy-toyota-damages-20131026-story.html http://www.latimes.com/business/autos/la-fi-hy-toyota-damage...
- thomnottom 11y ago
  That may be true of the largest banks, but there are hundreds of others that are running on fairly incompetent tech. Either way, external audits have been a part of them for much longer than the computerization has been.
- branchless 11y ago
  I've worked in two of the biggest US banks for years and I can confidently say both important departments were far from completely understood by most people working there, let alone outsiders doing an audit. Most of the complexity is in the messages passed between systems. In a code audit it's fine to remove sending tag X by system A but then you find out when you hit prod system B used tag X and now it's not there which causes a problem in system C. And that would be a simplified example. I'd be disturbed by any outsider claiming they understand the dynamics of multiple systems with millions of lines of code which often have feedback loops.
- SEJeff 11y ago
  Honest question, have you ever worked for a bank? Many still run on AS/400 mainframes in the backends. Knowing what I know, that comment actually couldn't be further from the truth. Banks are insanely risk adverse, and rightfully so. If it works perfectly, they absolutely will not change it, even if the newer shinier tech has real benefits. Source: I don't work for a bank, but have worked in electronic trading for the past 8 years and work directly with the (in)competent tech teams from other banks to clear and reconcile trading bits.
ArkyBeagle 11y ago
It'll take a lot of resources to develop institutions to do this. It'll take a long time to debug these institutions. We're still arguing over "goto considered harmful", "the billion dollar mistake" and how toolchains influence code-safety-behavior. And there exists a class of CASE tools from the '90s that, while being somewhat clunky, addressed some of the transparency issues. But those are hardly even relevant any more. And to make it worse - the final configuration of the system matters just as much as whether the code bases for the individual nodes have passed an audit. This is especially true of CAN networks.
AJ007 11y ago
The irony is that instead of incentivizing auditing of system, hackers and security researchers put themselves at huge risk whenever they look for, find, and report vulnerabilities. The companies that have bounties are doing it right.

InclinedPlane 11y ago

Absolutely. But finances are pretty standardized, software is vastly more complex. Audits are a good idea, but it's an incredibly hard problem.

tluyben2 11y ago
Maybe parent meant the software in finances, because that also requires external audits to some extend.
DrJokepu 11y ago
I think a better approach would be requiring that developers (and their managers and testers etc.) working on software that could kill or injure people if it malfunctioned have some sort of a professional license, that would be granted and revoked similarly to how medical and engineering licenses are granted and revoked.
- st0p 11y ago
  I'm not opposing this idea, but I'm not sure it would have helped in the VW case. There were some people (engineers? Managers?) who were cheating and they knew that what they were doing was wrong. I don't believe a license would have changed that.
  - michaelt 11y ago
    If a developer is asked to do something obviously wrong they might not feel they can refuse, because they can be replaced with someone willing to do it. If an architect is asked to design a bridge that isn't safe they can refuse, secure in the knowledge they can't be replaced with someone willing to do it, as no licensed architect will knowingly design an unsafe bridge. Of course, a licensing scheme would probably have a bunch of disadvantages.
  - ehvatum 11y ago
    Perhaps the angle is that this would constitute ethical turpitude sure to cause loss of license and ejection from one's specialty.
  - mtbcoder 11y ago
    Perhaps the threat of having their license pulled, thereby nullifying potential future employment might have caused them to think twice about wilfully cheating emissions controls?
    - thomnottom 11y ago
      While the FDA may not be a great regulatory group, if someone at a pharmaceutical were found to cheat like this they could potentially be barred from working in the industry again. This works in some cases, at least in theory.
  - acdha 11y ago
    Other people have raised the question of how well the prospect of losing a license would act as a deterrent. One other aspect which might be even stronger would be if the professional organization had a role not unlike a union in protecting its members’ professional decisions. Imagine if you worked at VW and your boss told you to make a change which affected safety, emissions, etc. – how different might your reaction be if you know that if you refused or reported it to the appropriate regulators and there were repercussions the Bitpackers Guild could provide legal representation and expert witnesses for you, stage a strike where no licensed engineer would work for an irresponsible company, or simply ensure a lot of publicity? Suddenly it's not “go lean on Sally until she gives the engineering sign-off. She can't afford to quit until her kid's out of college” but “do we want a team of professional engineers to hold a press conference saying we're cutting corners over our experts' judgement?” There are certainly potential downsides but … anyone who drives a car, uses medical equipment, etc. might reasonably conclude they're worth it, particularly if the system was structured to focus on transparency and due process rather than the pathology some unions are prone to where members are always defended even when they're in the wrong.
- redcalx 11y ago
  > some sort of a professional license Sure, as per the construction industry. Or perhaps simply the threat of being prosecuted for manslaughter or bodily harm, etc?
- cuckcuckspruce 11y ago
  While I don't agree with the license requirement the least we could require is publication of the source code and validation by an industry body made up for subject matter experts for safety critical code.
w_t_payne 11y ago
Even the possibility of an external source code, revision history, requirements etc... audit would change working practices dramatically ... particularly if there were legal penalties against developers found responsible for introducing bugs.
- logfromblammo 11y ago
  There's no way in Hell that I will consent to be held responsible for the output if I do not have full control over the inputs. If I am an employee of the company, and someone else is telling me what to do for my job, and particularly if they are telling me how to do my job, they must necessarily share responsibility for anything that I do pursuant to obeying those instructions. And threat of retribution leads to stupid practices: public void CoverYourAss() { try { int x = 0; } catch { throw; } } This is a simplified example of a real-world coding standard. At one of my former workplaces, everything had to be wrapped in a try-catch block, including statements that would only ever generate run-time exceptions, like out-of-memory exceptions. It didn't matter if you re-threw the exception you just caught. You just had to make sure the try-catch was there. In every function. Or you're fired. I am not making this up. If the software ever crashed to desktop for any reason, including a bad memory module in the computer running it, or someone nuking parts of the filesystem while it was running, or even a bullet striking the motherboard, someone was getting blamed for it on the development team, and fired. As it would be a witch hunt anyway, the inquisition squad would obviously look at the code written by those most threatening to them, or least popular, or both, before anyone else, and seize upon any irregularity to lay blame. You'd better believe I was sending out resumes the day I found out about that. I can only imagine how bad it would be if the penalty was to be fired plus arrested and/or sued.
  - JoBrad 11y ago
    But if there were a standard set of industry-specific tests that the program had to comply with, it's not like it would just be on you.
    - 11y ago
rayiner 11y ago
That's true, and it isn't hard problem. But note that audits are also a hard problem. Auditing teams don't go through and reconcile every transaction. They conduct spot checks of sample transactions and scrutinize controls, and aggressively follow up when any failure of controls is observed. I think a lot of those concepts could be applied to code audits.

simonh 11y ago

The thing to bear in mind about banks is that nowadays they are really computerized service companies that happen to specialize in finance. Competent tech isn't just a competitive edge, it's fundamental to being able to operate at all.

SideburnsOfDoom 11y ago
> about banks ... competent tech isn't just a competitive edge, it's fundamental to being able to operate at all. Automobiles are going along that curve too. Maybe this whole VW scandal is an indication of this change? edit and the Toyota firmware scandal in 2013 has similar implications: http://www.latimes.com/business/autos/la-fi-hy-toyota-damages-20131026-story.html http://www.latimes.com/business/autos/la-fi-hy-toyota-damage...
thomnottom 11y ago
That may be true of the largest banks, but there are hundreds of others that are running on fairly incompetent tech. Either way, external audits have been a part of them for much longer than the computerization has been.
branchless 11y ago
I've worked in two of the biggest US banks for years and I can confidently say both important departments were far from completely understood by most people working there, let alone outsiders doing an audit. Most of the complexity is in the messages passed between systems. In a code audit it's fine to remove sending tag X by system A but then you find out when you hit prod system B used tag X and now it's not there which causes a problem in system C. And that would be a simplified example. I'd be disturbed by any outsider claiming they understand the dynamics of multiple systems with millions of lines of code which often have feedback loops.
SEJeff 11y ago
Honest question, have you ever worked for a bank? Many still run on AS/400 mainframes in the backends. Knowing what I know, that comment actually couldn't be further from the truth. Banks are insanely risk adverse, and rightfully so. If it works perfectly, they absolutely will not change it, even if the newer shinier tech has real benefits. Source: I don't work for a bank, but have worked in electronic trading for the past 8 years and work directly with the (in)competent tech teams from other banks to clear and reconcile trading bits.

ArkyBeagle 11y ago

It'll take a lot of resources to develop institutions to do this. It'll take a long time to debug these institutions. We're still arguing over "goto considered harmful", "the billion dollar mistake" and how toolchains influence code-safety-behavior. And there exists a class of CASE tools from the '90s that, while being somewhat clunky, addressed some of the transparency issues. But those are hardly even relevant any more. And to make it worse - the final configuration of the system matters just as much as whether the code bases for the individual nodes have passed an audit. This is especially true of CAN networks.

AJ007 11y ago

The irony is that instead of incentivizing auditing of system, hackers and security researchers put themselves at huge risk whenever they look for, find, and report vulnerabilities. The companies that have bounties are doing it right.