7 ms·

The dangerous unreliability of software is an accountability problem, not a technology problem. Engineering disciplines are regulated so that engineers are hel

The dangerous unreliability of software is an accountability problem, not a technology problem. Engineering disciplines are regulated so that engineers are held personally and even criminally responsible if they endanger property or lives. Software, being the newcomer, has no such regulation. Instead we have people trying to claim that software deserves a free pass from due diligence because "software is hard," when in reality other engineering disciplines deal with unexpected failure modes all the time. The only difference is that these other engineers are motivated to be significantly more thorough about their jobs.

always_good 9y ago
>The only difference is that these other engineers are motivated to be significantly more thorough about their jobs. Well, it's a whole ecosystem problem. Most of the places I've worked, security was just supposed to be something you learned and credentialized on your own, and no time or resources were deliberately allocated to it. In fact, you're likely seen as an underperformer if you slow down to get things right or to refactor bad code. And when software does break, `git blame` tends to accuse the developer that was in the worst position to fix a ship that had begun sinking long ago. I'm not sure how engineer culpability works in other fields, but surely it's always coincided by bad process and management at which point the whole company is responsible.
- throwaway020217 9y ago
  In professional engineering the accountability comes with a stamp, and a professional ethics requirement to report up the chain if a critical problem has been found that could impact public safety. This includes whistleblowing if management ignores the problem. There are clear guidelines as to what types of products and services require a p.eng.'s stamp. There are also many professional engineers out there who's stamp has never touched ink. For example, the engineers drafting and churning out spreadsheets in the engineering sweatshops aren't accountable for the safety of the product, you'll have one or more engineers providing their stamp for that group's work. If we were to implement a similar model on software development, most developers would not have personal liability for work they do e.g. for a company or open source project. What is needed is better classification of software products/activities that require a software engineer's stamp[1], and a corresponding professional association. So no, developers should not be accountable. However, certain safety-critical products - where safety can be defined in various ways expanding the current relatively small set of NASA, aviation and health-care software - need engineers overseeing design, development, and QC to be accountable. And yes, this means that the self-taught developer, no matter how talented, will not be able to release certain types of software without some sort of oversight. [1]As a side-note - not everyone can call themselves an 'engineer' in other fields without the proper credentials and professional membership. I see developers calling themselves software engineers all the time though they don't have the chops. This would, of course, need to change. Updated to change asterisk to brackets
vtange 9y ago
I'm pretty sure less people would be rushing into the software engineer profession if they knew it came with the liabilities you described. That would drive up the prices of engineers.. and I'm pretty sure the industry in general doesn't want that. Maybe that's why the free pass mentality exists
- emodendroket 9y ago
  Or maybe the "free pass mentality" exists because development doesn't happen in a vacuum and errors don't necessarily find their way into programs because the programmers are stupid or careless.
tener 9y ago
Broken software does not typically kill people or makes you loose money (time or some other resources, maybe). In contrast bridges that fall apart tend to kill people in a nasty way. They are also quite hard to reboot.
- Bartweiss 9y ago
  An excellent framing I saw recently was "I can tell the people who write these articles don't actually want secure code, because they aren't paying me to write it." There's plenty of carefully-written, rigorously-tested code out there. Want to write software for air travel, drug manufacturing, or space flight? You'll have no choice but to write safe, first-rate code. Want a smartphone weather app written to those standards? Well, for $50 and a feature set five years behind the state of the art, you can have it.
  - blackth0rn 9y ago
    Isn't this a tooling and knowledge issue? If I was as productive with TLA+ or model based design as I am with React-Native/Ionic/Swift etc. then we could have a smartphone weather app written to those standards for the same price.
    - hwayne 9y ago
      While there's plenty of room for improvement, I legitimately believe that TLA+ is now in a place where it's convenient enough to use in conjunction with Ionic/Swift.
    - deleted 9y ago
      [deleted]
- IncRnd 9y ago
  Broken software has the capability to regularly kill people or cause people to lose money. It happens too often.
emodendroket 9y ago
Without radical, NASA-style changes to typical development, where each line spends much more time being audited than being written, it is completely unreasonable to expect that kind of liability from developers.
- amiga-workbench 9y ago
  Ditch the dumb deadlines and the tight budgets and it would become a hell of a lot more feasible. I WISH I had the time and budget to put due diligence into my work.
  - emodendroket 9y ago
    You have to think about what you're building too. Feature release would come much slower than it does today. Reasonable? If you're writing software that is absolutely safety-critical, that's likely. If you're writing iTunes, well, I have my doubts.
    - Bartweiss 9y ago
      It's not like reliable, secure software doesn't exist. It's just limited to spaces where people are willing to use years-old tech, forgo new features, and pay fortunes for auditing. Air travel, industrial machinery, medicine, and so on. Almost like the cost-benefit actually doesn't support making the development of trivial software ten times slower and more expensive... If there's a problem here, it's business externalities (Equifax won't pay adequately for screwing people), not engineering impossibility (the rest of us know how to patch Apache).
      - emodendroket 9y ago
        I mostly agree, but this article cites the national 911 software, which arguably should be treated as a critical system, failing, and some medical devices whose software killed people.
        9y ago
0xdeadbeefbabe 9y ago
> The only difference is that these other engineers are motivated to be significantly more thorough about their jobs. What evidence supports this as the only difference? I would argue these other engineers have significantly smaller jobs to do, so it's easier to be thorough. Edit: Why are you so keen to fix a problem you don't understand? Are you a programmer?
Bartweiss 9y ago
> The only difference is that these other engineers are motivated to be significantly more thorough about their jobs. Hang on. Sure, all engineering has hurdles. But how many engineering projects face regular enemy action? The last time somebody actively tried to compromise a system I was building was two weeks ago. How many engineers live like that? We hold civil engineers responsible when bridges collapse unprompted. We also hold them responsible if their bridges fail from some minor, anticipated harm like vandalism or a car crash. We don't hold them responsible for building bridges that can be blown up. If we want to hold software developers accountable for downtime, unprompted data leaks, and so on, fine. We should do it proportionate to the damage caused - there's no point in pretending that a shoddy smartphone game is as bad as a shoddy bridge - but fine. But let's not pretend that software has the same difficulties as every other type of engineering. Most engineering doesn't face an epidemic of people actively trying to disable safety precautions to hurt people, and a lot of it reliably fails as soon as it does face enemy action. Punishing every engineer who ever gets outwitted (say, by a leaked NSA-developed vulnerability) is absurd.
- scintill76 9y ago
  Most of the article, and the comment you're replying to, wasn't about hackers, though. It gave the example of people dying because the Therac machine or a Toyota car responded wrongly to non-malicious, normal user input, like a "bridge collapsing unprompted" in normal use.
  - Bartweiss 9y ago
    Yeah, I agree about the article. I wouldn't offer this criticism to the main essay - fundamental bugs in critical systems are absolutely an engineering failure, at least in the sense that we should look for engineering practices to solve them. But with the post I responded to? I see this idea of "it's because engineers don't care, so give them liability" a lot, and frankly it's stupid. It neglects both cost-benefit differences and any interest in the question of hacking, and then goes on to pretend that you can fix bad systemic pressures by beating up the guys at the bottom. The difficulty of writing good code is not a function of how heavily you punish bad code. I flatly don't believe that the Therac bug would have been caught by personal liability for the bug; either the code would have been cancelled altogether to avoid liability or the bug would have gone on unabated. Realistically, criminal liability for lethal bugs mostly sounds like a way to drive talented devs away from high-stakes applications and leave those apps to be handled by people desperate enough to run the risk, or by international shops operating outside the reach of prosecution. I don't love the thought of a shop unwilling to use git because it creates personal responsibility. "Oh, we just pass code around on thumbdrives, I can't remember who wrote that critical line." I'd shrug and let the whole thing go, but the rumblings of "regulate software" keep getting stronger from a lot of directions. I don't really want to wake up in a decade and find out that degree requirements and expensive-but-useless licensing have become mandatory, and it feels like pretending bugs come from lazy devs is another snowflake in that avalanche.
drinchev 9y ago
Regulation on programmers is not the solution. Regulation on companies might be. I'm usually really disappointed when a cheap re-seller takes a government purchase of a 911 or 112 ( here in EU ) and then hires a subcontractor, which hires a subcontractor, which ... That's how crappy software gets into production.
imh 9y ago
Can't it be both a technology problem and an accountability one? The tech problem might take ages to get solved without accountability forcing it. If we were accountable, we'd be forced to make better tools for writing reliable software more easily. That said, there are simpler low handing fruit "tools" like code review, safe git practices, etc that still aren't followed everywhere. Ignoring those low hanging fruit does point to accountability/prioritization being a cause of low reliability.
crdoconnor 9y ago
>The dangerous unreliability of software is an accountability problem, not a technology problem. Engineering disciplines are regulated so that engineers are held personally and even criminally responsible if they endanger property or lives. Software, being the newcomer, has no such regulation. Regulation like that tends to follow after a series of disasters causing massive loss of life. Software hasn't yet managed to achieve that.
wpietri 9y ago
Motivation seems far from the only difference. If we divide making into "builds with atoms" and "builds with bits", both sorts of things are generally unregulated. It's when you get into "builds life-critical things with atoms" that you see significant regulation. If you build a house or an office building, your engineering is regulated. If you build a doghouse or an office filing cabinet, it isn't. I agree that societal accountability for engineers working on life-critical system, software and hardware both, is vital. For most atom-making and bit-making, though, I think stringent professional licensing is overkill. Instead I'd rather we started with accountability for the people who have much more control over the outcomes than individual software engineers: executives and managers.
- IncRnd 9y ago
  > If you build a house or an office building, your engineering is regulated. If you build a doghouse or an office filing cabinet, it isn't. A filing cabinet must be fit for the purpose it is used, or there will be liability incurred. OSHA would frown on keeping hazardous chemicals in your filing cabinet. Software should also be fit for the purpose it is marketed and used. There are many kinds of paper, but paper being used improperly, by storing a list of credit card numbers on it, is quite similar to how many pieces of software are written.
FLUX-YOU 9y ago
You can't drop all of this responsibility on developers. You need people checking behind them. But if we want to pay developers like surgeons and have malpractice insurance, all the better, I guess.
hasenj 9y ago
You can definitely do that. You just have to pay them 20x more money and give them 50x more time and give them absolutely all the tools they need. Best possible hardware. Best possible software. Fastest possible internet connections. etc. Show me an employer willing to do that and I will immediately sign up. Most companies want to do the exact opposite. Spend the least amount possible, because, you know, the budget is tight and the business side wants all these features ready yesterday, nay, last week.

always_good 9y ago

>The only difference is that these other engineers are motivated to be significantly more thorough about their jobs. Well, it's a whole ecosystem problem. Most of the places I've worked, security was just supposed to be something you learned and credentialized on your own, and no time or resources were deliberately allocated to it. In fact, you're likely seen as an underperformer if you slow down to get things right or to refactor bad code. And when software does break, `git blame` tends to accuse the developer that was in the worst position to fix a ship that had begun sinking long ago. I'm not sure how engineer culpability works in other fields, but surely it's always coincided by bad process and management at which point the whole company is responsible.

throwaway020217 9y ago
In professional engineering the accountability comes with a stamp, and a professional ethics requirement to report up the chain if a critical problem has been found that could impact public safety. This includes whistleblowing if management ignores the problem. There are clear guidelines as to what types of products and services require a p.eng.'s stamp. There are also many professional engineers out there who's stamp has never touched ink. For example, the engineers drafting and churning out spreadsheets in the engineering sweatshops aren't accountable for the safety of the product, you'll have one or more engineers providing their stamp for that group's work. If we were to implement a similar model on software development, most developers would not have personal liability for work they do e.g. for a company or open source project. What is needed is better classification of software products/activities that require a software engineer's stamp[1], and a corresponding professional association. So no, developers should not be accountable. However, certain safety-critical products - where safety can be defined in various ways expanding the current relatively small set of NASA, aviation and health-care software - need engineers overseeing design, development, and QC to be accountable. And yes, this means that the self-taught developer, no matter how talented, will not be able to release certain types of software without some sort of oversight. [1]As a side-note - not everyone can call themselves an 'engineer' in other fields without the proper credentials and professional membership. I see developers calling themselves software engineers all the time though they don't have the chops. This would, of course, need to change. Updated to change asterisk to brackets

vtange 9y ago

I'm pretty sure less people would be rushing into the software engineer profession if they knew it came with the liabilities you described. That would drive up the prices of engineers.. and I'm pretty sure the industry in general doesn't want that. Maybe that's why the free pass mentality exists

emodendroket 9y ago
Or maybe the "free pass mentality" exists because development doesn't happen in a vacuum and errors don't necessarily find their way into programs because the programmers are stupid or careless.

tener 9y ago

Broken software does not typically kill people or makes you loose money (time or some other resources, maybe). In contrast bridges that fall apart tend to kill people in a nasty way. They are also quite hard to reboot.

Bartweiss 9y ago
An excellent framing I saw recently was "I can tell the people who write these articles don't actually want secure code, because they aren't paying me to write it." There's plenty of carefully-written, rigorously-tested code out there. Want to write software for air travel, drug manufacturing, or space flight? You'll have no choice but to write safe, first-rate code. Want a smartphone weather app written to those standards? Well, for $50 and a feature set five years behind the state of the art, you can have it.
- blackth0rn 9y ago
  Isn't this a tooling and knowledge issue? If I was as productive with TLA+ or model based design as I am with React-Native/Ionic/Swift etc. then we could have a smartphone weather app written to those standards for the same price.
  - hwayne 9y ago
    While there's plenty of room for improvement, I legitimately believe that TLA+ is now in a place where it's convenient enough to use in conjunction with Ionic/Swift.
  - deleted 9y ago
    [deleted]
IncRnd 9y ago
Broken software has the capability to regularly kill people or cause people to lose money. It happens too often.

emodendroket 9y ago

Without radical, NASA-style changes to typical development, where each line spends much more time being audited than being written, it is completely unreasonable to expect that kind of liability from developers.

amiga-workbench 9y ago
Ditch the dumb deadlines and the tight budgets and it would become a hell of a lot more feasible. I WISH I had the time and budget to put due diligence into my work.
- emodendroket 9y ago
  You have to think about what you're building too. Feature release would come much slower than it does today. Reasonable? If you're writing software that is absolutely safety-critical, that's likely. If you're writing iTunes, well, I have my doubts.
  - Bartweiss 9y ago
    It's not like reliable, secure software doesn't exist. It's just limited to spaces where people are willing to use years-old tech, forgo new features, and pay fortunes for auditing. Air travel, industrial machinery, medicine, and so on. Almost like the cost-benefit actually doesn't support making the development of trivial software ten times slower and more expensive... If there's a problem here, it's business externalities (Equifax won't pay adequately for screwing people), not engineering impossibility (the rest of us know how to patch Apache).
    - emodendroket 9y ago
      I mostly agree, but this article cites the national 911 software, which arguably should be treated as a critical system, failing, and some medical devices whose software killed people.
      - 9y ago

0xdeadbeefbabe 9y ago

> The only difference is that these other engineers are motivated to be significantly more thorough about their jobs. What evidence supports this as the only difference? I would argue these other engineers have significantly smaller jobs to do, so it's easier to be thorough. Edit: Why are you so keen to fix a problem you don't understand? Are you a programmer?

Bartweiss 9y ago

> The only difference is that these other engineers are motivated to be significantly more thorough about their jobs. Hang on. Sure, all engineering has hurdles. But how many engineering projects face regular enemy action? The last time somebody actively tried to compromise a system I was building was two weeks ago. How many engineers live like that? We hold civil engineers responsible when bridges collapse unprompted. We also hold them responsible if their bridges fail from some minor, anticipated harm like vandalism or a car crash. We don't hold them responsible for building bridges that can be blown up. If we want to hold software developers accountable for downtime, unprompted data leaks, and so on, fine. We should do it proportionate to the damage caused - there's no point in pretending that a shoddy smartphone game is as bad as a shoddy bridge - but fine. But let's not pretend that software has the same difficulties as every other type of engineering. Most engineering doesn't face an epidemic of people actively trying to disable safety precautions to hurt people, and a lot of it reliably fails as soon as it does face enemy action. Punishing every engineer who ever gets outwitted (say, by a leaked NSA-developed vulnerability) is absurd.

scintill76 9y ago
Most of the article, and the comment you're replying to, wasn't about hackers, though. It gave the example of people dying because the Therac machine or a Toyota car responded wrongly to non-malicious, normal user input, like a "bridge collapsing unprompted" in normal use.
- Bartweiss 9y ago
  Yeah, I agree about the article. I wouldn't offer this criticism to the main essay - fundamental bugs in critical systems are absolutely an engineering failure, at least in the sense that we should look for engineering practices to solve them. But with the post I responded to? I see this idea of "it's because engineers don't care, so give them liability" a lot, and frankly it's stupid. It neglects both cost-benefit differences and any interest in the question of hacking, and then goes on to pretend that you can fix bad systemic pressures by beating up the guys at the bottom. The difficulty of writing good code is not a function of how heavily you punish bad code. I flatly don't believe that the Therac bug would have been caught by personal liability for the bug; either the code would have been cancelled altogether to avoid liability or the bug would have gone on unabated. Realistically, criminal liability for lethal bugs mostly sounds like a way to drive talented devs away from high-stakes applications and leave those apps to be handled by people desperate enough to run the risk, or by international shops operating outside the reach of prosecution. I don't love the thought of a shop unwilling to use git because it creates personal responsibility. "Oh, we just pass code around on thumbdrives, I can't remember who wrote that critical line." I'd shrug and let the whole thing go, but the rumblings of "regulate software" keep getting stronger from a lot of directions. I don't really want to wake up in a decade and find out that degree requirements and expensive-but-useless licensing have become mandatory, and it feels like pretending bugs come from lazy devs is another snowflake in that avalanche.

drinchev 9y ago

Regulation on programmers is not the solution. Regulation on companies might be. I'm usually really disappointed when a cheap re-seller takes a government purchase of a 911 or 112 ( here in EU ) and then hires a subcontractor, which hires a subcontractor, which ... That's how crappy software gets into production.

imh 9y ago

Can't it be both a technology problem and an accountability one? The tech problem might take ages to get solved without accountability forcing it. If we were accountable, we'd be forced to make better tools for writing reliable software more easily. That said, there are simpler low handing fruit "tools" like code review, safe git practices, etc that still aren't followed everywhere. Ignoring those low hanging fruit does point to accountability/prioritization being a cause of low reliability.

crdoconnor 9y ago

>The dangerous unreliability of software is an accountability problem, not a technology problem. Engineering disciplines are regulated so that engineers are held personally and even criminally responsible if they endanger property or lives. Software, being the newcomer, has no such regulation. Regulation like that tends to follow after a series of disasters causing massive loss of life. Software hasn't yet managed to achieve that.

wpietri 9y ago

Motivation seems far from the only difference. If we divide making into "builds with atoms" and "builds with bits", both sorts of things are generally unregulated. It's when you get into "builds life-critical things with atoms" that you see significant regulation. If you build a house or an office building, your engineering is regulated. If you build a doghouse or an office filing cabinet, it isn't. I agree that societal accountability for engineers working on life-critical system, software and hardware both, is vital. For most atom-making and bit-making, though, I think stringent professional licensing is overkill. Instead I'd rather we started with accountability for the people who have much more control over the outcomes than individual software engineers: executives and managers.

IncRnd 9y ago
> If you build a house or an office building, your engineering is regulated. If you build a doghouse or an office filing cabinet, it isn't. A filing cabinet must be fit for the purpose it is used, or there will be liability incurred. OSHA would frown on keeping hazardous chemicals in your filing cabinet. Software should also be fit for the purpose it is marketed and used. There are many kinds of paper, but paper being used improperly, by storing a list of credit card numbers on it, is quite similar to how many pieces of software are written.

FLUX-YOU 9y ago

You can't drop all of this responsibility on developers. You need people checking behind them. But if we want to pay developers like surgeons and have malpractice insurance, all the better, I guess.

hasenj 9y ago

You can definitely do that. You just have to pay them 20x more money and give them 50x more time and give them absolutely all the tools they need. Best possible hardware. Best possible software. Fastest possible internet connections. etc. Show me an employer willing to do that and I will immediately sign up. Most companies want to do the exact opposite. Spend the least amount possible, because, you know, the budget is tight and the business side wants all these features ready yesterday, nay, last week.