5 ms·

> The only difference is that these other engineers are motivated to be significantly more thorough about their jobs. Hang on. Sure, all engineering has hurdle

> The only difference is that these other engineers are motivated to be significantly more thorough about their jobs. Hang on. Sure, all engineering has hurdles. But how many engineering projects face regular enemy action? The last time somebody actively tried to compromise a system I was building was two weeks ago. How many engineers live like that? We hold civil engineers responsible when bridges collapse unprompted. We also hold them responsible if their bridges fail from some minor, anticipated harm like vandalism or a car crash. We don't hold them responsible for building bridges that can be blown up. If we want to hold software developers accountable for downtime, unprompted data leaks, and so on, fine. We should do it proportionate to the damage caused - there's no point in pretending that a shoddy smartphone game is as bad as a shoddy bridge - but fine. But let's not pretend that software has the same difficulties as every other type of engineering. Most engineering doesn't face an epidemic of people actively trying to disable safety precautions to hurt people, and a lot of it reliably fails as soon as it does face enemy action. Punishing every engineer who ever gets outwitted (say, by a leaked NSA-developed vulnerability) is absurd.

scintill76 9y ago
Most of the article, and the comment you're replying to, wasn't about hackers, though. It gave the example of people dying because the Therac machine or a Toyota car responded wrongly to non-malicious, normal user input, like a "bridge collapsing unprompted" in normal use.
- Bartweiss 9y ago
  Yeah, I agree about the article. I wouldn't offer this criticism to the main essay - fundamental bugs in critical systems are absolutely an engineering failure, at least in the sense that we should look for engineering practices to solve them. But with the post I responded to? I see this idea of "it's because engineers don't care, so give them liability" a lot, and frankly it's stupid. It neglects both cost-benefit differences and any interest in the question of hacking, and then goes on to pretend that you can fix bad systemic pressures by beating up the guys at the bottom. The difficulty of writing good code is not a function of how heavily you punish bad code. I flatly don't believe that the Therac bug would have been caught by personal liability for the bug; either the code would have been cancelled altogether to avoid liability or the bug would have gone on unabated. Realistically, criminal liability for lethal bugs mostly sounds like a way to drive talented devs away from high-stakes applications and leave those apps to be handled by people desperate enough to run the risk, or by international shops operating outside the reach of prosecution. I don't love the thought of a shop unwilling to use git because it creates personal responsibility. "Oh, we just pass code around on thumbdrives, I can't remember who wrote that critical line." I'd shrug and let the whole thing go, but the rumblings of "regulate software" keep getting stronger from a lot of directions. I don't really want to wake up in a decade and find out that degree requirements and expensive-but-useless licensing have become mandatory, and it feels like pretending bugs come from lazy devs is another snowflake in that avalanche.
emodendroket 9y ago
Beyond that, the organization they're in sets the priorities; how is it going to fix anything if the engineers are on the hook for Equifax leaks but they still have the same (presumably not very security-conscious) managers?
vladimirralev 9y ago
You raise an interesting question here. Civil engineering these days is done on computers, mostly with incredibly convoluted FEM software and other simulation tools. All software vendors refuse to take any responsibility for bugs in their software and even block you from using the software if you don't agree on the terms.
IncRnd 9y ago
> But let's not pretend that software has the same difficulties as every other type of engineering. Most engineering doesn't face an epidemic of people actively trying to disable safety precautions to hurt people, and a lot of it reliably fails as soon as it does face enemy action. Punishing every engineer who ever gets outwitted (say, by a leaked NSA-developed vulnerability) is absurd. The issue isn't exactly how software fails, but that software today doesn't properly protect confidentiality and integrity of data, ensure services and data are available without providing oracles, authenticate and authorize correctly, log properly, etc. This is a well known methodology that is consistently ignored in preference to shipping products immediately. That's the issue. Unexpected things happen, accidentally or maliciously, but care should be taken - the proper kind of non-negligent care called due care. In general, software is not fit for the purpose it was created, appearing to work even though it really doesn't.
coldtea 9y ago
>* Hang on. Sure, all engineering has hurdles. But how many engineering projects face regular enemy action? The last time somebody actively tried to compromise a system I was building was two weeks ago. How many engineers live like that? We hold civil engineers responsible when bridges collapse unprompted. We also hold them responsible if their bridges fail from some minor, anticipated harm like vandalism or a car crash. We don't hold them responsible for building bridges that can be blown up.* Even more so, how many engineering problems have fuzzy requirements that on top of that change constantly and arbitrarily? If someone thinks building bridges is the same as writing some big software, they really have no idea about working in the trenches.
- hwayne 9y ago
  How many software projects have to deal with gravity? Weather patterns? How many software projects need teams of construction workers to actually build? Different kinds of engineering have different requirements. Just because the challenges in software are different from those in building bridges doesn't mean we get a free pass on correctness.
  - coldtea 9y ago
    >How many software projects have to deal with gravity? Gravity comes with a pretty little equation, and as far as problems goes is as predictable as it comes. Tolerances of various materials are also well known in advance, and their behavior under different designs and with different levels of stress assigned can be trivially modeled with CAD packages (and even manually). Same for weather. It comes down to a few behaviors (rain, wind, earthquakes of various degrees, sunlight) that people can model, and have been modeling for ages. We have 25+ centuries old buildings that still stand. >How many software projects need teams of construction workers to actually build? Not sure what you even mean here. >Different kinds of engineering have different requirements. That's obvious. The question we ask here is different: sure they are different, but are those requirements equally well defined and equally difficult across software and other engineering fields (like construction)? And to that I say no. Construction has pretty solid, rarely changing requirements (and almost never changing ONCE construction has begun), and works with specific materials with a limited set of interaction. With software modeling the entire universe is the limit as far as complexity goes.
toss1 9y ago
>> how many engineering projects face regular enemy action? The last time somebody actively tried to compromise a system I was building was two weeks ago. How many engineers live like that? Every bit of military hardware, or child's toy, for starters. More generally, you cite anticipated harm from vandalism or car crashes as being part of the general environment. It is far past time to recognize that the hacking environment in which we live IS THE NORMAL ENVIRONMENT, and these hacks ARE everyday anticipated harm. Software does not live in a virtual vacuum of space. It is on the internet, and will likely be scanned withing seconds if unprotected. Even tall buildings are now designed after 9/11 to better withstand jetliners crashing into them. To follow your metaphor, most of the hacks are completely unsophisticated, the equivalent of a bridge falling because some kid came along with a can of spray paint, or a screwdriver and started pulling screws. Let's also not pretend that other engineering disciplines doesn't also have to face real-world unpredictable and hostile inputs. Even children's toys have to be designed so as not to be hazardous from persistent, random, and clever unexpected attempts at use, misuse, and abuse from children. The same thing for consumer tools, which have many design features to frustrate the unending supply of clever idiots out there trying to win Darwin awards and sue you if they fail (or their families if they succeed). I've done both software and hardware, and your expressed attitude that software is somehow inherently more difficult or threatened is utter nonsense. The only basis for this is impatient project management where the key priority is to to ship yesterday because the management are idiots and think software is trivial. These approaches to start slinging code, write it fast and change it often are useful only for prototyping. In writing real software, my first two steps are 1) understand your problem in depth, and 2) work very hard for a long time to avoid writing code (i.e., better architecture to simplify, separate dependencies, etc.++). After substantial effort is put into those, then write only the code that is necessary. I take a similar approach to manufacturing, understand it, design a process with minimal key elements, then build those. Software is not a 'special snowflake'.
- arvinsim 9y ago
  > It is far past time to recognize that the hacking environment in which we live IS THE NORMAL ENVIRONMENT, and these hacks ARE everyday anticipated harm. > The only basis for this is impatient project management where the key priority is to to ship yesterday because the management are idiots and think software is trivial. I agree with the former but it is negated by the latter. The economic incentives certainly do. > These approaches to start slinging code, write it fast and change it often are useful only for prototyping. In writing real software, my first two steps are 1) understand your problem in depth, and 2) work very hard for a long time to avoid writing code (i.e., better architecture to simplify, separate dependencies, etc.++) Ideally this would be the case but this approach is a luxury many could not afford. This is especially true in an "Agile", first-to-ship mentality seen in most startups.
lomnakkus 9y ago
The problem with software as opposed to most artifacts that humans produce in the world is: "chaos". Sofware is insanely sensitive to initial conditions and a single logical error anywhere along the line leads to... well, chaos[1]. A single errant system may also have insane reach (as opposed to e.g. a bridge), so the consequenses just amplify. Unfortunately, we've started to rely on the reliability of software in unprecedented ways. (I'd guesstimate that it's probably mostly short-sighted economics that's driving it.) [1] This is inherent to Turing Machines[2], so if we keep insisting on Turing Complete languages in every realm of computation we'll forever suffer this problem. Please don't think that I'm making an argument against general computing, I'm not. I'm making an argument for restricted DSLs for things that don't need more. [2] We're not even talking the 'ordinary' chaos of dynamical systems here. Those can at least usually be approximated numerically for the near future unless they're actually really close to a point of divergence. Not so with programs.
HumanDrivenDev 9y ago
> Hang on. Sure, all engineering has hurdles. But how many engineering projects face regular enemy action? The last time somebody actively tried to compromise a system I was building was two weeks ago. How many engineers live like that? I don't think that's a good analogy. Engineers might claim we have it easy because we don't have to face earthquakes or electrical surges or birds flying into our turbines.
TheCapn 9y ago
> How many engineers live like that? Well for one thing the engineer planning the electrical isn't the engineer planning the structure. The engineer who scoped the HVAC isn't the guy who wrote the company's website either. This is a long term problem that we need to begin recognizing and addressing today. I wouldn't expect the programmer who is building the frontend to understand the network security. I wouldn't expect the database guy to know the server administrative role. Sure there's overlap but to have a team of people whose focus is that specific role is partially what engineers do. As a software engineer myself it's illegal for me to go onto my client's sites and start engineering their electrical system. I pass that effort to my coworkers who are electrical engineers in just the same way they ask me to design them their databases and their network.

scintill76 9y ago

Most of the article, and the comment you're replying to, wasn't about hackers, though. It gave the example of people dying because the Therac machine or a Toyota car responded wrongly to non-malicious, normal user input, like a "bridge collapsing unprompted" in normal use.

Bartweiss 9y ago
Yeah, I agree about the article. I wouldn't offer this criticism to the main essay - fundamental bugs in critical systems are absolutely an engineering failure, at least in the sense that we should look for engineering practices to solve them. But with the post I responded to? I see this idea of "it's because engineers don't care, so give them liability" a lot, and frankly it's stupid. It neglects both cost-benefit differences and any interest in the question of hacking, and then goes on to pretend that you can fix bad systemic pressures by beating up the guys at the bottom. The difficulty of writing good code is not a function of how heavily you punish bad code. I flatly don't believe that the Therac bug would have been caught by personal liability for the bug; either the code would have been cancelled altogether to avoid liability or the bug would have gone on unabated. Realistically, criminal liability for lethal bugs mostly sounds like a way to drive talented devs away from high-stakes applications and leave those apps to be handled by people desperate enough to run the risk, or by international shops operating outside the reach of prosecution. I don't love the thought of a shop unwilling to use git because it creates personal responsibility. "Oh, we just pass code around on thumbdrives, I can't remember who wrote that critical line." I'd shrug and let the whole thing go, but the rumblings of "regulate software" keep getting stronger from a lot of directions. I don't really want to wake up in a decade and find out that degree requirements and expensive-but-useless licensing have become mandatory, and it feels like pretending bugs come from lazy devs is another snowflake in that avalanche.

emodendroket 9y ago

Beyond that, the organization they're in sets the priorities; how is it going to fix anything if the engineers are on the hook for Equifax leaks but they still have the same (presumably not very security-conscious) managers?

vladimirralev 9y ago

You raise an interesting question here. Civil engineering these days is done on computers, mostly with incredibly convoluted FEM software and other simulation tools. All software vendors refuse to take any responsibility for bugs in their software and even block you from using the software if you don't agree on the terms.

IncRnd 9y ago

> But let's not pretend that software has the same difficulties as every other type of engineering. Most engineering doesn't face an epidemic of people actively trying to disable safety precautions to hurt people, and a lot of it reliably fails as soon as it does face enemy action. Punishing every engineer who ever gets outwitted (say, by a leaked NSA-developed vulnerability) is absurd. The issue isn't exactly how software fails, but that software today doesn't properly protect confidentiality and integrity of data, ensure services and data are available without providing oracles, authenticate and authorize correctly, log properly, etc. This is a well known methodology that is consistently ignored in preference to shipping products immediately. That's the issue. Unexpected things happen, accidentally or maliciously, but care should be taken - the proper kind of non-negligent care called due care. In general, software is not fit for the purpose it was created, appearing to work even though it really doesn't.

coldtea 9y ago

>* Hang on. Sure, all engineering has hurdles. But how many engineering projects face regular enemy action? The last time somebody actively tried to compromise a system I was building was two weeks ago. How many engineers live like that? We hold civil engineers responsible when bridges collapse unprompted. We also hold them responsible if their bridges fail from some minor, anticipated harm like vandalism or a car crash. We don't hold them responsible for building bridges that can be blown up.* Even more so, how many engineering problems have fuzzy requirements that on top of that change constantly and arbitrarily? If someone thinks building bridges is the same as writing some big software, they really have no idea about working in the trenches.

hwayne 9y ago
How many software projects have to deal with gravity? Weather patterns? How many software projects need teams of construction workers to actually build? Different kinds of engineering have different requirements. Just because the challenges in software are different from those in building bridges doesn't mean we get a free pass on correctness.
- coldtea 9y ago
  >How many software projects have to deal with gravity? Gravity comes with a pretty little equation, and as far as problems goes is as predictable as it comes. Tolerances of various materials are also well known in advance, and their behavior under different designs and with different levels of stress assigned can be trivially modeled with CAD packages (and even manually). Same for weather. It comes down to a few behaviors (rain, wind, earthquakes of various degrees, sunlight) that people can model, and have been modeling for ages. We have 25+ centuries old buildings that still stand. >How many software projects need teams of construction workers to actually build? Not sure what you even mean here. >Different kinds of engineering have different requirements. That's obvious. The question we ask here is different: sure they are different, but are those requirements equally well defined and equally difficult across software and other engineering fields (like construction)? And to that I say no. Construction has pretty solid, rarely changing requirements (and almost never changing ONCE construction has begun), and works with specific materials with a limited set of interaction. With software modeling the entire universe is the limit as far as complexity goes.

toss1 9y ago

>> how many engineering projects face regular enemy action? The last time somebody actively tried to compromise a system I was building was two weeks ago. How many engineers live like that? Every bit of military hardware, or child's toy, for starters. More generally, you cite anticipated harm from vandalism or car crashes as being part of the general environment. It is far past time to recognize that the hacking environment in which we live IS THE NORMAL ENVIRONMENT, and these hacks ARE everyday anticipated harm. Software does not live in a virtual vacuum of space. It is on the internet, and will likely be scanned withing seconds if unprotected. Even tall buildings are now designed after 9/11 to better withstand jetliners crashing into them. To follow your metaphor, most of the hacks are completely unsophisticated, the equivalent of a bridge falling because some kid came along with a can of spray paint, or a screwdriver and started pulling screws. Let's also not pretend that other engineering disciplines doesn't also have to face real-world unpredictable and hostile inputs. Even children's toys have to be designed so as not to be hazardous from persistent, random, and clever unexpected attempts at use, misuse, and abuse from children. The same thing for consumer tools, which have many design features to frustrate the unending supply of clever idiots out there trying to win Darwin awards and sue you if they fail (or their families if they succeed). I've done both software and hardware, and your expressed attitude that software is somehow inherently more difficult or threatened is utter nonsense. The only basis for this is impatient project management where the key priority is to to ship yesterday because the management are idiots and think software is trivial. These approaches to start slinging code, write it fast and change it often are useful only for prototyping. In writing real software, my first two steps are 1) understand your problem in depth, and 2) work very hard for a long time to avoid writing code (i.e., better architecture to simplify, separate dependencies, etc.++). After substantial effort is put into those, then write only the code that is necessary. I take a similar approach to manufacturing, understand it, design a process with minimal key elements, then build those. Software is not a 'special snowflake'.

arvinsim 9y ago
> It is far past time to recognize that the hacking environment in which we live IS THE NORMAL ENVIRONMENT, and these hacks ARE everyday anticipated harm. > The only basis for this is impatient project management where the key priority is to to ship yesterday because the management are idiots and think software is trivial. I agree with the former but it is negated by the latter. The economic incentives certainly do. > These approaches to start slinging code, write it fast and change it often are useful only for prototyping. In writing real software, my first two steps are 1) understand your problem in depth, and 2) work very hard for a long time to avoid writing code (i.e., better architecture to simplify, separate dependencies, etc.++) Ideally this would be the case but this approach is a luxury many could not afford. This is especially true in an "Agile", first-to-ship mentality seen in most startups.

lomnakkus 9y ago

The problem with software as opposed to most artifacts that humans produce in the world is: "chaos". Sofware is insanely sensitive to initial conditions and a single logical error anywhere along the line leads to... well, chaos[1]. A single errant system may also have insane reach (as opposed to e.g. a bridge), so the consequenses just amplify. Unfortunately, we've started to rely on the reliability of software in unprecedented ways. (I'd guesstimate that it's probably mostly short-sighted economics that's driving it.) [1] This is inherent to Turing Machines[2], so if we keep insisting on Turing Complete languages in every realm of computation we'll forever suffer this problem. Please don't think that I'm making an argument against general computing, I'm not. I'm making an argument for restricted DSLs for things that don't need more. [2] We're not even talking the 'ordinary' chaos of dynamical systems here. Those can at least usually be approximated numerically for the near future unless they're actually really close to a point of divergence. Not so with programs.

HumanDrivenDev 9y ago

> Hang on. Sure, all engineering has hurdles. But how many engineering projects face regular enemy action? The last time somebody actively tried to compromise a system I was building was two weeks ago. How many engineers live like that? I don't think that's a good analogy. Engineers might claim we have it easy because we don't have to face earthquakes or electrical surges or birds flying into our turbines.

TheCapn 9y ago

> How many engineers live like that? Well for one thing the engineer planning the electrical isn't the engineer planning the structure. The engineer who scoped the HVAC isn't the guy who wrote the company's website either. This is a long term problem that we need to begin recognizing and addressing today. I wouldn't expect the programmer who is building the frontend to understand the network security. I wouldn't expect the database guy to know the server administrative role. Sure there's overlap but to have a team of people whose focus is that specific role is partially what engineers do. As a software engineer myself it's illegal for me to go onto my client's sites and start engineering their electrical system. I pass that effort to my coworkers who are electrical engineers in just the same way they ask me to design them their databases and their network.