8 ms·

As we cast about trying to figure out ways to make software more secure or reliable, please remember that in other engineering fields (civil, chemical, mechanic

As we cast about trying to figure out ways to make software more secure or reliable, please remember that in other engineering fields (civil, chemical, mechanical, etc.) prioritizing safety and reliability is a _solved problem_. (1996) https://www.fastcompany.com/28121/they-write-right-stuff https://www.fastcompany.com/28121/they-write-right-stuff > It is perfect, as perfect as human beings have achieved. Consider these stats: the last three versions of the program — each 420,000 lines long-had just one error each. The last 11 versions of this software had a total of 17 errors. Commercial programs of equivalent complexity would have 5,000 errors. > The process isn’t even rocket science. Its standard practice in almost every engineering discipline except software engineering. The problem is consequences. We had centuries of people dying in bridge collapses before we got our shit together and started prioritizing safety in civil engineering (i.e. engineers and managers going to prison if they don't). The same will be true for software. As more people get harmed by thrown together software (e.g. mass panic in Hawaii, state psychological exploitation on social media), we'll start regulating it like other engineering fields. As a former chemical engineer, I welcome this transition, but I realize it will likely also take centuries of hard lessons.

pjmlp 9y ago
That is my point of view as well, software needs to be liable just like in other fields of engineering, done by people that actually understand its consequences. I don't do a car inspection at a guy that just happens to know some stuff about mechanics.
Joeri 9y ago
The difference is that while you can’t make a bridge in your bedroom you can make an app. Should we forbid people from writing code without the proper certification? Should we close down the open internet and replace it with a regulated zone where only compliant software can be run? I agree that we need a higher standard of engineering in software, but I’m not clear on how to achieve it without draconian measures.
- geocar 9y ago
  In the UK, you can build a bridge without certification as long as you have someone certified review the plans and the implementation before anyone else drives on it. I suspect this is similar for most civilised countries. A (perhaps short-term) idea would be to make software vendors liable, and do not permit them to sign away that liability.
  - dcow 9y ago
    I completely agree. But the vendor situation looks very different for software. Imagine being the sole person responsible for migrating Linux from ip/nftables to ebtables. You don't know how your stuff will be used downstream. So you license it with text in all caps reminding people that your software is provided WITHOUT ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. And people still use it to build oil pipelines that bleed toxic goop when sent a really weird keepalive message. I hope that the first place we see an uptick is security. The list has grown rather long: consumer/citizen identity breaches, hospital ransomware, digital asset theft, remote vehicle control, etc. Most security people I know are of the mindset that systems will be hacked. It's extremely.. pragmatic. You can't really fault the people responsible when many companies simply require good damage control over actual security in order to be successful. But that's exactly the point. If the consequences were increased, not so many C[I]SOs would be okay with engineers using hot new software every few quarters. And those old boxes running Ruby 1.87 would sure as hell be patched or isolated to oblivion. Companies or projects with good security would flourish. Maybe some would be pressured to operate more like the archetype they're defending against (more red team, more organizational commitment to physical and operational security). I worked at a security company that had to get rid of a slack bot screen lock game because it hurt some people's feels. So yeah I think some of the priorities in this industry are messed up.
    - geocar 9y ago
      Those people who build oil pipelines should be able to sue whoever sold them the Linux distribution (probably Red Hat or IBM) which was vulnerable to that weird keepalive message. In that case, a judge (and perhaps a jury) could hear how Red Hat did everything they possibly could to protect from the vulnerability as evidenced by their ISO QA processes and the fact that everyone else was vulnerable to the same "bug" … or from the other side how Microsoft and Apple weren't at-risk, so Red Hat should've caught it. C[I]SOs would want to be patched, because ISO recommends they would be patched. > You can't really fault the people responsible when many companies simply require good damage control over actual security in order to be successful. Which is why I propose legislation, so "good damage control" wouldn't be enough. You better believe that oil company would want some evidence of testing and proper specifications, and to have them reviewed by a couple independent parties if the government could take them for a percent of gross revenue for the security vulnerabilities alone.
riffraff 9y ago
I am sure other fields have a better development story these days, but isn't it a matter of perceived risk and relative effort? I.e. we rarely hear about plane crashes caused by software, or power plants accidentally blowing up due to an integer underflow. Civil engineers make solid bridges, but most software is not the golden gate, it's a shed thrown together by some guy who's not an engineer nor a professional mason.
fh973 9y ago
I think the difference is that in engineering you are legally obliged to follow this process and in software you are not. The leads to a race to the bottom for which both the software vendors and consumers are responsible. Everybody wants the latest stuff as soon as possible, and wants to play less. If you release software only when everything's "right", the market will have moved on.
- geocar 9y ago
  > If you release software only when everything's "right", the market will have moved on. ... unless we the people demand legislation that protects us from software "vendors" selling us their bugs. I think it's interesting that people can see a car that doesn't work, and a house that's falling apart, but they can't really see software. Can you imagine someone saying "my house is leaking" and the landlord saying, with all seriousness: "Have you tried turning it off and on?" Of course, we have legislature for those things, so why not software?
cageface 9y ago
Follow the money. If people were willing to commit to a specification with the same level of precision as a civil engineering blueprint and then stick to that specification then you’d see a dramatic uptick in software quality, along with a dramatic uptick in price and time to develop. But since hardly anybody wants to commit to that kind of precise design or pay the cost we get what we have today instead. Come to SE Asia and I’ll show you plenty of civil engineering projects with similarly poor planning and execution.
- cuckcuckspruce 9y ago
  There are software engineering fields where people commit to precise specifications - they typically interface with humans on some biological level (medical devices, etc), and they are quite expensive.
  - pas 9y ago
    Of course, the same goes for military/infrastructure (self-driving trains) software. And there are solutions for improving software safety, it starts by improving the process of software development. And for that there's at least one sort of official framework: CMMI. (Even though DoD stopped requiring CMMI, but it still considers it a big plus, and companies for important stuff all use it.) Of course, it'll be interesting to see what goes on with the F-35, as a lot of testing is now "skipped". (Sure, that is not software testing, but still, I can't imagine the software dev process is in a better shape after all these delays, cost overruns and probably still ongoing requirement changes.)
LudoA 9y ago
I read an article many many years ago explaining why software engineering can't be compared to mechanical engineering. I can't find it now, but one of its points of comparison was: if a nut isn't tightened enough on a bridge, you can tighten it a bit more -- turn the nut 3 or maybe 6 millimeters more. In software engineering, if you tighten the nut only 3 millimeters instead of 6, one of the bridge's endpoints now ends in a different dimension. I think there is a valid point to it. I'm always amazed in housing construction how things are done "approximately". A wall should be vertically straight... give or take a few centimeters. That kind of approximation doesn't work in software engineering. It has a whole other type of complexity to it.
- KozmoNau7 9y ago
  If you're building houses, and your walls are a "few centimeters" off from straight, you'll get a real asswhoopin'. Especially if you're building something with more than one story, and you expect people to actually pay for it.
  - someone454 9y ago
    I’ve never found a perfect 90 degree angle in any house, and I’ve built portions of quite a few. Wood expands and contracts with the seasons.
    - KozmoNau7 9y ago
      Which also makes it a really silly choice for actual construction. Brick and concrete make a lot more sense.
      - pixl97 9y ago
        Expanding and contracting is good. Steel also expands and contracts. Bricks and un-reinforced concrete crack and crumble.
        KozmoNau7 9y ago
incompatible 9y ago
From the article (1996): And it’s the dominant image of the software development world: Gen-Xers sporting T-shirts and distracted looks, squeezing too much heroic code writing into too little time; rollerblades and mountain bikes tucked in corners; pizza boxes and Starbucks cups discarded in conference rooms; dueling tunes from Smashing Pumpkins, Alanis Morrisette and the Fugees. Its the world made famous, romantic, even inevitable by stories out of Sun Microsystems, Microsoft, and Netscape. It's incredible how much software development has changed over the last 21 years. Change Gen-Xers to Millennials, update the playlist, and replace Sun Microsystems, Microsoft, and Netscape with Apple, Facebook and Google.
- gjvc 9y ago
  I can't tell if you're being sarcastic or not in your second paragraph. :-)
jballanc 9y ago
You over-estimate the degree to which the other engineering fields you mentioned have solved safety and reliability. Just to name a few: - Tacoma Narrows bridge: https://en.wikipedia.org/wiki/Tacoma_Narrows_Bridge https://en.wikipedia.org/wiki/Tacoma_Narrows_Bridge - Thalidomide: https://en.wikipedia.org/wiki/Thalidomide https://en.wikipedia.org/wiki/Thalidomide - Hyatt Regency walkway collapse: https://en.wikipedia.org/wiki/Hyatt_Regency_walkway_collapse https://en.wikipedia.org/wiki/Hyatt_Regency_walkway_collapse - Challenger disaster: https://en.wikipedia.org/wiki/Space_Shuttle_Columbia_disaster https://en.wikipedia.org/wiki/Space_Shuttle_Columbia_disaste... - Columbia disaster: https://en.wikipedia.org/wiki/Space_Shuttle_Columbia_disaster https://en.wikipedia.org/wiki/Space_Shuttle_Columbia_disaste... - Vioxx: https://en.wikipedia.org/wiki/Rofecoxib https://en.wikipedia.org/wiki/Rofecoxib ...and plenty more. The only thing that separates software engineering with the other engineering disciplines is that there is a structure (usually in the form of code/spec enforcement) for internalizing and learning from disasters when they happen.
- minikites 9y ago
  The Tacoma Narrows bridge collapse happened 80 years ago, the Hyatt Regency was nearly 40. When did you last read about a software bug?
  - jcranmer 9y ago
    Phones that suddenly combust came out in 2016. The Big Dig ceiling collapse was 2006. Rana Plaza collapsed in 2013. Serious engineering failures are still happening. Sure, the most famous ones may be decades old, but those aren't the only ones.
  - xenophonf 9y ago
    Professional engineering has plenty of recent disasters to point to, whether failures in design, construction, operation, or security: The Waco fertilizer plant explosion (2013), cause determined to be arson (2016) San Francisco Bay Bridge seismic bolt failures (2013), caused during construction: https://www.nace.org/CORROSION-FAILURES-San-Francisco-Bay-Bridge-Bolt-Failure.aspx https://www.nace.org/CORROSION-FAILURES-San-Francisco-Bay-Br... The Fukushima Daiichi nuclear disaster (2011), design/regulatory failure And so forth: https://en.wikipedia.org/wiki/Category:Industrial_disasters_by_year https://en.wikipedia.org/wiki/Category:Industrial_disasters_... And so on: https://en.wikipedia.org/wiki/Category:Engineering_failures https://en.wikipedia.org/wiki/Category:Engineering_failures
viraptor 9y ago
> in other engineering fields (civil, chemical, mechanical, etc.) prioritizing safety and reliability is a _solved problem_. One big difference between those and programming is that there (outside of military, physical security, and few other exceptions) there are no active adversaries for your project. You design a X that will serve a defined purpose. You design for some parameters (known wind speeds or local natural hazards). You don't normally design with someone actively trying to destroy your building in mind. You don't design thinking how to avoid the current issues and to isolate the impact of potential future attackers. Your product doesn't have a "well funded attackers will spend undefined amount of time trying to make your project fail" requirement.
- gkya 9y ago
  > You don't normally design with someone actively trying to destroy your building in mind. You do, actually. When designing a building, one considers many "attacks" it can be subject to, depending on the geography and demography of the site: what are the precipitation patterns like, what is the situation WRT sysmical activity, how is the soil, what is beneath, what is the crime rate in the area, what kind of animals live there, where does the wind tend to come from, how much sunlight is received, etc. etc., about all of these questions are in one way or another is relevant to security of a building, and you have to consider each and every one of them, and design accordingly. You don't design against destruction because that's hardly possible, there isn't much you can do to mitigate someone bombing a building or attacking with heavy machines. Think of that as analogous to using the ME in an Intel CPU.
  - jerf 9y ago
    'When designing a building, one considers many "attacks" it can be subject to' Your use of scare quotes is appropriate, because there is a qualitative difference between intelligently-driven and unintelligent attacks. Not to mention the intelligently-driven attacks when the attackers are manifestly more intelligent and skilled than the defenders! If computer security experts didn't have to worry about intelligent attackers, computer security would be very nearly a solved problem.
pizza234 9y ago
> As we cast about trying to figure out ways to make software more secure or reliable, please remember that in other engineering fields (civil, chemical, mechanical, etc.) prioritizing safety and reliability is a _solved problem_. The article makes an unfair comparison; if the solution to the problem is like the one below (extract from the article), the change, reasonably, will never happen in commercial programs (the ones "with 5000 errors"). > Take the upgrade of the software to permit the shuttle to navigate with Global Positioning Satellites, a change that involves just 1.5% of the program, or 6,366 lines of code. The specs for that one change run 2,500 pages, a volume thicker than a phone book. The specs for the current program fill 30 volumes and run 40,000 pages.
mpweiher 9y ago
How many people die because of software? How many die because of cars? We could make cars that don't kill people. We could make software that doesn't break.

pjmlp 9y ago

That is my point of view as well, software needs to be liable just like in other fields of engineering, done by people that actually understand its consequences. I don't do a car inspection at a guy that just happens to know some stuff about mechanics.

Joeri 9y ago

The difference is that while you can’t make a bridge in your bedroom you can make an app. Should we forbid people from writing code without the proper certification? Should we close down the open internet and replace it with a regulated zone where only compliant software can be run? I agree that we need a higher standard of engineering in software, but I’m not clear on how to achieve it without draconian measures.

geocar 9y ago
In the UK, you can build a bridge without certification as long as you have someone certified review the plans and the implementation before anyone else drives on it. I suspect this is similar for most civilised countries. A (perhaps short-term) idea would be to make software vendors liable, and do not permit them to sign away that liability.
- dcow 9y ago
  I completely agree. But the vendor situation looks very different for software. Imagine being the sole person responsible for migrating Linux from ip/nftables to ebtables. You don't know how your stuff will be used downstream. So you license it with text in all caps reminding people that your software is provided WITHOUT ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. And people still use it to build oil pipelines that bleed toxic goop when sent a really weird keepalive message. I hope that the first place we see an uptick is security. The list has grown rather long: consumer/citizen identity breaches, hospital ransomware, digital asset theft, remote vehicle control, etc. Most security people I know are of the mindset that systems will be hacked. It's extremely.. pragmatic. You can't really fault the people responsible when many companies simply require good damage control over actual security in order to be successful. But that's exactly the point. If the consequences were increased, not so many C[I]SOs would be okay with engineers using hot new software every few quarters. And those old boxes running Ruby 1.87 would sure as hell be patched or isolated to oblivion. Companies or projects with good security would flourish. Maybe some would be pressured to operate more like the archetype they're defending against (more red team, more organizational commitment to physical and operational security). I worked at a security company that had to get rid of a slack bot screen lock game because it hurt some people's feels. So yeah I think some of the priorities in this industry are messed up.
  - geocar 9y ago
    Those people who build oil pipelines should be able to sue whoever sold them the Linux distribution (probably Red Hat or IBM) which was vulnerable to that weird keepalive message. In that case, a judge (and perhaps a jury) could hear how Red Hat did everything they possibly could to protect from the vulnerability as evidenced by their ISO QA processes and the fact that everyone else was vulnerable to the same "bug" … or from the other side how Microsoft and Apple weren't at-risk, so Red Hat should've caught it. C[I]SOs would want to be patched, because ISO recommends they would be patched. > You can't really fault the people responsible when many companies simply require good damage control over actual security in order to be successful. Which is why I propose legislation, so "good damage control" wouldn't be enough. You better believe that oil company would want some evidence of testing and proper specifications, and to have them reviewed by a couple independent parties if the government could take them for a percent of gross revenue for the security vulnerabilities alone.

riffraff 9y ago

I am sure other fields have a better development story these days, but isn't it a matter of perceived risk and relative effort? I.e. we rarely hear about plane crashes caused by software, or power plants accidentally blowing up due to an integer underflow. Civil engineers make solid bridges, but most software is not the golden gate, it's a shed thrown together by some guy who's not an engineer nor a professional mason.

fh973 9y ago

I think the difference is that in engineering you are legally obliged to follow this process and in software you are not. The leads to a race to the bottom for which both the software vendors and consumers are responsible. Everybody wants the latest stuff as soon as possible, and wants to play less. If you release software only when everything's "right", the market will have moved on.

geocar 9y ago
> If you release software only when everything's "right", the market will have moved on. ... unless we the people demand legislation that protects us from software "vendors" selling us their bugs. I think it's interesting that people can see a car that doesn't work, and a house that's falling apart, but they can't really see software. Can you imagine someone saying "my house is leaking" and the landlord saying, with all seriousness: "Have you tried turning it off and on?" Of course, we have legislature for those things, so why not software?

cageface 9y ago

Follow the money. If people were willing to commit to a specification with the same level of precision as a civil engineering blueprint and then stick to that specification then you’d see a dramatic uptick in software quality, along with a dramatic uptick in price and time to develop. But since hardly anybody wants to commit to that kind of precise design or pay the cost we get what we have today instead. Come to SE Asia and I’ll show you plenty of civil engineering projects with similarly poor planning and execution.

cuckcuckspruce 9y ago
There are software engineering fields where people commit to precise specifications - they typically interface with humans on some biological level (medical devices, etc), and they are quite expensive.
- pas 9y ago
  Of course, the same goes for military/infrastructure (self-driving trains) software. And there are solutions for improving software safety, it starts by improving the process of software development. And for that there's at least one sort of official framework: CMMI. (Even though DoD stopped requiring CMMI, but it still considers it a big plus, and companies for important stuff all use it.) Of course, it'll be interesting to see what goes on with the F-35, as a lot of testing is now "skipped". (Sure, that is not software testing, but still, I can't imagine the software dev process is in a better shape after all these delays, cost overruns and probably still ongoing requirement changes.)

LudoA 9y ago

I read an article many many years ago explaining why software engineering can't be compared to mechanical engineering. I can't find it now, but one of its points of comparison was: if a nut isn't tightened enough on a bridge, you can tighten it a bit more -- turn the nut 3 or maybe 6 millimeters more. In software engineering, if you tighten the nut only 3 millimeters instead of 6, one of the bridge's endpoints now ends in a different dimension. I think there is a valid point to it. I'm always amazed in housing construction how things are done "approximately". A wall should be vertically straight... give or take a few centimeters. That kind of approximation doesn't work in software engineering. It has a whole other type of complexity to it.

KozmoNau7 9y ago
If you're building houses, and your walls are a "few centimeters" off from straight, you'll get a real asswhoopin'. Especially if you're building something with more than one story, and you expect people to actually pay for it.
- someone454 9y ago
  I’ve never found a perfect 90 degree angle in any house, and I’ve built portions of quite a few. Wood expands and contracts with the seasons.
  - KozmoNau7 9y ago
    Which also makes it a really silly choice for actual construction. Brick and concrete make a lot more sense.
    - pixl97 9y ago
      Expanding and contracting is good. Steel also expands and contracts. Bricks and un-reinforced concrete crack and crumble.
      - KozmoNau7 9y ago

incompatible 9y ago

From the article (1996): And it’s the dominant image of the software development world: Gen-Xers sporting T-shirts and distracted looks, squeezing too much heroic code writing into too little time; rollerblades and mountain bikes tucked in corners; pizza boxes and Starbucks cups discarded in conference rooms; dueling tunes from Smashing Pumpkins, Alanis Morrisette and the Fugees. Its the world made famous, romantic, even inevitable by stories out of Sun Microsystems, Microsoft, and Netscape. It's incredible how much software development has changed over the last 21 years. Change Gen-Xers to Millennials, update the playlist, and replace Sun Microsystems, Microsoft, and Netscape with Apple, Facebook and Google.

gjvc 9y ago
I can't tell if you're being sarcastic or not in your second paragraph. :-)

jballanc 9y ago

You over-estimate the degree to which the other engineering fields you mentioned have solved safety and reliability. Just to name a few: - Tacoma Narrows bridge: https://en.wikipedia.org/wiki/Tacoma_Narrows_Bridge https://en.wikipedia.org/wiki/Tacoma_Narrows_Bridge - Thalidomide: https://en.wikipedia.org/wiki/Thalidomide https://en.wikipedia.org/wiki/Thalidomide - Hyatt Regency walkway collapse: https://en.wikipedia.org/wiki/Hyatt_Regency_walkway_collapse https://en.wikipedia.org/wiki/Hyatt_Regency_walkway_collapse - Challenger disaster: https://en.wikipedia.org/wiki/Space_Shuttle_Columbia_disaster https://en.wikipedia.org/wiki/Space_Shuttle_Columbia_disaste... - Columbia disaster: https://en.wikipedia.org/wiki/Space_Shuttle_Columbia_disaster https://en.wikipedia.org/wiki/Space_Shuttle_Columbia_disaste... - Vioxx: https://en.wikipedia.org/wiki/Rofecoxib https://en.wikipedia.org/wiki/Rofecoxib ...and plenty more. The only thing that separates software engineering with the other engineering disciplines is that there is a structure (usually in the form of code/spec enforcement) for internalizing and learning from disasters when they happen.

minikites 9y ago
The Tacoma Narrows bridge collapse happened 80 years ago, the Hyatt Regency was nearly 40. When did you last read about a software bug?
- jcranmer 9y ago
  Phones that suddenly combust came out in 2016. The Big Dig ceiling collapse was 2006. Rana Plaza collapsed in 2013. Serious engineering failures are still happening. Sure, the most famous ones may be decades old, but those aren't the only ones.
- xenophonf 9y ago
  Professional engineering has plenty of recent disasters to point to, whether failures in design, construction, operation, or security: The Waco fertilizer plant explosion (2013), cause determined to be arson (2016) San Francisco Bay Bridge seismic bolt failures (2013), caused during construction: https://www.nace.org/CORROSION-FAILURES-San-Francisco-Bay-Bridge-Bolt-Failure.aspx https://www.nace.org/CORROSION-FAILURES-San-Francisco-Bay-Br... The Fukushima Daiichi nuclear disaster (2011), design/regulatory failure And so forth: https://en.wikipedia.org/wiki/Category:Industrial_disasters_by_year https://en.wikipedia.org/wiki/Category:Industrial_disasters_... And so on: https://en.wikipedia.org/wiki/Category:Engineering_failures https://en.wikipedia.org/wiki/Category:Engineering_failures

viraptor 9y ago

> in other engineering fields (civil, chemical, mechanical, etc.) prioritizing safety and reliability is a _solved problem_. One big difference between those and programming is that there (outside of military, physical security, and few other exceptions) there are no active adversaries for your project. You design a X that will serve a defined purpose. You design for some parameters (known wind speeds or local natural hazards). You don't normally design with someone actively trying to destroy your building in mind. You don't design thinking how to avoid the current issues and to isolate the impact of potential future attackers. Your product doesn't have a "well funded attackers will spend undefined amount of time trying to make your project fail" requirement.

gkya 9y ago
> You don't normally design with someone actively trying to destroy your building in mind. You do, actually. When designing a building, one considers many "attacks" it can be subject to, depending on the geography and demography of the site: what are the precipitation patterns like, what is the situation WRT sysmical activity, how is the soil, what is beneath, what is the crime rate in the area, what kind of animals live there, where does the wind tend to come from, how much sunlight is received, etc. etc., about all of these questions are in one way or another is relevant to security of a building, and you have to consider each and every one of them, and design accordingly. You don't design against destruction because that's hardly possible, there isn't much you can do to mitigate someone bombing a building or attacking with heavy machines. Think of that as analogous to using the ME in an Intel CPU.
- jerf 9y ago
  'When designing a building, one considers many "attacks" it can be subject to' Your use of scare quotes is appropriate, because there is a qualitative difference between intelligently-driven and unintelligent attacks. Not to mention the intelligently-driven attacks when the attackers are manifestly more intelligent and skilled than the defenders! If computer security experts didn't have to worry about intelligent attackers, computer security would be very nearly a solved problem.

pizza234 9y ago

> As we cast about trying to figure out ways to make software more secure or reliable, please remember that in other engineering fields (civil, chemical, mechanical, etc.) prioritizing safety and reliability is a _solved problem_. The article makes an unfair comparison; if the solution to the problem is like the one below (extract from the article), the change, reasonably, will never happen in commercial programs (the ones "with 5000 errors"). > Take the upgrade of the software to permit the shuttle to navigate with Global Positioning Satellites, a change that involves just 1.5% of the program, or 6,366 lines of code. The specs for that one change run 2,500 pages, a volume thicker than a phone book. The specs for the current program fill 30 volumes and run 40,000 pages.

mpweiher 9y ago

How many people die because of software? How many die because of cars? We could make cars that don't kill people. We could make software that doesn't break.