5 ms·

The current state of software safety discussion resembles the state of medical safety discussion 2, 3 decades ago (yeah, software is really really behind time).

by magice 9y ago

The current state of software safety discussion resembles the state of medical safety discussion 2, 3 decades ago (yeah, software is really really behind time). Back then, too, the thoughts on medical safety also were divided into 2 schools: the professionalism and the process oriented. The former school argues more or less what Uncle Bob argues: blame the damned and * who made the mistakes; be more careful, damn it. But of course, that stupidity fell out of favor. After all, when mistakes kill, people are serious about it. After a while, serious people realize that blaming and clamoring for care backfires big time. That's when they applied, you know, science and statistic to safety. So, tools are upgraded: better color coded medicine boxes, for example, or checklists in surgery. But it's more. They figured out what trainings and processes provide high impacts and do them rigorously. Nurses are taught (I am not kidding you) how to question doctors when weird things happen; identity verification (ever notice why nurses ask your birthday like a thousand times a day?) got extremely serious; etc. My take: give it a few more years, and software, too, probably will follow the same path. We needs more data, though.

blub 9y ago
Aviation safety is also a domain to look up to.
naasking 9y ago
> My take: give it a few more years, and software, too, probably will follow the same path. I doubt it. The stakes are much lower than people's lives and health.
- marpstar 9y ago
  Agreed. Let's not forget that a huge portion of software written is games.
  - ecshafer 9y ago
    I would bet it's a not huge fraction of the software written is for games. Games are a niche if software.
- lylejohnson 9y ago
  The Atlantic article that Uncle Bob was writing in response to lists several specific examples of software failures that did, or could have, resulted in loss of life: a failure of the 911 system; air traffic control systems; medical devices.
  - naasking 9y ago
    But then these scenarios already have the required incentives. Even software has liability issues if the stakes are this high.
- wtflmaohnisdumb 9y ago
  Our algorithms are also harming people's lives and wealth but no one cares because "computers can't be biased!", and these people are generally poor and have no idea.
sheepmullet 9y ago
> Back then, too, the thoughts on medical safety also were divided into 2 schools: the professionalism and the process oriented The key difference is in the medical world safety has been a primary concern from day one. I.e. There has always been a high level of professionalism. That is not true in the software world. Imagine a doctor saying it's 5pm on a Friday and I'm meeting a friend in an hour so I'll just do a rush job of this surgery and it will probably work out fine. I've seen devs happily check in shoddy work just to be finished hundreds of times in my career.
- blub 9y ago
  I don't know about that... Doctors famously didn't even wash their hands between patients. In fact many still don't. The best practices and process-oriented thinking seem to not be uniformly spread. I'm also reading Normal Accidents by Perrow and what he says about nuclear safety up to the 80s (that's when the book was published) is scary. My thoughts: our safest endeavours look and feel safe, but they are still more failure-prone than one would assume and making them safer is incredibly hard because of social, technical and human issues exhibiting hidden coupling - system issues.
- sgift 9y ago
  > Imagine a doctor saying it's 5pm on a Friday and I'm meeting a friend in an hour so I'll just do a rush job of this surgery and it will probably work out fine. Imaging right now. No problem. Humans are humans. Humans sometimes do rush jobs. Or are overworked. Or stressed. Maybe they have private problems, so their mind isn't 100% on the task. Your suggestion that doctors are all professionals and wouldn't do such things is actually the exact opposite of the medicine professions conclusions: Humans make errors. Doctors, software developers, we all. And what helps to fix these things are better tools and allowing other people to check something (see above, nurses training to question doctors is exactly for this problems).
  - Turing_Machine 9y ago
    > Imaging right now. No problem. Same here. While some argue that the medical malpractice system has been abused, I don't think anyone argues that genuine malpractice doesn't exist. It happens all the time. As you say, doctors are human beings, and sometimes they get lazy, careless, or overworked.
maxxxxx 9y ago
I don't think you can compare software to other disciplines like medical, air transportation or architecture. These areas are well understood and pretty mature and move pretty slowly. If we ran air transportation like software somebody would already have self flying airplanes in service. They would crash from time to time though. I personally like the imperfection of software development and the freedom to imagine new things. If we want to be more mature we also have to accept much slower development cycles and innovation will be slower.
- masklinn 9y ago
  > I don't think you can compare software to other disciplines like medical, air transportation or architecture. These areas are well understood and pretty mature and move pretty slowly. Architecture maybe. Air transportation is barely a century old, though you could make the argument that it's a offshoot of other industrial engineering disciplines. When it comes to medicine however, while modern medicine is older than computer science it's not by much, and what's called medicine until the late 19th or early 20th century is as similar to modern medicine as computers of the early 20th century (aka rooms of people with rulers and mechanical calculators) are to computers of the second half. And medicine has in fact changed at a pretty frenetic pace, the "miasma theory" was only vanquished circa 1880~1890, and we can now sample and edit living beings at coffee-cup price levels.
  - maxxxxx 9y ago
    I still would argue that medical is moving much slower than software. I work in medical devices and simple experiments can take years to get done once humans are involved. In software we would get the result in a few days or weeks. I am not advocating easing the restrictions in human experimentation but these rules definitely slow down progress.
    - Silhouette 9y ago
      I am not advocating easing the restrictions in human experimentation but these rules definitely slow down progress. They surely slow down the work, but whether they slow down useful progress is a different question. Given the amount of time and money that depend on software systems today, as well as the more general effect of software on our quality of life, poor quality software costs society as a whole a great deal. If we built our software more slowly and carefully but also with higher reliability and fewer issues with security, privacy and so on, would we really be worse off?
walterstucco 9y ago
So basically work is safer when done by robots Or if you spend a lot of resources in training people to robotize them I prefer real robots
- koide 9y ago
  Except when something unexpected happens, in which case you'd love having had a human. The best is combined teams of man and machines. This also has its own problems, but nothing is ever perfect.
  - walterstucco 9y ago
    I was being sarcastic The original post was about how professionalism was replaced by method in healthcare But in my opinion saying "you have to be professional" and "you have to follow this checklist" it's the same thinkg A doctor that was amputating the wrong leg without checking was not making a mistake because "tools", but because was being unprofessional. A paramedic that is not checking all the vitals because it wasn't on the checklist, is unprofessional Not having a checklist in the first place is unprofessional! It's the same thing, said in a different way, just to have more doctors onboard willing to be trained to perform procedures like robots, that bring profits into private hospitals. I think it's mainly an US thing though, I've worked in healthcare in Europe and it's not like that Having said that, programmers skill vary a lot based on the type of work they do, the industry they work in, the experience they have, the context, where they come from etc. etc. etc. Uncle Bob might be wrong or not showing enough data, but the lack of tools is still a programmer's fault. You can blame management if they are doing it poorly, but if the tool you're using are not the best for your job, you're just being less than professional. Maybe it's enough most of the time, but that doesn't mean that being professional is something else.
mercer 9y ago
I'm reminded of an article I read a while back about the rather sizable effectiveness of checklists in the health sector, and how they're still not 'standard practice'. I think it's this article, but I'm not sure: http://www.nytimes.com/2010/01/24/books/review/Jauhar-t.html http://www.nytimes.com/2010/01/24/books/review/Jauhar-t.html

by magice 9y ago

blub 9y ago

Aviation safety is also a domain to look up to.

naasking 9y ago

> My take: give it a few more years, and software, too, probably will follow the same path. I doubt it. The stakes are much lower than people's lives and health.

marpstar 9y ago
Agreed. Let's not forget that a huge portion of software written is games.
- ecshafer 9y ago
  I would bet it's a not huge fraction of the software written is for games. Games are a niche if software.
lylejohnson 9y ago
The Atlantic article that Uncle Bob was writing in response to lists several specific examples of software failures that did, or could have, resulted in loss of life: a failure of the 911 system; air traffic control systems; medical devices.
- naasking 9y ago
  But then these scenarios already have the required incentives. Even software has liability issues if the stakes are this high.
wtflmaohnisdumb 9y ago
Our algorithms are also harming people's lives and wealth but no one cares because "computers can't be biased!", and these people are generally poor and have no idea.

sheepmullet 9y ago

> Back then, too, the thoughts on medical safety also were divided into 2 schools: the professionalism and the process oriented The key difference is in the medical world safety has been a primary concern from day one. I.e. There has always been a high level of professionalism. That is not true in the software world. Imagine a doctor saying it's 5pm on a Friday and I'm meeting a friend in an hour so I'll just do a rush job of this surgery and it will probably work out fine. I've seen devs happily check in shoddy work just to be finished hundreds of times in my career.

blub 9y ago
I don't know about that... Doctors famously didn't even wash their hands between patients. In fact many still don't. The best practices and process-oriented thinking seem to not be uniformly spread. I'm also reading Normal Accidents by Perrow and what he says about nuclear safety up to the 80s (that's when the book was published) is scary. My thoughts: our safest endeavours look and feel safe, but they are still more failure-prone than one would assume and making them safer is incredibly hard because of social, technical and human issues exhibiting hidden coupling - system issues.
sgift 9y ago
> Imagine a doctor saying it's 5pm on a Friday and I'm meeting a friend in an hour so I'll just do a rush job of this surgery and it will probably work out fine. Imaging right now. No problem. Humans are humans. Humans sometimes do rush jobs. Or are overworked. Or stressed. Maybe they have private problems, so their mind isn't 100% on the task. Your suggestion that doctors are all professionals and wouldn't do such things is actually the exact opposite of the medicine professions conclusions: Humans make errors. Doctors, software developers, we all. And what helps to fix these things are better tools and allowing other people to check something (see above, nurses training to question doctors is exactly for this problems).
- Turing_Machine 9y ago
  > Imaging right now. No problem. Same here. While some argue that the medical malpractice system has been abused, I don't think anyone argues that genuine malpractice doesn't exist. It happens all the time. As you say, doctors are human beings, and sometimes they get lazy, careless, or overworked.

maxxxxx 9y ago

I don't think you can compare software to other disciplines like medical, air transportation or architecture. These areas are well understood and pretty mature and move pretty slowly. If we ran air transportation like software somebody would already have self flying airplanes in service. They would crash from time to time though. I personally like the imperfection of software development and the freedom to imagine new things. If we want to be more mature we also have to accept much slower development cycles and innovation will be slower.

masklinn 9y ago
> I don't think you can compare software to other disciplines like medical, air transportation or architecture. These areas are well understood and pretty mature and move pretty slowly. Architecture maybe. Air transportation is barely a century old, though you could make the argument that it's a offshoot of other industrial engineering disciplines. When it comes to medicine however, while modern medicine is older than computer science it's not by much, and what's called medicine until the late 19th or early 20th century is as similar to modern medicine as computers of the early 20th century (aka rooms of people with rulers and mechanical calculators) are to computers of the second half. And medicine has in fact changed at a pretty frenetic pace, the "miasma theory" was only vanquished circa 1880~1890, and we can now sample and edit living beings at coffee-cup price levels.
- maxxxxx 9y ago
  I still would argue that medical is moving much slower than software. I work in medical devices and simple experiments can take years to get done once humans are involved. In software we would get the result in a few days or weeks. I am not advocating easing the restrictions in human experimentation but these rules definitely slow down progress.
  - Silhouette 9y ago
    I am not advocating easing the restrictions in human experimentation but these rules definitely slow down progress. They surely slow down the work, but whether they slow down useful progress is a different question. Given the amount of time and money that depend on software systems today, as well as the more general effect of software on our quality of life, poor quality software costs society as a whole a great deal. If we built our software more slowly and carefully but also with higher reliability and fewer issues with security, privacy and so on, would we really be worse off?

walterstucco 9y ago

So basically work is safer when done by robots Or if you spend a lot of resources in training people to robotize them I prefer real robots

koide 9y ago
Except when something unexpected happens, in which case you'd love having had a human. The best is combined teams of man and machines. This also has its own problems, but nothing is ever perfect.
- walterstucco 9y ago
  I was being sarcastic The original post was about how professionalism was replaced by method in healthcare But in my opinion saying "you have to be professional" and "you have to follow this checklist" it's the same thinkg A doctor that was amputating the wrong leg without checking was not making a mistake because "tools", but because was being unprofessional. A paramedic that is not checking all the vitals because it wasn't on the checklist, is unprofessional Not having a checklist in the first place is unprofessional! It's the same thing, said in a different way, just to have more doctors onboard willing to be trained to perform procedures like robots, that bring profits into private hospitals. I think it's mainly an US thing though, I've worked in healthcare in Europe and it's not like that Having said that, programmers skill vary a lot based on the type of work they do, the industry they work in, the experience they have, the context, where they come from etc. etc. etc. Uncle Bob might be wrong or not showing enough data, but the lack of tools is still a programmer's fault. You can blame management if they are doing it poorly, but if the tool you're using are not the best for your job, you're just being less than professional. Maybe it's enough most of the time, but that doesn't mean that being professional is something else.

mercer 9y ago

I'm reminded of an article I read a while back about the rather sizable effectiveness of checklists in the health sector, and how they're still not 'standard practice'. I think it's this article, but I'm not sure: http://www.nytimes.com/2010/01/24/books/review/Jauhar-t.html http://www.nytimes.com/2010/01/24/books/review/Jauhar-t.html