9 ms·

We need formal codes of practice, we need good institutions, but we also need a cultural change. We no longer have the luxury of saying "oh, it's just a CRUD a

We need formal codes of practice, we need good institutions, but we also need a cultural change. We no longer have the luxury of saying "oh, it's just a CRUD app, it doesn't matter if things go wrong". Software has become too important, it has become too deeply intertwined in our daily lives. Errors and leaks from trivial little apps can have life-changing consequences for users. If your software handles personally identifying data, there's a non-zero chance that you could ruin someone's life through negligence. Legal case management software might be an obvious example, but it's the tip of a very big iceberg. If you have several million users, one-in-a-million edge cases are going to happen constantly. Weird things happen at scale. We need to treat every Android permission and every Facebook API call as a potential matter of life or death. We need the personal courage and the support of our peers to say "no, I will not implement that feature", "no, I will not store that user data", "no, I will not transmit that in plaintext", "no, I will not commit that to production without unit testing". https://en.wikipedia.org/wiki/Iron_Ring https://en.wikipedia.org/wiki/Iron_Ring

tener 10y ago
Don't put the blame on the developers. Too frequently they are put under huge pressure and cannot say "no" to their managers. There are people which are responsible for delivery, testing and they are paid way more than the average developer. Unlike in engineering, with software you can - and indeed should - test it before deployment in thousands of ways. Finally there is also a receiving party that should ensure they are not accepting broken software. They have responsibility too.
- CodeMage 10y ago
  I would amend that to "don't put the blame solely on the developers". I've seen way too many developers whose mentality is "ship it, it seems good enough, we'll fix any bugs as we go".
  - tener 10y ago
    If the company culture is supporting this attitude, then sure - you will have developers saying that. But it isn't the developers that create that culture. This is something the management is responsible for.
    - mseebach 10y ago
      Exactly. This is why bosses make the big bucks. Developers are just code-machines and they will perform poorly or well entirely dependent on their external environment and culture created by their management. (Oh, it's a one way thing, you say? Bosses are to blame for failure, but devs get credit for successes? Head I win, tails you lose?)
    - FullMtlAlcoholc 10y ago
      The fact that a person acted pursuant to order of of a superior does not relieve him from responsibility. This can be a difficult decision to make in these circumstances, but I just couldn't ship buggy code that may be responsible for someone being unfairly jailed. That's why I try to always document as much as possible in the workplace.
    - CodeMage 10y ago
      People aren't computers, they don't just run the company culture program in their brains. Sure, developers are influenced by company culture, but they contribute to it as well. At one end of the spectrum, you have junior devs, fresh out of school, highly susceptible to the effects of company culture. At the other, you have senior devs with more than a decade of experience, who do a great deal to set the tone and shape the culture. Thank about what kind of cultural influence you get from a senior dev who has accumulated more than ten years of experience of moving fast and breaking things. That's a lesson that I found very hard to learn: just because someone has lots of experience and skill and knowledge, doesn't mean they are better developers. Mind you, I'm not saying the blame for the state of our industry lies solely with devs. I agree with you that it's the management that has the last word, but that doesn't absolve the rest of us of all responsibility.
      - tener 10y ago
  - 0xdeadbeefbabe 10y ago
    Even bringing developers (or anyone really) into it is like blaming the prisoners for having a dilemma. It's the nature of software that it does not fit inside one person's head, and this leads to problems.
  - tomjen3 10y ago
    Seems perfectly reasonable to me.
- jdietrich 10y ago
  >Don't put the blame on the developers. Too frequently they are put under huge pressure and cannot say "no" to their managers. Befehl ist Befehl. Blaming management is literally the Nuremberg defence. If we feel that we have to do reckless and dangerous things in order to keep our jobs, then we desperately need to unionise. It's just not good enough for us to throw up our hands and blame the PHBs. Management need to change, but they're not going to do it voluntarily. Pressure needs to come from users, but also from developers. We need to fight back against poor practices, as millions of workers have done in other industries. We're the guys on the shop floor, we're the people who know what the issues are and how to fix them. If we don't take a stand, nobody else will. How would we react if this story was about a plane crash or a nuclear accident caused by negligence? Would we be so quick to absolve employees of responsibility, or would we be asking why nobody blew the whistle?
  - Terr_ 10y ago
    > Nuremberg defence Except we're not just talking about clearly malicious or fraudulent software with sins of commission. The bigger issue is cutting corners, taking shortcuts, and individual weaknesses that break the whole system.
    - heisenbit 10y ago
      Successive and cumulative corner cutting. And management should provide oversight covering several steps. But often they close their eyes and the last developer who touched the code is getting the blame.
  - Spivak 10y ago
    > Befehl ist Befehl. Blaming management is literally the Nuremberg defence. This isn't really about shifting blame or absolving responsibility. It is everyone's responsibility to ensure that safety and the public good are put first. If developers don't have the ability to say no to their management because there's no institution backing up that decision then we are to blame. I know it's a cynical world view but management will trade just about anything for profits and employees will compromise their principles for their livelihood. Hoping that this system of perverse incentives will correct itself internally using only the individual employee's right to resign far too hopeful for my taste.
- ThatGeoGuy 10y ago
  > Too frequently they are put under huge pressure and cannot say "no" to their managers. Actually, this is precisely why something like the "Iron Ring", linked above, exists. Standards bodies like APEGA (https://www.apega.ca https://www.apega.ca) exist precisely so that you have someone backing you if you say "no" to a manager. If you're an engineer and you put the public at harm in any way, APEGA will come after you. Part of being able to call yourself an engineer in Canada is the distinction that specifically you won't just say "yes" and bend over because your manager wills it. Your duty is first and foremost to the public. This is something that works incredibly well for engineering, geology, and geoscientists, and I can't see why it won't work for software. Developers who are contracted to build systems for the public like this absolutely _should_ have liability here.
  - tener 10y ago
    Software projects that have significant impact on public safety/well being are a relatively rare kind. You can easily apply additional policies on them - and indeed this is what is being done (see airplane software, medical software etc.). But will you really call your union because your manager forces you to skip writing unit tests for each and every class you write? Having in mind you are just churning out some crappy e-commerce website? Completely another thing is that you might be careful with your code, the third party components you are using are more lax. This is well recognized e.g. in medical software where any third party code must identified and held to the same standards as the software you write. It is very difficult to say who is to blame for creating a certain software bugs. Is it the person that configures the software, becuase he chose invalid config parameters? Is it the intern that wrote invalid piece of parsing code? Is it the system architect who put wrong data into the specification? For any given piece of software there are many people responsible in various ways for its creation. In such contexts you cannot be blaming individual persons especially that their intentions might be good. Even very careful programmers will make bugs.
    - Retric 10y ago
      This comes back to how software is designed and built. On it's own a single bad weld should never endanger a bridge, but we accept a single mistake can bring down software. At it's core that's why software fails so often AND that's what we need to fix.
JadeNB 10y ago
> If your software handles personally identifying data, there's a non-zero chance that you could ruin someone's life through negligence. And, at the scale you're discussing, if there is a non-zero chance that this will happen in any given instance, then there is a near-100% chance that it will happen in some instance.
treehau5 10y ago
Meh. That's all I have to say about that. We already have the distinction between software that can actually affect people's lives in a negative/positive way and software that is merely a nuance or inconvenient. The former is heavily regulated. Mistakes in financial accounting software, healthcare software, or mission critical software such as industrial automation or say launching satellites cause companies to vanish Your task management crud app that crashes every now and then isn't going to cause you to die. Marketing and advertisement software may be a large nuisance, but it isn't killing anyone. I've worked in both environments. The cultures are appropriate for each of the given situations.
- jonathankoren 10y ago
  No we don't have that distinction. We have places where outsiders, typically from heavily regulated industries like finance, like medicine, like areospace, have imposed testing and validation procedures and best practices onto the software industry. Left to our own devices, we get IoT bullshit like Nest thermostats that randomly turn the temperature down because they turn off[0], security exploit via unneeded running service part 2028, the 9382nd installment of security exploit via default credentials, the 11 millionth edition of buffer overflow security exploit, and the list goes on. These problems screw up not just it, but other systems (see mirari dns ddos, see ransomware, see ransomware on MUNI, see all the other effects from people that coded up things that relied on a ubiquitous persistent internet connection) [0] http://mobile.nytimes.com/2016/01/14/fashion/nest-thermostat-glitch-battery-dies-software-freeze.html?referer= http://mobile.nytimes.com/2016/01/14/fashion/nest-thermostat...
  - noir_lord 10y ago
    Completely agree with all of this and it has to be addressed soon.
- jdietrich 10y ago
  There's no such thing as harmless software any more. Our devices are too interconnected and we are too reliant upon them. Failure modes can cascade in unpredictable ways. "Harmless" systems often contain potentially harmful data; highly sensitive systems often share network resources with IoT junk. Some semi-hypothetical scenarios: A data leak in an appointment management app allows an abusive ex-partner to track down and kill their victim. Mobile malware spread by an in-game advertising network causes a massive DDoS of the cellular network. Some areas are left without cell coverage for several hours while operators scramble to reconfigure their base station equipment; thousands of 911 calls go unanswered until service is resumed. Poor encryption in a chat app allows an oppressive regime to identify, surveil and "disappear" human rights activists. A bug in an integrated logistics management system causes some parcels to be tagged with the wrong attributes. A consignment of temperature-sensitive medicine is inadvertently shipped in an unrefrigerated container; the mistake goes unnoticed until patients start dying, because the "smart" temperature sensor inside the package thought that the shipment was safe to transport at ambient temperatures. A disreputable dating app is hacked and the names of the users published, leading to several suicides. A wifi-enabled printer is connected to the secure side of a hospital network by an unwitting member of staff. The printer is pwned, bringing down the entire network for 36 hours. Two ICU patients die as a result of preventable errors, because their medical records were unavailable. Several models of aftermarket in-car entertainment units are pwned due to an unpatched vulnerability in Android. The units are connected to the CAN-bus in order to use the media controls on the steering wheel. The poorly-written malware intermittently floods the CAN-bus with malformed packets, causing sudden failures of the braking system in several cars. I could go on ad nauseum.

tener 10y ago

Don't put the blame on the developers. Too frequently they are put under huge pressure and cannot say "no" to their managers. There are people which are responsible for delivery, testing and they are paid way more than the average developer. Unlike in engineering, with software you can - and indeed should - test it before deployment in thousands of ways. Finally there is also a receiving party that should ensure they are not accepting broken software. They have responsibility too.

CodeMage 10y ago
I would amend that to "don't put the blame solely on the developers". I've seen way too many developers whose mentality is "ship it, it seems good enough, we'll fix any bugs as we go".
- tener 10y ago
  If the company culture is supporting this attitude, then sure - you will have developers saying that. But it isn't the developers that create that culture. This is something the management is responsible for.
  - mseebach 10y ago
    Exactly. This is why bosses make the big bucks. Developers are just code-machines and they will perform poorly or well entirely dependent on their external environment and culture created by their management. (Oh, it's a one way thing, you say? Bosses are to blame for failure, but devs get credit for successes? Head I win, tails you lose?)
  - FullMtlAlcoholc 10y ago
    The fact that a person acted pursuant to order of of a superior does not relieve him from responsibility. This can be a difficult decision to make in these circumstances, but I just couldn't ship buggy code that may be responsible for someone being unfairly jailed. That's why I try to always document as much as possible in the workplace.
  - CodeMage 10y ago
    People aren't computers, they don't just run the company culture program in their brains. Sure, developers are influenced by company culture, but they contribute to it as well. At one end of the spectrum, you have junior devs, fresh out of school, highly susceptible to the effects of company culture. At the other, you have senior devs with more than a decade of experience, who do a great deal to set the tone and shape the culture. Thank about what kind of cultural influence you get from a senior dev who has accumulated more than ten years of experience of moving fast and breaking things. That's a lesson that I found very hard to learn: just because someone has lots of experience and skill and knowledge, doesn't mean they are better developers. Mind you, I'm not saying the blame for the state of our industry lies solely with devs. I agree with you that it's the management that has the last word, but that doesn't absolve the rest of us of all responsibility.
    - tener 10y ago
- 0xdeadbeefbabe 10y ago
  Even bringing developers (or anyone really) into it is like blaming the prisoners for having a dilemma. It's the nature of software that it does not fit inside one person's head, and this leads to problems.
- tomjen3 10y ago
  Seems perfectly reasonable to me.
jdietrich 10y ago
>Don't put the blame on the developers. Too frequently they are put under huge pressure and cannot say "no" to their managers. Befehl ist Befehl. Blaming management is literally the Nuremberg defence. If we feel that we have to do reckless and dangerous things in order to keep our jobs, then we desperately need to unionise. It's just not good enough for us to throw up our hands and blame the PHBs. Management need to change, but they're not going to do it voluntarily. Pressure needs to come from users, but also from developers. We need to fight back against poor practices, as millions of workers have done in other industries. We're the guys on the shop floor, we're the people who know what the issues are and how to fix them. If we don't take a stand, nobody else will. How would we react if this story was about a plane crash or a nuclear accident caused by negligence? Would we be so quick to absolve employees of responsibility, or would we be asking why nobody blew the whistle?
- Terr_ 10y ago
  > Nuremberg defence Except we're not just talking about clearly malicious or fraudulent software with sins of commission. The bigger issue is cutting corners, taking shortcuts, and individual weaknesses that break the whole system.
  - heisenbit 10y ago
    Successive and cumulative corner cutting. And management should provide oversight covering several steps. But often they close their eyes and the last developer who touched the code is getting the blame.
- Spivak 10y ago
  > Befehl ist Befehl. Blaming management is literally the Nuremberg defence. This isn't really about shifting blame or absolving responsibility. It is everyone's responsibility to ensure that safety and the public good are put first. If developers don't have the ability to say no to their management because there's no institution backing up that decision then we are to blame. I know it's a cynical world view but management will trade just about anything for profits and employees will compromise their principles for their livelihood. Hoping that this system of perverse incentives will correct itself internally using only the individual employee's right to resign far too hopeful for my taste.
ThatGeoGuy 10y ago
> Too frequently they are put under huge pressure and cannot say "no" to their managers. Actually, this is precisely why something like the "Iron Ring", linked above, exists. Standards bodies like APEGA (https://www.apega.ca https://www.apega.ca) exist precisely so that you have someone backing you if you say "no" to a manager. If you're an engineer and you put the public at harm in any way, APEGA will come after you. Part of being able to call yourself an engineer in Canada is the distinction that specifically you won't just say "yes" and bend over because your manager wills it. Your duty is first and foremost to the public. This is something that works incredibly well for engineering, geology, and geoscientists, and I can't see why it won't work for software. Developers who are contracted to build systems for the public like this absolutely _should_ have liability here.
- tener 10y ago
  Software projects that have significant impact on public safety/well being are a relatively rare kind. You can easily apply additional policies on them - and indeed this is what is being done (see airplane software, medical software etc.). But will you really call your union because your manager forces you to skip writing unit tests for each and every class you write? Having in mind you are just churning out some crappy e-commerce website? Completely another thing is that you might be careful with your code, the third party components you are using are more lax. This is well recognized e.g. in medical software where any third party code must identified and held to the same standards as the software you write. It is very difficult to say who is to blame for creating a certain software bugs. Is it the person that configures the software, becuase he chose invalid config parameters? Is it the intern that wrote invalid piece of parsing code? Is it the system architect who put wrong data into the specification? For any given piece of software there are many people responsible in various ways for its creation. In such contexts you cannot be blaming individual persons especially that their intentions might be good. Even very careful programmers will make bugs.
  - Retric 10y ago
    This comes back to how software is designed and built. On it's own a single bad weld should never endanger a bridge, but we accept a single mistake can bring down software. At it's core that's why software fails so often AND that's what we need to fix.

JadeNB 10y ago

> If your software handles personally identifying data, there's a non-zero chance that you could ruin someone's life through negligence. And, at the scale you're discussing, if there is a non-zero chance that this will happen in any given instance, then there is a near-100% chance that it will happen in some instance.

treehau5 10y ago

Meh. That's all I have to say about that. We already have the distinction between software that can actually affect people's lives in a negative/positive way and software that is merely a nuance or inconvenient. The former is heavily regulated. Mistakes in financial accounting software, healthcare software, or mission critical software such as industrial automation or say launching satellites cause companies to vanish Your task management crud app that crashes every now and then isn't going to cause you to die. Marketing and advertisement software may be a large nuisance, but it isn't killing anyone. I've worked in both environments. The cultures are appropriate for each of the given situations.

jonathankoren 10y ago
No we don't have that distinction. We have places where outsiders, typically from heavily regulated industries like finance, like medicine, like areospace, have imposed testing and validation procedures and best practices onto the software industry. Left to our own devices, we get IoT bullshit like Nest thermostats that randomly turn the temperature down because they turn off[0], security exploit via unneeded running service part 2028, the 9382nd installment of security exploit via default credentials, the 11 millionth edition of buffer overflow security exploit, and the list goes on. These problems screw up not just it, but other systems (see mirari dns ddos, see ransomware, see ransomware on MUNI, see all the other effects from people that coded up things that relied on a ubiquitous persistent internet connection) [0] http://mobile.nytimes.com/2016/01/14/fashion/nest-thermostat-glitch-battery-dies-software-freeze.html?referer= http://mobile.nytimes.com/2016/01/14/fashion/nest-thermostat...
- noir_lord 10y ago
  Completely agree with all of this and it has to be addressed soon.
jdietrich 10y ago
There's no such thing as harmless software any more. Our devices are too interconnected and we are too reliant upon them. Failure modes can cascade in unpredictable ways. "Harmless" systems often contain potentially harmful data; highly sensitive systems often share network resources with IoT junk. Some semi-hypothetical scenarios: A data leak in an appointment management app allows an abusive ex-partner to track down and kill their victim. Mobile malware spread by an in-game advertising network causes a massive DDoS of the cellular network. Some areas are left without cell coverage for several hours while operators scramble to reconfigure their base station equipment; thousands of 911 calls go unanswered until service is resumed. Poor encryption in a chat app allows an oppressive regime to identify, surveil and "disappear" human rights activists. A bug in an integrated logistics management system causes some parcels to be tagged with the wrong attributes. A consignment of temperature-sensitive medicine is inadvertently shipped in an unrefrigerated container; the mistake goes unnoticed until patients start dying, because the "smart" temperature sensor inside the package thought that the shipment was safe to transport at ambient temperatures. A disreputable dating app is hacked and the names of the users published, leading to several suicides. A wifi-enabled printer is connected to the secure side of a hospital network by an unwitting member of staff. The printer is pwned, bringing down the entire network for 36 hours. Two ICU patients die as a result of preventable errors, because their medical records were unavailable. Several models of aftermarket in-car entertainment units are pwned due to an unpatched vulnerability in Android. The units are connected to the CAN-bus in order to use the media controls on the steering wheel. The poorly-written malware intermittently floods the CAN-bus with malformed packets, causing sudden failures of the braking system in several cars. I could go on ad nauseum.