7 ms·

Let's blame the dev who pressed "Deploy"

84 points by skwee357 2y ago

chrisjj 2y ago
> it makes sense to run EDR on a mission-critical machine, but on a dumb display of information Major logic error. See how much chaos ensued when that display went down? That's why it needs protection to keep the display up. Dumb or not is irrelevent. It is mission-critical regardless.
- skwee357 2y ago
  My point was, why a display at a check in counter needs to run EDR software to begin with? Why can't it run a locked down, slimmed version of Windows in an isolated network that has very low potential to get malware on it? I know why, because there is, probably, a regulation that says that if you run an airline company, you need to have malware protection on all machines. I bet, some IT guy even tried to question the need to run EDR on a non-mission-critical machine, but he was stopped by a wall of "it is what it is".
  - 0xBDB 2y ago
    >My point was, why a display at a check in counter needs to run EDR software to begin with? Because the thermostat on a fish tank has been used as a critical entry point into a casino network[1], and the point of EDR is not just to prevent that sort of thing if possible but also provide the telemetry into a SIEM for incident responders to know that it has happened after the fact and get the adversary out. So there is value in running it anywhere it can run. I've seen a lot of contempt on HN threads today for compliance regulations and insurance demands that require things like EDR be installed where possible. As a Red Teamer I used to share that contempt for the non-technical types, but I don't now. It's true compliance is not security, but also true that Chesterton's Fence should apply here: just because you shouldn't be checking the box blindly doesn't mean you shouldn't be either checking it or documenting why not. The people who created the box were (probably) not actually idiots. It's there because somebody else had a very bad day. 1. https://www.washingtonpost.com/news/innovations/wp/2017/07/21/how-a-fish-tank-helped-hack-a-casino/ https://www.washingtonpost.com/news/innovations/wp/2017/07/2...
  - passwordoops 2y ago
    "I know why, because there is, probably, a regulation" Instead of assuming a regulation and writing a blog about it, do the research and find out. To quote the irreplaceable Benny Hill, "You mustn't assume, because it will make an ass out of you and me." Also, and more important, why default to regulation and not airline directors pushing ill-advised modernization strategies pushed by M$?
    - ses1984 2y ago
      I've done the research. It’s called first hand experience. I was the guy making the arguments, that controls we already have in place obviate the need for edr everywhere, but I was told it doesn’t matter, gotta check the box.
      - passwordoops 2y ago
        Awesome. Is the box there because of government regulation or someone in corporate deciding it's necessary?
        betaby 2y ago
        Corporation and consultants mostly, judging from my experience. If asked about precise law or regulation they just wave hands.
        ses1984 2y ago
        In my case it was a pci-dss (payment card industry data security standard) audit.
        moomin 2y ago
        The thing is, you read regulations, and they pretty much always tell you to do something, but it’s always heavily principle based. Companies are left with extraordinary leeway as to how these regulations are actually implemented.
        2y ago
  - chrisjj 2y ago
    Because isolating the display and every machine to which it is necessarily connected obstructs monitoring and greatly increases the cost and delay of fixes if something should go wrong. Also I doubt any slimmed version of Windows is sufficiently malware proof without added EPS.
  - hug 2y ago
    Low potential is not no potential, and most everyone is looking for swiss-cheese defense when it comes to these devices. In the case of a display at a check-in counter: - The display needs to be on a network, because it needs to collect information from elsewhere to display it. - It's on a network, so it needs to be kept updated, because a compromised host elsewhere on the same network will be able to compromise it, and anyway the display vendor won't support you if your product is nine versions behind current. - Since it needs updates for various components, it almost certainly needs some amount of outbound internet access, and it's also vulnerable to supply-chain attacks from those updates. - Since it is on a network, and has internet access, it needs to be running some kind of EDR or feed for a SIEM, because it is compromisable and the last thing you want is an unmonitored compromised host on your internal network talking back to C2. Anything that can be used for lateral movement will be used for lateral movement, and if we can get logs from it we want logs from it. A cross-platform EDR solution is perfect for these scenarios.
    - chrisjj 2y ago
      Agreed. Re: "- It's on a network, so it needs to be kept updated, because a compromised host elsewhere on the same network will be able to compromise it" the suggested solution was "an isolated network".0 The problem there is the operator would have to use SD cards to update the adverts... :)
- sussmannbaka 2y ago
  the more software you add to the display, including EDR, the higher the chance it will go down. you don’t reduce attack surface by introducing more surface and you don’t improve uptime by introducing additional single point of failures.
  - chrisjj 2y ago
    > the more software you add to the display, including EDR, the higher the chance it will go down I am pretty sure that removing EDR from an internetted Windows airport display would ensure it goes down. And probably then come up with a ransomware demand.
    - sussmannbaka 2y ago
      Why would it do that? How would a ransomware demand come up on it? Honestly, please tell me because this sounds utterly outlandish to me.
      - chrisjj 2y ago
        It would do that because the internet is awash with malware.
        sussmannbaka 2y ago
        Is the airport display downloading charlixcx.mp3.exe? Is it clicking malware attachments? I seriously want to know.
Retr0id 2y ago
A slightly bewildering fact is that CrowdStrike's terms and conditions say not to use it for critical infrastructure: > Neither the offerings nor crowdstrike tools are for use in the operation of aircraft navigation, nuclear facilities, communication systems, weapons systems, direct or indirect life-support systems, air traffic control, or any application or installation where failure could result in death, severe physical injury, or property damage. (Originally in all-caps, presumably to make it sound more legally binding) https://www.crowdstrike.com/terms-conditions/ https://www.crowdstrike.com/terms-conditions/
- lbourdages 2y ago
  That's a pretty standard clause I've seen in every software ToS I ever gazed my eyes upon. As far as I know, those applications weren't affected either - airports were affected because the ticketing systems were offline. Companies not using Crowdstrike were able to fly just fine. Edit: 911 was affected in many places, and that is definitely concerning as it is very much life critical.
  - Retr0id 2y ago
    Sure, the small-print is always going to say "anything bad that happens is someone else's fault", but it's a bit embarrassing when you put it next to their marketing materials.
    - 01HNNWZ0MV43FF 2y ago
      I recently was asked to sign a thing saying it's my fault if I die while getting my hair colored
      - gibbitz 2y ago
        Oddly this is a thing: https://en.m.wikipedia.org/wiki/Vertebrobasilar_insufficiency https://en.m.wikipedia.org/wiki/Vertebrobasilar_insufficienc...
pylua 2y ago
The practice of building software would have to completely be flipped on its head to support professional liability. Agile would be the first casualty.
- ClumsyPilot 2y ago
  I think Boeing clearly tried to apply Agile to their planes
  - synicalx 2y ago
    And their doors as well apparently, those are VERY agile.
JKCalhoun 2y ago
Sure, blame the engineer(s). And from now on every engineer now has the autonomy to refuse to use certain libraries and software stacks they are unfamiliar with, can refuse to submit a change, will have control over whether to push out software they worked on, etc. And a huge pay bonus as well since they have all this CEO-like risk/responsibility now.
- mariusor 2y ago
  Frankly, sign me up for that job, please.
- jfim 2y ago
  That level of agency exists today. Any software developer can tell their manager that no, they're not doing X. The risk is that they get written up for insubordination or have to look for new employment. What the author of the article doesn't know is that the structural engineer can also get fired if management deems the engineer too troublesome. The only difference is that the PE has a code of ethics and professional responsibility to uphold, and failing to do so would mean risking their license to practice engineering.
  - betaby 2y ago
    In practice PE rarely held accountable in Quebec.
  - strken 2y ago
    The most important difference is that in practice the SWE can't go and report the employer to anyone for breaching e.g. occupational health and safety laws, or any others, because those laws don't exist for the majority of software flaws. At best you have privacy laws like the GDPR or industry-specific ones covering e.g. medical devices...but CrowdStrike isn't meant to run in those environments anyway, see, look at this text buried in the license agreement.
deleted 2y ago
[deleted]
skwirl 2y ago
> If a software engineer tells you that this code needs to be 100% test covered, that AI won’t replace them, and that they need 3 months of development—then you better shut the fuck up and let them do their job. This is so naive that it is impossible to take the author seriously.
- ses1984 2y ago
  It’s called hyperbole.
  - skwirl 2y ago
    No, this paragraph was not hyperbole. This was in conclusion and the author’s serious opinion.
    - Kiro 2y ago
      You're replying to the author but I agree. Too much unfounded speculation. I'm glad the author is being called out for lack of research in a sibling comment.
    - skwee357 2y ago
      It was a hyperbolic statement related to the fact that if you want developers to take full responsibility, like surgeons, then you need trust their authority. I haven’t heard of a surgeon who said “this operation will take as much time as needed”, and the hospital manager pressured him to “finish it in 8 hours, and not use too many syringes”.
    - ses1984 2y ago
      The whole post is a rant.
- jkercher 2y ago
  What's naive?
- dakiol 2y ago
  Who else is in a better position to give an estimate than the people who design and build the thing?
softwaredoug 2y ago
OK but structural engineering and anesthesiology are sciences with a lot of hard data behind them… … arguing for 100% test coverage or SOLID principles are more like philosophies and anecdotes without a lot of hard data supporting them. Software engineering looks less like engineering and more like a lot of bike shedding conversations.
- ClumsyPilot 2y ago
  We do have hard data as well, we have safe languages like ADA and Erlang, we have embedded engineers and special operating systems that are embedded with you applications. It’s just nobody wants to pay for these specialist skills and would rather slap something on top of windows and call it a day
- skwee357 2y ago
  Structural engineer can refuse to sign off a design that is not 100% fail proof. If he will be pressured into approving such design, he will ask the pressurer to put his own signature and bear the risk. Now try to pushback on your managers request to “cut this long deploy process just once because this big client wants it fast”.
- pylua 2y ago
  In software development, the edge cases are hardly every fully documented or understood. That is because software is so malleable that there are so many different input / output combinations. Other engineering professions are much more constrained on their input.
Aurornis 2y ago
Responsibility lies on large numbers of processes, teams, and people. I thought this blog might have some substance about proper postmortem investigations and how to evaluate and address the circumstances that led to a failure like this, but it has none of that. It’s just a very angry rant about CEOs and middle management. The premise is that engineers can’t bear any responsibility for their actions because they don’t get “respect” This has to be the 10th time I’ve seen arguments that “blame” is the right action in this case, but with the key exception that we’re only allowed to blame people other than the engineers. The last article was a lengthy rant about how it’s actually QA’s fault and engineers shouldn’t be expected to ensure their own code is correct, therefore engineers are blameless. This is empty calories for people who like ragebait, but nothing more.
- handwarmers 2y ago
  The blog responds to a very particular aspect of the fallout from the crowdstrike outage. The "Responsibility lies on large numbers of processes, teams, and people" was actually addressed in the article. It makes the case that executives claim that responsibility is correlated to pay. All the author asks in this case is for them to walk the walk. Go back to middle management.
- mondrian 2y ago
  > because they don’t get “respect” Nah, that's not what the blog post says. It says engineers aren't given sufficient responsibility, so that they can assume the respective blame, when something goes wrong. If an engineer says it would take 1 month to build a feature so it's sufficiently reliable, and the manager says "nah that's ridiculous, you have 1 week", then when the cobbled together feature breaks in production the manager should take the blame, since they effectively took the responsibility away from the engineer, upon themselves.
  - 9dev 2y ago
    I used to think that way, until I had my own junior developers, and felt like that guy trying to bake a cake with three beagles: They veer off, constantly, in all directions. Developers deep in the trenches tend to have a bad feeling for business requirements or constraints; coupled with a knack for perfectionism and premature optimisation, that really often results in ridiculous time frames that are just plain unrealistic and would ruin the organisation long term. I don’t have any profound insights, though: The only sane mantra can be keeping things in balance. Too much management, you drive your devs insane; too much engineer control, and the architecture astronauts reinvent the wheel every other day.
jmclnx 2y ago
>and that they need 3 months of development then you better shut the f** up and let them do their job. This in a way covers the issue. Never in all my decades developing have had an estimate accepted. All the developers get is "It must be done by ...., no exceptions". So you end up pulling all nighters for weeks, and because you are tired, errors or bad decisions creep in. So yes, the issue is fully with upper management. I left a company because a big project was about to start and I could see it would be a big cluster**. People who stayed told me that is what happened.
batch12 2y ago
Sure, the dev is wrong, but so is the process that allowed their error to impact the product. If a single person can make this choice on accident, then a single person can make it on purpose either by being malicious themselves or being otherwise compromised. If the company has hung their entire security posture and operational success on the choices of one person, they have a problem. Especially a security company.
giantg2 2y ago
'But then the author engages in an absurd rant about how the entire software engineering industry is a “bit of a clusterfuck”,' I mean, it is a bit of a cluster fuck. Most of the issues I've seen are due to speed or cost. We don't have time for tests.we don't have time to record the business requirements in a collective place (just look through multiple JIRA stories and piece it together). We don't have money for dedicated QA roles. So of course problems happen. Luckily my team only works on a lower criticality site. None of this is really a dev's fault. It's leadership and the culture they incentivise.
- acdha 2y ago
  I blame the courts largely buying the argument that companies can disclaim all responsibility with terms of service. Most of these problems would become far less common if the situation was more balanced and managers had to consider the possibility of serious financial consequences for rushing something out without adequate testing.
  - giantg2 2y ago
    I mostly agree, but that still won't fix this issue. Nothing has 100% uptime. Let's say Crowdstrike has an SLA of 99.99%. Even with this outage, they probably meet that depeding on the time frame. So which mangers bear the blame here? The ones who likely met their SLAs, or the ones who knew their systems relied on software that wasn't 100% reliable and didn't have a backup plan?
    - acdha 2y ago
      Pure SLAs matter less than the negligence: this showed critical lack of testing, following robust engineering practices for progressive deployments or fail safe design, etc. If a bank loses $1M they don’t get to argue that they shouldn’t compensate clients because they processed 99.9999% of their transactions correctly.
      - giantg2 2y ago
        In that example, the banks wouldn't compensate people if their payment system was down and couldn't process orders. That really what happened here - loss of sales on the clients' end. If it were securities, then they would have as-of processing to fix it. But the same argument about lack of testing can be made for the companies using the system. Do they have tests for what happens when pieces of their systems or infrastructure go down? Did they ever do a disaster recovery drill? Honestly, there's blame all around because almost nobody is doing it right. Even the ones who are have failures.
potatoman22 2y ago
As a developer, it's nice to shift the blame to other people, and other people do share some of the blame. But we also need to be responsible for our code functioning. It reads like the author is looking for anyone else to blame besides the devs.
hypeatei 2y ago
It's very surprising that critical infrastructure has been "infected" by the compliance checkbox obsession rather than just being designed in a way that completely eliminates the need for any shitty off-the-shelf security product. There is incompetence and complacency showing at every level after the CrowdStrike outage. Both the people selling CrowdStrike and the ones implementing it.
gizmo 2y ago
The author argues that the CEO of CrowdStrike has failed upwards. I don't agree. CrowdStrike makes compliance software. CrowdStrike's main purpose isn't to provide protection against cyber attacks. Businesses don't care enough about that. Businesses do care about simplifying their compliance burden and limiting their liability when they get hacked. That's where CrowdStrike excels, and that's why CrowdStrike can charge so much for their services. It's not easy to build a 70 billion dollar business and CrowdStrike serves a real business need. > I remember times when leaders had dignity and self-respect. They would go on stage and apologize Etiquette rules change over time, but a constant throughout history is that people in power don't take responsibility voluntarily. The "good old days" where leaders had dignity and self-respect never existed. > [...] delusional claim how software engineers should bear the responsibility for bugs and outages The /opinion/ that people /should/ be held responsible can't be delusional. Apparently the author believes that software engineers should bear zero responsibility -- even when their software kills people -- because we don't get enough respect. I don't agree with that opinion but it's a bit rich to call other people delusional when making one unfounded claim after another. The post as a whole is way too angry and too cynical for me.
josephg 2y ago
You know, I sort of agree with this article while completely disagreeing with it. The article says: > You want software engineers to be accountable for their code, then give them the respect they deserve. The problem is, respect is something that's taken as much as its something given. We can't even decide for ourselves if software is worthy of respect. Can anyone learn to code from a coding bootcamp, where getting a job is all that matters? Or is it a discipline that takes years to master, where mistakes can and will bring down the global economy? Are we glorified plumbers, or are we mathematicians and civil engineers combined? If you see yourself as a code monkey, of course you can't be "held responsible" for the results of your work. Coding bootcamps don't teach infosec. Its up to your company to set good practices and your job is just to follow them. Its only if you personally want to take your role in society seriously that it makes sense to consider not just your job, but the effect your job has on the wider world. I'm personally of the opinion that this mindset is almost always long term positive for your career. Its less "blame the dev for hitting deploy" and more "I'm the dev. No, I won't hit deploy on that code in the state its in." I didn't go to a coding bootcamp. I went to university. There they forced all of us engineering & CS students to do an ethics course - which was actually fantastic. There they taught us about the Therac-25: a computer controlled radiation machine which killed a bunch of people. The engineers on the ground knew that it needed more testing, but the company insisted it was fine and pushed it out the door. Here's the question: If you were one of those engineers, what would you do? If you knew, or suspected, that a bug in your code could bring down 911 services and hospitals, ground planes or give people a lethal dose of radiation, do you really trust your manager to make the engineering call? Do you think the CEO understands the risk that is being taken by hitting deploy? Of course, if we're playing the blame game, the blame ultimately the blame falls on the CEO of the company or something. But forget the blame game. You won't be fired. You won't lose your cushy job. The question is: Who do you want to be in situations like this? Think about this question now. You won't have time in the moment when your boss tells you to hit deploy, and you have second thoughts. Me? I want to be someone who would say no.
- skwee357 2y ago
  I agree. But in order to be the "person who would say no", the industry needs to understand that your opinion and expertise--matter. It could be a cultural shift, or a gate-keeping style shift where we protect the title "Engineer", like they do in some professions in some countries. But given the current state, you can't on one hand blame the developers, and on the other hand treat them like spoiled kids who make too much money and in any way AI can replace most of them. It doesn't work this way. A Structural Engineer bears the responsibility because he has the authority, and respect to his knowledge, to refuse to sign off a broken design. This is not the case in software engineering.
betaby 2y ago
There is no law requiring EDR. It's purely constructed nonsense copied by security clowns from one company to another. Now it's kind of mandatory. It has self regulated everything to the ground
dkarter 2y ago
Weak post. Agree with others that this is just rage bait lacking substance. Blaming the developers (or any specific individual/group for that matter) is a cop out, it’s easy and lazy and doesn’t get to the root of the problem, which is more often than not a lack of processes, tools, information and lack of time/desire from leadership to address “technical debt” (for lack of a better term), no matter how many times the devs bring that up. When you blame an individual or a group you can close the case shut on the post mortem and not get to any substantive improvements, meaning this can and will happen again, just to somebody else. That’s why blameless postmortem and a blameless culture is so important. This is a good article about that philosophy: > My summary of blameless culture is: when there is an outage, incident, or escaped bug in your service, assume the individuals involved had the best of intentions, and either they did not have the correct information to make a better decision, or the tools allowed them to make a mistake. https://www.gybe.ca/a-few-words-about-blameless-culture/ https://www.gybe.ca/a-few-words-about-blameless-culture/

84 points by skwee357 2y ago

> it makes sense to run EDR on a mission-critical machine, but on a dumb display of information Major logic error. See how much chaos ensued when that display went down? That's why it needs protection to keep the display up. Dumb or not is irrelevent. It is mission-critical regardless.

skwee357 2y ago
My point was, why a display at a check in counter needs to run EDR software to begin with? Why can't it run a locked down, slimmed version of Windows in an isolated network that has very low potential to get malware on it? I know why, because there is, probably, a regulation that says that if you run an airline company, you need to have malware protection on all machines. I bet, some IT guy even tried to question the need to run EDR on a non-mission-critical machine, but he was stopped by a wall of "it is what it is".
- 0xBDB 2y ago
  >My point was, why a display at a check in counter needs to run EDR software to begin with? Because the thermostat on a fish tank has been used as a critical entry point into a casino network[1], and the point of EDR is not just to prevent that sort of thing if possible but also provide the telemetry into a SIEM for incident responders to know that it has happened after the fact and get the adversary out. So there is value in running it anywhere it can run. I've seen a lot of contempt on HN threads today for compliance regulations and insurance demands that require things like EDR be installed where possible. As a Red Teamer I used to share that contempt for the non-technical types, but I don't now. It's true compliance is not security, but also true that Chesterton's Fence should apply here: just because you shouldn't be checking the box blindly doesn't mean you shouldn't be either checking it or documenting why not. The people who created the box were (probably) not actually idiots. It's there because somebody else had a very bad day. 1. https://www.washingtonpost.com/news/innovations/wp/2017/07/21/how-a-fish-tank-helped-hack-a-casino/ https://www.washingtonpost.com/news/innovations/wp/2017/07/2...
- passwordoops 2y ago
  "I know why, because there is, probably, a regulation" Instead of assuming a regulation and writing a blog about it, do the research and find out. To quote the irreplaceable Benny Hill, "You mustn't assume, because it will make an ass out of you and me." Also, and more important, why default to regulation and not airline directors pushing ill-advised modernization strategies pushed by M$?
  - ses1984 2y ago
    I've done the research. It’s called first hand experience. I was the guy making the arguments, that controls we already have in place obviate the need for edr everywhere, but I was told it doesn’t matter, gotta check the box.
    - passwordoops 2y ago
      Awesome. Is the box there because of government regulation or someone in corporate deciding it's necessary?
      - betaby 2y ago
        Corporation and consultants mostly, judging from my experience. If asked about precise law or regulation they just wave hands.
      - ses1984 2y ago
        In my case it was a pci-dss (payment card industry data security standard) audit.
        moomin 2y ago
        The thing is, you read regulations, and they pretty much always tell you to do something, but it’s always heavily principle based. Companies are left with extraordinary leeway as to how these regulations are actually implemented.
        2y ago
- chrisjj 2y ago
  Because isolating the display and every machine to which it is necessarily connected obstructs monitoring and greatly increases the cost and delay of fixes if something should go wrong. Also I doubt any slimmed version of Windows is sufficiently malware proof without added EPS.
- hug 2y ago
  Low potential is not no potential, and most everyone is looking for swiss-cheese defense when it comes to these devices. In the case of a display at a check-in counter: - The display needs to be on a network, because it needs to collect information from elsewhere to display it. - It's on a network, so it needs to be kept updated, because a compromised host elsewhere on the same network will be able to compromise it, and anyway the display vendor won't support you if your product is nine versions behind current. - Since it needs updates for various components, it almost certainly needs some amount of outbound internet access, and it's also vulnerable to supply-chain attacks from those updates. - Since it is on a network, and has internet access, it needs to be running some kind of EDR or feed for a SIEM, because it is compromisable and the last thing you want is an unmonitored compromised host on your internal network talking back to C2. Anything that can be used for lateral movement will be used for lateral movement, and if we can get logs from it we want logs from it. A cross-platform EDR solution is perfect for these scenarios.
  - chrisjj 2y ago
    Agreed. Re: "- It's on a network, so it needs to be kept updated, because a compromised host elsewhere on the same network will be able to compromise it" the suggested solution was "an isolated network".0 The problem there is the operator would have to use SD cards to update the adverts... :)
sussmannbaka 2y ago
the more software you add to the display, including EDR, the higher the chance it will go down. you don’t reduce attack surface by introducing more surface and you don’t improve uptime by introducing additional single point of failures.
- chrisjj 2y ago
  > the more software you add to the display, including EDR, the higher the chance it will go down I am pretty sure that removing EDR from an internetted Windows airport display would ensure it goes down. And probably then come up with a ransomware demand.
  - sussmannbaka 2y ago
    Why would it do that? How would a ransomware demand come up on it? Honestly, please tell me because this sounds utterly outlandish to me.
    - chrisjj 2y ago
      It would do that because the internet is awash with malware.
      - sussmannbaka 2y ago
        Is the airport display downloading charlixcx.mp3.exe? Is it clicking malware attachments? I seriously want to know.

Retr0id 2y ago

A slightly bewildering fact is that CrowdStrike's terms and conditions say not to use it for critical infrastructure: > Neither the offerings nor crowdstrike tools are for use in the operation of aircraft navigation, nuclear facilities, communication systems, weapons systems, direct or indirect life-support systems, air traffic control, or any application or installation where failure could result in death, severe physical injury, or property damage. (Originally in all-caps, presumably to make it sound more legally binding) https://www.crowdstrike.com/terms-conditions/ https://www.crowdstrike.com/terms-conditions/

lbourdages 2y ago
That's a pretty standard clause I've seen in every software ToS I ever gazed my eyes upon. As far as I know, those applications weren't affected either - airports were affected because the ticketing systems were offline. Companies not using Crowdstrike were able to fly just fine. Edit: 911 was affected in many places, and that is definitely concerning as it is very much life critical.
- Retr0id 2y ago
  Sure, the small-print is always going to say "anything bad that happens is someone else's fault", but it's a bit embarrassing when you put it next to their marketing materials.
  - 01HNNWZ0MV43FF 2y ago
    I recently was asked to sign a thing saying it's my fault if I die while getting my hair colored
    - gibbitz 2y ago
      Oddly this is a thing: https://en.m.wikipedia.org/wiki/Vertebrobasilar_insufficiency https://en.m.wikipedia.org/wiki/Vertebrobasilar_insufficienc...

pylua 2y ago

The practice of building software would have to completely be flipped on its head to support professional liability. Agile would be the first casualty.

ClumsyPilot 2y ago
I think Boeing clearly tried to apply Agile to their planes
- synicalx 2y ago
  And their doors as well apparently, those are VERY agile.

JKCalhoun 2y ago

Sure, blame the engineer(s). And from now on every engineer now has the autonomy to refuse to use certain libraries and software stacks they are unfamiliar with, can refuse to submit a change, will have control over whether to push out software they worked on, etc. And a huge pay bonus as well since they have all this CEO-like risk/responsibility now.

mariusor 2y ago
Frankly, sign me up for that job, please.
jfim 2y ago
That level of agency exists today. Any software developer can tell their manager that no, they're not doing X. The risk is that they get written up for insubordination or have to look for new employment. What the author of the article doesn't know is that the structural engineer can also get fired if management deems the engineer too troublesome. The only difference is that the PE has a code of ethics and professional responsibility to uphold, and failing to do so would mean risking their license to practice engineering.
- betaby 2y ago
  In practice PE rarely held accountable in Quebec.
- strken 2y ago
  The most important difference is that in practice the SWE can't go and report the employer to anyone for breaching e.g. occupational health and safety laws, or any others, because those laws don't exist for the majority of software flaws. At best you have privacy laws like the GDPR or industry-specific ones covering e.g. medical devices...but CrowdStrike isn't meant to run in those environments anyway, see, look at this text buried in the license agreement.

deleted 2y ago

[deleted]

skwirl 2y ago

> If a software engineer tells you that this code needs to be 100% test covered, that AI won’t replace them, and that they need 3 months of development—then you better shut the fuck up and let them do their job. This is so naive that it is impossible to take the author seriously.

ses1984 2y ago
It’s called hyperbole.
- skwirl 2y ago
  No, this paragraph was not hyperbole. This was in conclusion and the author’s serious opinion.
  - Kiro 2y ago
    You're replying to the author but I agree. Too much unfounded speculation. I'm glad the author is being called out for lack of research in a sibling comment.
  - skwee357 2y ago
    It was a hyperbolic statement related to the fact that if you want developers to take full responsibility, like surgeons, then you need trust their authority. I haven’t heard of a surgeon who said “this operation will take as much time as needed”, and the hospital manager pressured him to “finish it in 8 hours, and not use too many syringes”.
  - ses1984 2y ago
    The whole post is a rant.
jkercher 2y ago
What's naive?
dakiol 2y ago
Who else is in a better position to give an estimate than the people who design and build the thing?

softwaredoug 2y ago

OK but structural engineering and anesthesiology are sciences with a lot of hard data behind them… … arguing for 100% test coverage or SOLID principles are more like philosophies and anecdotes without a lot of hard data supporting them. Software engineering looks less like engineering and more like a lot of bike shedding conversations.

ClumsyPilot 2y ago
We do have hard data as well, we have safe languages like ADA and Erlang, we have embedded engineers and special operating systems that are embedded with you applications. It’s just nobody wants to pay for these specialist skills and would rather slap something on top of windows and call it a day
skwee357 2y ago
Structural engineer can refuse to sign off a design that is not 100% fail proof. If he will be pressured into approving such design, he will ask the pressurer to put his own signature and bear the risk. Now try to pushback on your managers request to “cut this long deploy process just once because this big client wants it fast”.
pylua 2y ago
In software development, the edge cases are hardly every fully documented or understood. That is because software is so malleable that there are so many different input / output combinations. Other engineering professions are much more constrained on their input.

Aurornis 2y ago

Responsibility lies on large numbers of processes, teams, and people. I thought this blog might have some substance about proper postmortem investigations and how to evaluate and address the circumstances that led to a failure like this, but it has none of that. It’s just a very angry rant about CEOs and middle management. The premise is that engineers can’t bear any responsibility for their actions because they don’t get “respect” This has to be the 10th time I’ve seen arguments that “blame” is the right action in this case, but with the key exception that we’re only allowed to blame people other than the engineers. The last article was a lengthy rant about how it’s actually QA’s fault and engineers shouldn’t be expected to ensure their own code is correct, therefore engineers are blameless. This is empty calories for people who like ragebait, but nothing more.

handwarmers 2y ago
The blog responds to a very particular aspect of the fallout from the crowdstrike outage. The "Responsibility lies on large numbers of processes, teams, and people" was actually addressed in the article. It makes the case that executives claim that responsibility is correlated to pay. All the author asks in this case is for them to walk the walk. Go back to middle management.
mondrian 2y ago
> because they don’t get “respect” Nah, that's not what the blog post says. It says engineers aren't given sufficient responsibility, so that they can assume the respective blame, when something goes wrong. If an engineer says it would take 1 month to build a feature so it's sufficiently reliable, and the manager says "nah that's ridiculous, you have 1 week", then when the cobbled together feature breaks in production the manager should take the blame, since they effectively took the responsibility away from the engineer, upon themselves.
- 9dev 2y ago
  I used to think that way, until I had my own junior developers, and felt like that guy trying to bake a cake with three beagles: They veer off, constantly, in all directions. Developers deep in the trenches tend to have a bad feeling for business requirements or constraints; coupled with a knack for perfectionism and premature optimisation, that really often results in ridiculous time frames that are just plain unrealistic and would ruin the organisation long term. I don’t have any profound insights, though: The only sane mantra can be keeping things in balance. Too much management, you drive your devs insane; too much engineer control, and the architecture astronauts reinvent the wheel every other day.

jmclnx 2y ago

>and that they need 3 months of development then you better shut the f** up and let them do their job. This in a way covers the issue. Never in all my decades developing have had an estimate accepted. All the developers get is "It must be done by ...., no exceptions". So you end up pulling all nighters for weeks, and because you are tired, errors or bad decisions creep in. So yes, the issue is fully with upper management. I left a company because a big project was about to start and I could see it would be a big cluster**. People who stayed told me that is what happened.

batch12 2y ago

Sure, the dev is wrong, but so is the process that allowed their error to impact the product. If a single person can make this choice on accident, then a single person can make it on purpose either by being malicious themselves or being otherwise compromised. If the company has hung their entire security posture and operational success on the choices of one person, they have a problem. Especially a security company.

giantg2 2y ago

'But then the author engages in an absurd rant about how the entire software engineering industry is a “bit of a clusterfuck”,' I mean, it is a bit of a cluster fuck. Most of the issues I've seen are due to speed or cost. We don't have time for tests.we don't have time to record the business requirements in a collective place (just look through multiple JIRA stories and piece it together). We don't have money for dedicated QA roles. So of course problems happen. Luckily my team only works on a lower criticality site. None of this is really a dev's fault. It's leadership and the culture they incentivise.

acdha 2y ago
I blame the courts largely buying the argument that companies can disclaim all responsibility with terms of service. Most of these problems would become far less common if the situation was more balanced and managers had to consider the possibility of serious financial consequences for rushing something out without adequate testing.
- giantg2 2y ago
  I mostly agree, but that still won't fix this issue. Nothing has 100% uptime. Let's say Crowdstrike has an SLA of 99.99%. Even with this outage, they probably meet that depeding on the time frame. So which mangers bear the blame here? The ones who likely met their SLAs, or the ones who knew their systems relied on software that wasn't 100% reliable and didn't have a backup plan?
  - acdha 2y ago
    Pure SLAs matter less than the negligence: this showed critical lack of testing, following robust engineering practices for progressive deployments or fail safe design, etc. If a bank loses $1M they don’t get to argue that they shouldn’t compensate clients because they processed 99.9999% of their transactions correctly.
    - giantg2 2y ago
      In that example, the banks wouldn't compensate people if their payment system was down and couldn't process orders. That really what happened here - loss of sales on the clients' end. If it were securities, then they would have as-of processing to fix it. But the same argument about lack of testing can be made for the companies using the system. Do they have tests for what happens when pieces of their systems or infrastructure go down? Did they ever do a disaster recovery drill? Honestly, there's blame all around because almost nobody is doing it right. Even the ones who are have failures.

potatoman22 2y ago

As a developer, it's nice to shift the blame to other people, and other people do share some of the blame. But we also need to be responsible for our code functioning. It reads like the author is looking for anyone else to blame besides the devs.

hypeatei 2y ago

It's very surprising that critical infrastructure has been "infected" by the compliance checkbox obsession rather than just being designed in a way that completely eliminates the need for any shitty off-the-shelf security product. There is incompetence and complacency showing at every level after the CrowdStrike outage. Both the people selling CrowdStrike and the ones implementing it.

gizmo 2y ago

The author argues that the CEO of CrowdStrike has failed upwards. I don't agree. CrowdStrike makes compliance software. CrowdStrike's main purpose isn't to provide protection against cyber attacks. Businesses don't care enough about that. Businesses do care about simplifying their compliance burden and limiting their liability when they get hacked. That's where CrowdStrike excels, and that's why CrowdStrike can charge so much for their services. It's not easy to build a 70 billion dollar business and CrowdStrike serves a real business need. > I remember times when leaders had dignity and self-respect. They would go on stage and apologize Etiquette rules change over time, but a constant throughout history is that people in power don't take responsibility voluntarily. The "good old days" where leaders had dignity and self-respect never existed. > [...] delusional claim how software engineers should bear the responsibility for bugs and outages The /opinion/ that people /should/ be held responsible can't be delusional. Apparently the author believes that software engineers should bear zero responsibility -- even when their software kills people -- because we don't get enough respect. I don't agree with that opinion but it's a bit rich to call other people delusional when making one unfounded claim after another. The post as a whole is way too angry and too cynical for me.

josephg 2y ago

You know, I sort of agree with this article while completely disagreeing with it. The article says: > You want software engineers to be accountable for their code, then give them the respect they deserve. The problem is, respect is something that's taken as much as its something given. We can't even decide for ourselves if software is worthy of respect. Can anyone learn to code from a coding bootcamp, where getting a job is all that matters? Or is it a discipline that takes years to master, where mistakes can and will bring down the global economy? Are we glorified plumbers, or are we mathematicians and civil engineers combined? If you see yourself as a code monkey, of course you can't be "held responsible" for the results of your work. Coding bootcamps don't teach infosec. Its up to your company to set good practices and your job is just to follow them. Its only if you personally want to take your role in society seriously that it makes sense to consider not just your job, but the effect your job has on the wider world. I'm personally of the opinion that this mindset is almost always long term positive for your career. Its less "blame the dev for hitting deploy" and more "I'm the dev. No, I won't hit deploy on that code in the state its in." I didn't go to a coding bootcamp. I went to university. There they forced all of us engineering & CS students to do an ethics course - which was actually fantastic. There they taught us about the Therac-25: a computer controlled radiation machine which killed a bunch of people. The engineers on the ground knew that it needed more testing, but the company insisted it was fine and pushed it out the door. Here's the question: If you were one of those engineers, what would you do? If you knew, or suspected, that a bug in your code could bring down 911 services and hospitals, ground planes or give people a lethal dose of radiation, do you really trust your manager to make the engineering call? Do you think the CEO understands the risk that is being taken by hitting deploy? Of course, if we're playing the blame game, the blame ultimately the blame falls on the CEO of the company or something. But forget the blame game. You won't be fired. You won't lose your cushy job. The question is: Who do you want to be in situations like this? Think about this question now. You won't have time in the moment when your boss tells you to hit deploy, and you have second thoughts. Me? I want to be someone who would say no.

skwee357 2y ago
I agree. But in order to be the "person who would say no", the industry needs to understand that your opinion and expertise--matter. It could be a cultural shift, or a gate-keeping style shift where we protect the title "Engineer", like they do in some professions in some countries. But given the current state, you can't on one hand blame the developers, and on the other hand treat them like spoiled kids who make too much money and in any way AI can replace most of them. It doesn't work this way. A Structural Engineer bears the responsibility because he has the authority, and respect to his knowledge, to refuse to sign off a broken design. This is not the case in software engineering.

betaby 2y ago

There is no law requiring EDR. It's purely constructed nonsense copied by security clowns from one company to another. Now it's kind of mandatory. It has self regulated everything to the ground

dkarter 2y ago

Weak post. Agree with others that this is just rage bait lacking substance. Blaming the developers (or any specific individual/group for that matter) is a cop out, it’s easy and lazy and doesn’t get to the root of the problem, which is more often than not a lack of processes, tools, information and lack of time/desire from leadership to address “technical debt” (for lack of a better term), no matter how many times the devs bring that up. When you blame an individual or a group you can close the case shut on the post mortem and not get to any substantive improvements, meaning this can and will happen again, just to somebody else. That’s why blameless postmortem and a blameless culture is so important. This is a good article about that philosophy: > My summary of blameless culture is: when there is an outage, incident, or escaped bug in your service, assume the individuals involved had the best of intentions, and either they did not have the correct information to make a better decision, or the tools allowed them to make a mistake. https://www.gybe.ca/a-few-words-about-blameless-culture/ https://www.gybe.ca/a-few-words-about-blameless-culture/