5 ms·

If you had hired competent people to begin with, the configs would have been backed up. I don't agree with the other posters recommendation, but I understand t

If you had hired competent people to begin with, the configs would have been backed up. I don't agree with the other posters recommendation, but I understand the sentiment. The same incompetence that allowed the event to happen is the same incompetence that allowed it to be swept under the rug for almost 6 years. I've seen paragons of hiring get upset because failures were made visible, not understanding that the failures were there all along and they were playing a game of chicken with reality, only now they have a chance of winning.

deleted 7y ago
[deleted]
setr 7y ago
>If you had hired competent people to begin with, the configs would have been backed up What happy-go-lucky universe do you hail from? You could hire the most competent people in the world and you'll still end up with a horribly complex shitshow with lost programs, untested backups, missing docs, implicit information out the ass, and so on. Software development has never had the discipline to claim compotent employees produce competent results -- they're just less likely to be shitty results. And we spend 90÷ of our time dealing with every layer of incompetence our competent, and incompetent, friends have provided us
- celticmusic 7y ago
  It's possible to avoid losing a configuration.
  - setr 7y ago
    It's possible, and it's possible to avoid losing source code, and its.possible to have valid backups, and keep your servers decemtly secure... yet it's sufficiently common (especially as the program ages) that its not trivially attributed to competency; rather you need continuous, unfailing competency, in the face of continuous environmental change (local changes, within the business, and external changes, as business needs change, and mergers occur, and people quit/fired, etc)
    - celticmusic 7y ago
      > rather you need continuous, unfailing competency That's called doing it manually. What you're doing is making excuses rather than holding yourself to a higher level of quality.
      - setr 7y ago
        >That's called doing it manually. Yes, a common feature in unstable environments. Automation is an ideal, but not a necessity, and it's not always viable or cost-effective. But also that the unfailing competency means that you need to contiously maintain (all aspects of the) automation without fail, again in spite of a changing environment. Across everyone who does things. >What you're doing is making excuses rather than holding yourself to a higher level of quality. And what you're doing is pretending that firing off blind, trivial prescriptions is a useful activity -- perhaps I'll soon find you selling self-help books as well :-) Automation is not an answer that can always be applied trivially; especially retroactively. Even despite all the marketing that will tell you otherwise.
        celticmusic 7y ago
        > Automation is an ideal, but not a necessity You're just bad. You automate specifically to take the human error risk out of the equation, that's a part of what it means to be competent. We don't automate our deploys because it's an ideal that is not a necessity, but so some jackass doesn't delete a configuration while also never putting it into a backup, or source control, or on their dev machine, ad nauseum. It's not an accident that some of us have been working 20+ years in this industry and have never been the cause of something like this. > Automation is not an answer that can always be applied trivially; especially retroactively. Even despite all the marketing that will tell you otherwise. shit, I guess I didn't read the right pamphlet...
    - workingpatrick 7y ago
      It's more than possible, it's expected. While all of these things do happen, they are preventable and should not be treated as excusable, people need to be held accountable and systems / processes need to be put in place to ensure that these things don't reoccur.

deleted 7y ago

[deleted]

setr 7y ago

>If you had hired competent people to begin with, the configs would have been backed up What happy-go-lucky universe do you hail from? You could hire the most competent people in the world and you'll still end up with a horribly complex shitshow with lost programs, untested backups, missing docs, implicit information out the ass, and so on. Software development has never had the discipline to claim compotent employees produce competent results -- they're just less likely to be shitty results. And we spend 90÷ of our time dealing with every layer of incompetence our competent, and incompetent, friends have provided us

celticmusic 7y ago
It's possible to avoid losing a configuration.
- setr 7y ago
  It's possible, and it's possible to avoid losing source code, and its.possible to have valid backups, and keep your servers decemtly secure... yet it's sufficiently common (especially as the program ages) that its not trivially attributed to competency; rather you need continuous, unfailing competency, in the face of continuous environmental change (local changes, within the business, and external changes, as business needs change, and mergers occur, and people quit/fired, etc)
  - celticmusic 7y ago
    > rather you need continuous, unfailing competency That's called doing it manually. What you're doing is making excuses rather than holding yourself to a higher level of quality.
    - setr 7y ago
      >That's called doing it manually. Yes, a common feature in unstable environments. Automation is an ideal, but not a necessity, and it's not always viable or cost-effective. But also that the unfailing competency means that you need to contiously maintain (all aspects of the) automation without fail, again in spite of a changing environment. Across everyone who does things. >What you're doing is making excuses rather than holding yourself to a higher level of quality. And what you're doing is pretending that firing off blind, trivial prescriptions is a useful activity -- perhaps I'll soon find you selling self-help books as well :-) Automation is not an answer that can always be applied trivially; especially retroactively. Even despite all the marketing that will tell you otherwise.
      - celticmusic 7y ago
        > Automation is an ideal, but not a necessity You're just bad. You automate specifically to take the human error risk out of the equation, that's a part of what it means to be competent. We don't automate our deploys because it's an ideal that is not a necessity, but so some jackass doesn't delete a configuration while also never putting it into a backup, or source control, or on their dev machine, ad nauseum. It's not an accident that some of us have been working 20+ years in this industry and have never been the cause of something like this. > Automation is not an answer that can always be applied trivially; especially retroactively. Even despite all the marketing that will tell you otherwise. shit, I guess I didn't read the right pamphlet...
  - workingpatrick 7y ago
    It's more than possible, it's expected. While all of these things do happen, they are preventable and should not be treated as excusable, people need to be held accountable and systems / processes need to be put in place to ensure that these things don't reoccur.