3 ms·

Since decisions are often made under the pressure of business goals from a technical standpoint they are almost always the wrong decision. In instances like tha

Since decisions are often made under the pressure of business goals from a technical standpoint they are almost always the wrong decision. In instances like that it takes a certain amount of experience and backbone to either say no to horrible hacks or put in the right kind of mitigation and fixes as soon as the business goals have been met. If it intuitively feels wrong then I don't really care what justifications people had at the time for writing a horrible mess. Objectively it is a horrible mess still even if I forgive the people that made the decisions that led to that mess. Since I wrote that post I've seen the same mistakes repeated over and over again. My tools of choice at the current time are terraform, packer, some kind of secure storage mechanism like hashicorp vault, and plain old shell scripts or rake files. I'm also not advocating checking in secret tokens into the code base but you need something that gives you access to the place that holds the secure tokens. It is important to draw that distinction since the extra layer of indirection actually provides you enough control to revoke and recycle tokens much more quickly than if you had checked everything into the code directly.

brazzledazzle 10y ago
I agree, it does take experience and backbone and even then it may not be enough. It's basically technical debt and unfortunately that's still something many organizations struggle with today. While I think you're right about many things being objectively bad from a technical perspective there are a lot of decisions that aren't just technical. You may end up using the lesser of two options of the lesser one is more compatible with the expertise present in your organization. Sometimes it's not clear cut at all and only time and real world usage reveals the pitfalls. It's easy to point at that and say "that's stupid and this is terrible" a few months or years down the line. The hard part is trying to make those decisions. Let's look at Puppet as an example. Let's say you have an organization that has a couple people out of a dozen that can write shell scripts that aren't spaghetti. But if everything depends on them they become a bottleneck. You know there's a bunch of halfway decent modules written out there for a lot of stuff you need and your two people can close the gap. The rest of the team can build knowledge and skill over time but can start putting modules together to build infrastructure today. So you can choose shell scripts and fixing your personnel/skill issues or you can choose Puppet and deal with the warts. Training or hiring new people would be obvious but given the reality of budget and politics at $bigcorp it's a no-go or you're looking at years of effort. Even if it's cut and dry technically it's really not always a simple decision.
- dkarapetyan 10y ago
  I have seen this argument > Let's look at Puppet as an example. Let's say you have an organization that has a couple people out of a dozen that can write shell scripts that aren't spaghetti. But if everything depends on them they become a bottleneck. You know there's a bunch of halfway decent modules written out there for a lot of stuff you need and your two people can close the gap. The rest of the team can build knowledge and skill over time but can start putting modules together to build infrastructure today. So you can choose shell scripts and fixing your personnel/skill issues or you can choose Puppet and deal with the warts. Training or hiring new people would be obvious but given the reality of budget and politics at $bigcorp it's a no-go or you're looking at years of effort. Even if it's cut and dry technically it's really not always a simple decision. at least 5 different times and each time whoever argued that point ended up being wrong 3 months down the line. What I don't get is why it keeps being propagated. You should indeed take those 2 people, make them a bottleneck, and let them build the organizational tools and expertise around what they consider is correct instead of doing what you suggested. I think this argument is propagated because it provides the illusion that programmers are interchangeable. If somehow you can leverage all the Puppet knowledge in the world you will reduce your risk and dependence on single points of failure like one programmer that understands how the tower of shell scripts fit together. I think this is based on a false premise. Programmers are not interchangeable and no matter how many frameworks and tools you layer on it will not be possible to get rid of the dependence of good programmers doing what is right from a technical standpoint. The alternative just doesn't make sense really. You can't take a bunch of idiots and make them brilliant through tooling.

brazzledazzle 10y ago

I agree, it does take experience and backbone and even then it may not be enough. It's basically technical debt and unfortunately that's still something many organizations struggle with today. While I think you're right about many things being objectively bad from a technical perspective there are a lot of decisions that aren't just technical. You may end up using the lesser of two options of the lesser one is more compatible with the expertise present in your organization. Sometimes it's not clear cut at all and only time and real world usage reveals the pitfalls. It's easy to point at that and say "that's stupid and this is terrible" a few months or years down the line. The hard part is trying to make those decisions. Let's look at Puppet as an example. Let's say you have an organization that has a couple people out of a dozen that can write shell scripts that aren't spaghetti. But if everything depends on them they become a bottleneck. You know there's a bunch of halfway decent modules written out there for a lot of stuff you need and your two people can close the gap. The rest of the team can build knowledge and skill over time but can start putting modules together to build infrastructure today. So you can choose shell scripts and fixing your personnel/skill issues or you can choose Puppet and deal with the warts. Training or hiring new people would be obvious but given the reality of budget and politics at $bigcorp it's a no-go or you're looking at years of effort. Even if it's cut and dry technically it's really not always a simple decision.

dkarapetyan 10y ago
I have seen this argument > Let's look at Puppet as an example. Let's say you have an organization that has a couple people out of a dozen that can write shell scripts that aren't spaghetti. But if everything depends on them they become a bottleneck. You know there's a bunch of halfway decent modules written out there for a lot of stuff you need and your two people can close the gap. The rest of the team can build knowledge and skill over time but can start putting modules together to build infrastructure today. So you can choose shell scripts and fixing your personnel/skill issues or you can choose Puppet and deal with the warts. Training or hiring new people would be obvious but given the reality of budget and politics at $bigcorp it's a no-go or you're looking at years of effort. Even if it's cut and dry technically it's really not always a simple decision. at least 5 different times and each time whoever argued that point ended up being wrong 3 months down the line. What I don't get is why it keeps being propagated. You should indeed take those 2 people, make them a bottleneck, and let them build the organizational tools and expertise around what they consider is correct instead of doing what you suggested. I think this argument is propagated because it provides the illusion that programmers are interchangeable. If somehow you can leverage all the Puppet knowledge in the world you will reduce your risk and dependence on single points of failure like one programmer that understands how the tower of shell scripts fit together. I think this is based on a false premise. Programmers are not interchangeable and no matter how many frameworks and tools you layer on it will not be possible to get rid of the dependence of good programmers doing what is right from a technical standpoint. The alternative just doesn't make sense really. You can't take a bunch of idiots and make them brilliant through tooling.