4 ms·

First, let me say I totally agree with the premise here--that all parts of a software engineering organization need to be on board with DevOps, buying in to the

First, let me say I totally agree with the premise here--that all parts of a software engineering organization need to be on board with DevOps, buying in to the vision, learning the full stack, taking responsibility. However, this article does serve as a good example of one of the big problems with most DevOps/SRE evangelism: It's usually coming from the Dev- perspective, rather than the -Ops perspective (or better, a balanced perspective). In particular, this focuses mostly on getting code to users, and speeding up that process. While that's certainly important, it leaves out any discussion of: * security * infrastructure design * instrumentation and monitoring * planning for failure * security * performance evaluation and tuning * disaster recovery planning * security * off-hours on-call responsibilities * security Successful DevOps cultures must address these things just as much as they address code feedback loops, time-to-production, rollback strategies, code pipeline automation, etc.

beobab 9y ago
Speaking as someone who once had to patch up a legacy system, I appreciate items 1, 5, 8 and 10 far more than I did before.
- ben_jones 9y ago
  Number 11 is championing 1, 5, 8, and 10, to management and other developers. Number 12 is stealing developer time for 1, 5, 8, and 10.
dimitar 9y ago
My experience is that even customers who are very enthusiastic about these new processes and methodologies quickly turn back to the old-school way of doing things once someone in upper management raises these bullets. And they weren't considered before because the customer Ops people were not invited to any discussions. So in order to truly get the whole organization on board you do need to include your solutions to the above points in your evangelism.. and start really early.
David 9y ago
Agreed, it seems like a lot of people think about devops as a way to make dev's lives better without actively harming ops. The article does have a section on getting Ops on board with devops, but it's much weaker than the previous section on getting Dev on board. Devops done right should really improve both sides, as it explicitly recognizes the tradeoffs involved with the added automation / processes / policies. I've been reading Google's Site Reliability Engineering book ([1], free online, highly recommend so far) and it lays out pretty clearly how to do devops in a more balanced way. Key points are: - Define service level indicators (SLIs), metrics you can track to see how you're doing on the things your users care about: uptime, latency, security, etc. For instance, measure uptime as successful responses / total requests. - Define service level objectives (SLOs) based on your SLIs, e.g. 99.9% successful responses / total requests (aka 99.9% uptime) - Establish an error budget which controls your deployments. Based on your SLOs you can accept a certain number of errors (failed requests, outages, slow response times, etc) for a given period. If you spend your entire error budget, no new code can be deployed. The deployment shutoff has to be agreed upon in advance and it has to be a hard and fast rule, so it's not ops' fault that dev can't deploy code. - If you reach your error budget, devs need to work on things to reduce error rate going forward instead of developing new features. It can be tests, monitoring and alerting, rollback process, or pretty much anything, but devs should know that they're doing it so they can deploy more code faster in the future. I think this makes a lot of sense and helps to align dev and ops on uptime goals. Specifically, dev's ability to push code is now directly tied to service level, and ops is not shooting for 100% reliability. Dev can choose how to spend the error budget, so ops isn't dictating how they do things; and ops knows and expects a certain number of problems, so they don't resent dev every time they push buggy code. [1] https://landing.google.com/sre/book.html https://landing.google.com/sre/book.html
CaptSpify 9y ago
A company I worked for started a devops team. 3 years later and there's still not a single ops person on it. There were so many security holes in their group we had to ban them from releasing to our servers, so now they have their own cloud setup that is bleeding money. The frustrating thing is: I'd have loved to work closer with them. But every time we tried, the response was "You have no place here, your slowing the process down". We just wanted to automate the security testing, they decided that throwing the security tests out altogether was a better idea. I do admit that it was indeed faster....
- lowbloodsugar 9y ago
  Just because someone has an Ops title doesn't mean they know anything about security. I've worked places where the Ops team caused every security incident. Its a business leadership issue. When teams accept requests from the business and prioritize those requests over security then we have problems. Ops teams can be just as guilty of this as any other team.
nunez 9y ago
That's interesting because most of the content I've seen has been on the infra side of things (deploying AWS, config management, security, etc) but, yes, you need both sides of the coin for the devops idea to happen

beobab 9y ago

Speaking as someone who once had to patch up a legacy system, I appreciate items 1, 5, 8 and 10 far more than I did before.

ben_jones 9y ago
Number 11 is championing 1, 5, 8, and 10, to management and other developers. Number 12 is stealing developer time for 1, 5, 8, and 10.

dimitar 9y ago

My experience is that even customers who are very enthusiastic about these new processes and methodologies quickly turn back to the old-school way of doing things once someone in upper management raises these bullets. And they weren't considered before because the customer Ops people were not invited to any discussions. So in order to truly get the whole organization on board you do need to include your solutions to the above points in your evangelism.. and start really early.

David 9y ago

Agreed, it seems like a lot of people think about devops as a way to make dev's lives better without actively harming ops. The article does have a section on getting Ops on board with devops, but it's much weaker than the previous section on getting Dev on board. Devops done right should really improve both sides, as it explicitly recognizes the tradeoffs involved with the added automation / processes / policies. I've been reading Google's Site Reliability Engineering book ([1], free online, highly recommend so far) and it lays out pretty clearly how to do devops in a more balanced way. Key points are: - Define service level indicators (SLIs), metrics you can track to see how you're doing on the things your users care about: uptime, latency, security, etc. For instance, measure uptime as successful responses / total requests. - Define service level objectives (SLOs) based on your SLIs, e.g. 99.9% successful responses / total requests (aka 99.9% uptime) - Establish an error budget which controls your deployments. Based on your SLOs you can accept a certain number of errors (failed requests, outages, slow response times, etc) for a given period. If you spend your entire error budget, no new code can be deployed. The deployment shutoff has to be agreed upon in advance and it has to be a hard and fast rule, so it's not ops' fault that dev can't deploy code. - If you reach your error budget, devs need to work on things to reduce error rate going forward instead of developing new features. It can be tests, monitoring and alerting, rollback process, or pretty much anything, but devs should know that they're doing it so they can deploy more code faster in the future. I think this makes a lot of sense and helps to align dev and ops on uptime goals. Specifically, dev's ability to push code is now directly tied to service level, and ops is not shooting for 100% reliability. Dev can choose how to spend the error budget, so ops isn't dictating how they do things; and ops knows and expects a certain number of problems, so they don't resent dev every time they push buggy code. [1] https://landing.google.com/sre/book.html https://landing.google.com/sre/book.html

CaptSpify 9y ago

A company I worked for started a devops team. 3 years later and there's still not a single ops person on it. There were so many security holes in their group we had to ban them from releasing to our servers, so now they have their own cloud setup that is bleeding money. The frustrating thing is: I'd have loved to work closer with them. But every time we tried, the response was "You have no place here, your slowing the process down". We just wanted to automate the security testing, they decided that throwing the security tests out altogether was a better idea. I do admit that it was indeed faster....

lowbloodsugar 9y ago
Just because someone has an Ops title doesn't mean they know anything about security. I've worked places where the Ops team caused every security incident. Its a business leadership issue. When teams accept requests from the business and prioritize those requests over security then we have problems. Ops teams can be just as guilty of this as any other team.

nunez 9y ago

That's interesting because most of the content I've seen has been on the infra side of things (deploying AWS, config management, security, etc) but, yes, you need both sides of the coin for the devops idea to happen