7 ms·

To make this dream come true is far beyond the scope of just dev ops. It's also silly because if the problems are known this well the runbook will only be this

To make this dream come true is far beyond the scope of just dev ops. It's also silly because if the problems are known this well the runbook will only be this good for a moment while the devs are working on fixing the bugs and the runbook will need to be updated constantly after each release. This requires more testing than most are willing to do and for dev ops to still always be on call to find the new bugs first and document them this way while waiting for devs to fix them and for management to always reserve time to keep this updated. This means there's a long enough release cycle for this to ever happen which is also unlikely.

namaria 3y ago
In my experience, this is signaled by companies hiring "devops engineers". It shows lack of understanding that "devops" means devs and ops people should work together. And usually means they want someone to hold the bag because they can't seem to make their system work well, for some reason.
GauntletWizard 3y ago
I highly recommend against runbooks. There should be no need for runbooks. Anything that can be identified down to a single solution should have an automated response. What you should have is hintbooks. For each alert, have something that identifies why you made the alert. What dashboards to check, what logs to read, what else might be going wrong.
- ryanklee 3y ago
  Automated responses take time and tech to implement. Playbooks are a way to iterate into automation. Good automated solutions often don't emerge fully baked.
  - GauntletWizard 3y ago
    By the time you have enough clarity to create a "playbook", you probably have enough clarity to reproduce and to automate. That said, there are still a ton of solutions where "playbook" is kind of the right call. "DB is out of Disk Space" is probably "Increase the size of the disk reservation" for most modern cloud-native companies. They rarely rise to the level of detail that proponents of "Playbooks" suggest, though. How do you do that? Is it clicking through the UI? Do you have the CLI command prepared? Have you thought about why disk space usage has grown suddenly? This is the sort of thing that may be a no brainer you were expecting (in which case - Why didn't you complete the operation in-hours?) or it could be the "That's funny[1]" moment that precedes a major discovery. [1] https://quoteinvestigator.com/2015/03/02/eureka-funny/ https://quoteinvestigator.com/2015/03/02/eureka-funny/
WJW 3y ago
If the runbooks are this clear (and the monitoring this accurate), you don't really need an on call engineer at all tbh. Just have an automated system look at the error graph and roll back to the previous release if the errors go up too much. There's a bunch of blog posts out there of companies that already do this.
- midasuni 3y ago
  Im not sure why a new version was rolled out by a team who weren’t monitoring it for at least the next 5 minutes.
- pjc50 3y ago
  Indeed. If you have a detailed, unambiguous, exact process that can resolve issues without a human having to make a judgement call .. why are you giving it to a human to execute? The human's role in an ops situation is like that of the "safety driver" in self driving cars: do nothing most of the time, deal with emergencies, and take the blame for the failure of the system.
jasonlotito 3y ago
> To make this dream come true is far beyond the scope of just dev ops. Nah. It's not unreasonable. The oddest thing was the feature that would save state and logs in a document somewhere. Logs and dashboards already allow you to focus on certain times, so no need to store that state somewhere else. In fact, that would be worse as it would discourage people from digging further. > the runbook will need to be updated constantly after each release. Nah. Nothing in there was unique or special about that specific release outside of the failover. And even the failover is general enough to be a failover. Everything else is standardized to the point of being boring. > This means there's a long enough release cycle for this to ever happen which is also unlikely. Daily. Listen, nothing here was crazy. I've lived this. It's not hard. It's just boring conventions and consistency. Again, literally the strangest part was saving the state and logs in some extra document. The other weird thing was not checking the release logs (The release time was weird, too, but I'll ignore that). Rolling that release back would have been my first thought. And none of this is difficult. I realize not everyone is going to have the same experience, but it's not a dream, and thinking it's only a dream ignores how within-reach it really is.
- flerchin 3y ago
  To me the strangest thing was that she rolled back the change, but somehow not with cicd. Meaning it's going to blow up again on next deploy.
  - jasonlotito 3y ago
    Not necessarily.

namaria 3y ago

In my experience, this is signaled by companies hiring "devops engineers". It shows lack of understanding that "devops" means devs and ops people should work together. And usually means they want someone to hold the bag because they can't seem to make their system work well, for some reason.

GauntletWizard 3y ago

I highly recommend against runbooks. There should be no need for runbooks. Anything that can be identified down to a single solution should have an automated response. What you should have is hintbooks. For each alert, have something that identifies why you made the alert. What dashboards to check, what logs to read, what else might be going wrong.

ryanklee 3y ago
Automated responses take time and tech to implement. Playbooks are a way to iterate into automation. Good automated solutions often don't emerge fully baked.
- GauntletWizard 3y ago
  By the time you have enough clarity to create a "playbook", you probably have enough clarity to reproduce and to automate. That said, there are still a ton of solutions where "playbook" is kind of the right call. "DB is out of Disk Space" is probably "Increase the size of the disk reservation" for most modern cloud-native companies. They rarely rise to the level of detail that proponents of "Playbooks" suggest, though. How do you do that? Is it clicking through the UI? Do you have the CLI command prepared? Have you thought about why disk space usage has grown suddenly? This is the sort of thing that may be a no brainer you were expecting (in which case - Why didn't you complete the operation in-hours?) or it could be the "That's funny[1]" moment that precedes a major discovery. [1] https://quoteinvestigator.com/2015/03/02/eureka-funny/ https://quoteinvestigator.com/2015/03/02/eureka-funny/

WJW 3y ago

If the runbooks are this clear (and the monitoring this accurate), you don't really need an on call engineer at all tbh. Just have an automated system look at the error graph and roll back to the previous release if the errors go up too much. There's a bunch of blog posts out there of companies that already do this.

midasuni 3y ago
Im not sure why a new version was rolled out by a team who weren’t monitoring it for at least the next 5 minutes.
pjc50 3y ago
Indeed. If you have a detailed, unambiguous, exact process that can resolve issues without a human having to make a judgement call .. why are you giving it to a human to execute? The human's role in an ops situation is like that of the "safety driver" in self driving cars: do nothing most of the time, deal with emergencies, and take the blame for the failure of the system.

jasonlotito 3y ago

> To make this dream come true is far beyond the scope of just dev ops. Nah. It's not unreasonable. The oddest thing was the feature that would save state and logs in a document somewhere. Logs and dashboards already allow you to focus on certain times, so no need to store that state somewhere else. In fact, that would be worse as it would discourage people from digging further. > the runbook will need to be updated constantly after each release. Nah. Nothing in there was unique or special about that specific release outside of the failover. And even the failover is general enough to be a failover. Everything else is standardized to the point of being boring. > This means there's a long enough release cycle for this to ever happen which is also unlikely. Daily. Listen, nothing here was crazy. I've lived this. It's not hard. It's just boring conventions and consistency. Again, literally the strangest part was saving the state and logs in some extra document. The other weird thing was not checking the release logs (The release time was weird, too, but I'll ignore that). Rolling that release back would have been my first thought. And none of this is difficult. I realize not everyone is going to have the same experience, but it's not a dream, and thinking it's only a dream ignores how within-reach it really is.

flerchin 3y ago
To me the strangest thing was that she rolled back the change, but somehow not with cicd. Meaning it's going to blow up again on next deploy.
- jasonlotito 3y ago
  Not necessarily.