4 ms·

But who plans a New Year's Eve datacenter migration? What kind of major site maintenance is needed on Christmas? It's one thing if it's an unplanned emergency

But who plans a New Year's Eve datacenter migration? What kind of major site maintenance is needed on Christmas? It's one thing if it's an unplanned emergency. It's another entirely if it's planned. Planning for work on Christmas or New Year's Eve would be enough to convince me to find employment elsewhere.

tdurden 13y ago
exactly.
sokoloff 13y ago
Well, we planned a NYE DC move (5 years ago I think), and years before that we didn't have the capability to do certain DB tasks with the site on-line, so we also planned the 2300 on 12/25 work. (And back in 2004, we couldn't even do releases with the site on-line, so we did regularly scheduled releases on Mondays starting at 1 AM with the outage starting at 2 AM...) How does this happen? Suppose we find ourselves running a high-margin e-commerce business and needing to take downtime sometime to accomplish some maintenance work. When should we schedule it? Managers and directors of teams in my Ops org researched the expected bookings during various windows and suggested those times. They recommended them to me. I spent some of the planning effort double-checking and making sure the teams would be signed up for that, and got a consistent message of "This is what Ops does; we all knew what we were signing up for when we took this job." (Side note: Of the 9 people on that DC migration trip, 8 still work here, 5+ years later.) On the 11PM on 12/25 outage, we were upgrading OS on a few key infrastructure servers that were singletons. (We relied on DB and file servers in a way that didn't give us the ability to do maintenance and serve traffic at the same time.) On the DC move, we were literally decommissioning our main DC and moving those services to a new DC in the same town. We bought and brought online as much duplicate gear as we could, but the core transactional systems were a singleton and needed to be moved, not replicated, and that was going to be 4-8 hours of downtime. My teams chose to put that downtime into a time when we'd lose $1MM of high-margin bookings rather than $3MM of high-margin bookings. Our CEO, without me asking, was also keeping tabs on the migration (from his house) and sent IMs and emails congratulating and thanking the team for our work when we wrapped things up around 4 AM on 1/1. So, I got to see both sides of it: doing what I believed was right as a leader and being present and engaged when they were doing a less pleasant part of their job, and I got to feel what it was like to get clear engagement and genuine thanks from above, even though it was technically "just" a symbolic gesture. He could have sent that mail at 10 AM the next day, but knowing him, he was keeping tabs on the work not to check up on us or because he was petrified of a multi-day outage, but because he wanted to be able to be the first one to thank us. Side note: these experiences allowed me to better understand what Ops goes through, and have caused us to prioritize (and since complete) architectural projects that permit us to do almost every bit of periodic maintenance with the site online. We haven't taken planned downtime in several years now, because we've made those improvements. From a pure dollars-and-sense standpoint, those projects "shouldn't win" over larger business opportunities that we have, but they did win, in significant part because they were the right thing to do for our people, and having been there, I can argue for them from a position of first-hand knowledge. There is an awful lot of negativity and anti-company sentiment whenever inevitable situations come up where, on an instantaneous basis, either the employee can get what's best for them XOR the company can get what's best for it. I don't personally see these situations to be quite the same as Walmart opening at 5 PM on Thanksgiving Day, but I'm open to the idea that others see them as being exactly the same thing. I understand and respect the perspective of someone who would never do planned work on a holiday, or even those who would never do planned work on a weekend. It's a perfectly reasonable point of view, and I will happily employ those people in roles that don't require off-hours work. Those people will presumably not choose a career in operations, or at least they probably shouldn't. It is always the case that I speak for myself and not my employer, but it's worth calling that out on this post in particular.
- mratzloff 13y ago
  Thank you for the detailed response. It's a good outcome that the sacrifices of systems teams are more widely recognized in your company now; at many companies they are not. Online maintenance is how we do all our maintenance. Getting to that point as early as possible is essential for redundancy and failover, anyway, so it is both good for the company and good for the systems team.
  - sokoloff 13y ago
    Online maintenance is huge, and it's somewhat easier to design for now than it was 10 years ago. It's very nice to be able to schedule releases at a human/humane time and there's really only one point in our release process now where it's impossible to very obviously and fairly automatically revert and regroup. That one point is clearly called out, and the decision is easy to make when the site has been operating on live traffic for 60 minutes with zero perceptible technical and business metrics degradation. I'd not consider online maint capability to be a necessary part of an MVP, since any time you spend on it is unlikely to help you understand whether you even HAVE a business, but I'd prioritize it pretty highly in the very next phase of company development.

tdurden 13y ago

exactly.

sokoloff 13y ago

Well, we planned a NYE DC move (5 years ago I think), and years before that we didn't have the capability to do certain DB tasks with the site on-line, so we also planned the 2300 on 12/25 work. (And back in 2004, we couldn't even do releases with the site on-line, so we did regularly scheduled releases on Mondays starting at 1 AM with the outage starting at 2 AM...) How does this happen? Suppose we find ourselves running a high-margin e-commerce business and needing to take downtime sometime to accomplish some maintenance work. When should we schedule it? Managers and directors of teams in my Ops org researched the expected bookings during various windows and suggested those times. They recommended them to me. I spent some of the planning effort double-checking and making sure the teams would be signed up for that, and got a consistent message of "This is what Ops does; we all knew what we were signing up for when we took this job." (Side note: Of the 9 people on that DC migration trip, 8 still work here, 5+ years later.) On the 11PM on 12/25 outage, we were upgrading OS on a few key infrastructure servers that were singletons. (We relied on DB and file servers in a way that didn't give us the ability to do maintenance and serve traffic at the same time.) On the DC move, we were literally decommissioning our main DC and moving those services to a new DC in the same town. We bought and brought online as much duplicate gear as we could, but the core transactional systems were a singleton and needed to be moved, not replicated, and that was going to be 4-8 hours of downtime. My teams chose to put that downtime into a time when we'd lose $1MM of high-margin bookings rather than $3MM of high-margin bookings. Our CEO, without me asking, was also keeping tabs on the migration (from his house) and sent IMs and emails congratulating and thanking the team for our work when we wrapped things up around 4 AM on 1/1. So, I got to see both sides of it: doing what I believed was right as a leader and being present and engaged when they were doing a less pleasant part of their job, and I got to feel what it was like to get clear engagement and genuine thanks from above, even though it was technically "just" a symbolic gesture. He could have sent that mail at 10 AM the next day, but knowing him, he was keeping tabs on the work not to check up on us or because he was petrified of a multi-day outage, but because he wanted to be able to be the first one to thank us. Side note: these experiences allowed me to better understand what Ops goes through, and have caused us to prioritize (and since complete) architectural projects that permit us to do almost every bit of periodic maintenance with the site online. We haven't taken planned downtime in several years now, because we've made those improvements. From a pure dollars-and-sense standpoint, those projects "shouldn't win" over larger business opportunities that we have, but they did win, in significant part because they were the right thing to do for our people, and having been there, I can argue for them from a position of first-hand knowledge. There is an awful lot of negativity and anti-company sentiment whenever inevitable situations come up where, on an instantaneous basis, either the employee can get what's best for them XOR the company can get what's best for it. I don't personally see these situations to be quite the same as Walmart opening at 5 PM on Thanksgiving Day, but I'm open to the idea that others see them as being exactly the same thing. I understand and respect the perspective of someone who would never do planned work on a holiday, or even those who would never do planned work on a weekend. It's a perfectly reasonable point of view, and I will happily employ those people in roles that don't require off-hours work. Those people will presumably not choose a career in operations, or at least they probably shouldn't. It is always the case that I speak for myself and not my employer, but it's worth calling that out on this post in particular.

mratzloff 13y ago
Thank you for the detailed response. It's a good outcome that the sacrifices of systems teams are more widely recognized in your company now; at many companies they are not. Online maintenance is how we do all our maintenance. Getting to that point as early as possible is essential for redundancy and failover, anyway, so it is both good for the company and good for the systems team.
- sokoloff 13y ago
  Online maintenance is huge, and it's somewhat easier to design for now than it was 10 years ago. It's very nice to be able to schedule releases at a human/humane time and there's really only one point in our release process now where it's impossible to very obviously and fairly automatically revert and regroup. That one point is clearly called out, and the decision is easy to make when the site has been operating on live traffic for 60 minutes with zero perceptible technical and business metrics degradation. I'd not consider online maint capability to be a necessary part of an MVP, since any time you spend on it is unlikely to help you understand whether you even HAVE a business, but I'd prioritize it pretty highly in the very next phase of company development.