4 ms·

> When we started pulling consistent coding weekends, we brought in the entire management team to serve the engineers: We brought them food, washed their cars,

by tdurden 13y ago

> When we started pulling consistent coding weekends, we brought in the entire management team to serve the engineers: We brought them food, washed their cars, got oil changes, took in their dry cleaning, and arranged for childcare for their kids in the office. Sorry, but while admirable, this is not normal. I may differ from most of HN, but I simply don't see the point in making these kinds of sacrifices as an engineer just because I get free dinner or childcare. It isn't healthy...or worth it.

sokoloff 13y ago
That's not the point. The management team is showing solidarity with their team by serving them in the only (more symbolic than economically valuable) ways that they can. It's a symbolic (but real) sharing of the sacrifice, not a straight up bribe. Same reason I've come in on Christmas night (we scheduled the work after the families are reasonably in bed, but the calendar still said "12/25") to spend time with my teams doing major site maintenance activities, or why I travelled with the team for a major as-the-ball-dropped New Year's Eve datacenter migration. It's not because I'm especially good at carrying servers or building SAN racks... The engineers make the sacrifice because they believe it's necessary for the product/company to succeed. They feel better about it because their executives are physically present there with them rather than on a golf course or diving into their pools of gold coins...
- mratzloff 13y ago
  But who plans a New Year's Eve datacenter migration? What kind of major site maintenance is needed on Christmas? It's one thing if it's an unplanned emergency. It's another entirely if it's planned. Planning for work on Christmas or New Year's Eve would be enough to convince me to find employment elsewhere.
  - tdurden 13y ago
    exactly.
  - sokoloff 13y ago
    Well, we planned a NYE DC move (5 years ago I think), and years before that we didn't have the capability to do certain DB tasks with the site on-line, so we also planned the 2300 on 12/25 work. (And back in 2004, we couldn't even do releases with the site on-line, so we did regularly scheduled releases on Mondays starting at 1 AM with the outage starting at 2 AM...) How does this happen? Suppose we find ourselves running a high-margin e-commerce business and needing to take downtime sometime to accomplish some maintenance work. When should we schedule it? Managers and directors of teams in my Ops org researched the expected bookings during various windows and suggested those times. They recommended them to me. I spent some of the planning effort double-checking and making sure the teams would be signed up for that, and got a consistent message of "This is what Ops does; we all knew what we were signing up for when we took this job." (Side note: Of the 9 people on that DC migration trip, 8 still work here, 5+ years later.) On the 11PM on 12/25 outage, we were upgrading OS on a few key infrastructure servers that were singletons. (We relied on DB and file servers in a way that didn't give us the ability to do maintenance and serve traffic at the same time.) On the DC move, we were literally decommissioning our main DC and moving those services to a new DC in the same town. We bought and brought online as much duplicate gear as we could, but the core transactional systems were a singleton and needed to be moved, not replicated, and that was going to be 4-8 hours of downtime. My teams chose to put that downtime into a time when we'd lose $1MM of high-margin bookings rather than $3MM of high-margin bookings. Our CEO, without me asking, was also keeping tabs on the migration (from his house) and sent IMs and emails congratulating and thanking the team for our work when we wrapped things up around 4 AM on 1/1. So, I got to see both sides of it: doing what I believed was right as a leader and being present and engaged when they were doing a less pleasant part of their job, and I got to feel what it was like to get clear engagement and genuine thanks from above, even though it was technically "just" a symbolic gesture. He could have sent that mail at 10 AM the next day, but knowing him, he was keeping tabs on the work not to check up on us or because he was petrified of a multi-day outage, but because he wanted to be able to be the first one to thank us. Side note: these experiences allowed me to better understand what Ops goes through, and have caused us to prioritize (and since complete) architectural projects that permit us to do almost every bit of periodic maintenance with the site online. We haven't taken planned downtime in several years now, because we've made those improvements. From a pure dollars-and-sense standpoint, those projects "shouldn't win" over larger business opportunities that we have, but they did win, in significant part because they were the right thing to do for our people, and having been there, I can argue for them from a position of first-hand knowledge. There is an awful lot of negativity and anti-company sentiment whenever inevitable situations come up where, on an instantaneous basis, either the employee can get what's best for them XOR the company can get what's best for it. I don't personally see these situations to be quite the same as Walmart opening at 5 PM on Thanksgiving Day, but I'm open to the idea that others see them as being exactly the same thing. I understand and respect the perspective of someone who would never do planned work on a holiday, or even those who would never do planned work on a weekend. It's a perfectly reasonable point of view, and I will happily employ those people in roles that don't require off-hours work. Those people will presumably not choose a career in operations, or at least they probably shouldn't. It is always the case that I speak for myself and not my employer, but it's worth calling that out on this post in particular.
- tdurden 13y ago
  > The engineers make the sacrifice because they believe it's necessary for the product/company to succeed. They feel better about it because their executives are physically present there with them Really? Your engineers are so easily swayed by having a couple execs present at the office on Christmas eve that they happily ignore their own lives?

by tdurden 13y ago

sokoloff 13y ago

That's not the point. The management team is showing solidarity with their team by serving them in the only (more symbolic than economically valuable) ways that they can. It's a symbolic (but real) sharing of the sacrifice, not a straight up bribe. Same reason I've come in on Christmas night (we scheduled the work after the families are reasonably in bed, but the calendar still said "12/25") to spend time with my teams doing major site maintenance activities, or why I travelled with the team for a major as-the-ball-dropped New Year's Eve datacenter migration. It's not because I'm especially good at carrying servers or building SAN racks... The engineers make the sacrifice because they believe it's necessary for the product/company to succeed. They feel better about it because their executives are physically present there with them rather than on a golf course or diving into their pools of gold coins...

mratzloff 13y ago
But who plans a New Year's Eve datacenter migration? What kind of major site maintenance is needed on Christmas? It's one thing if it's an unplanned emergency. It's another entirely if it's planned. Planning for work on Christmas or New Year's Eve would be enough to convince me to find employment elsewhere.
- tdurden 13y ago
  exactly.
- sokoloff 13y ago
  Well, we planned a NYE DC move (5 years ago I think), and years before that we didn't have the capability to do certain DB tasks with the site on-line, so we also planned the 2300 on 12/25 work. (And back in 2004, we couldn't even do releases with the site on-line, so we did regularly scheduled releases on Mondays starting at 1 AM with the outage starting at 2 AM...) How does this happen? Suppose we find ourselves running a high-margin e-commerce business and needing to take downtime sometime to accomplish some maintenance work. When should we schedule it? Managers and directors of teams in my Ops org researched the expected bookings during various windows and suggested those times. They recommended them to me. I spent some of the planning effort double-checking and making sure the teams would be signed up for that, and got a consistent message of "This is what Ops does; we all knew what we were signing up for when we took this job." (Side note: Of the 9 people on that DC migration trip, 8 still work here, 5+ years later.) On the 11PM on 12/25 outage, we were upgrading OS on a few key infrastructure servers that were singletons. (We relied on DB and file servers in a way that didn't give us the ability to do maintenance and serve traffic at the same time.) On the DC move, we were literally decommissioning our main DC and moving those services to a new DC in the same town. We bought and brought online as much duplicate gear as we could, but the core transactional systems were a singleton and needed to be moved, not replicated, and that was going to be 4-8 hours of downtime. My teams chose to put that downtime into a time when we'd lose $1MM of high-margin bookings rather than $3MM of high-margin bookings. Our CEO, without me asking, was also keeping tabs on the migration (from his house) and sent IMs and emails congratulating and thanking the team for our work when we wrapped things up around 4 AM on 1/1. So, I got to see both sides of it: doing what I believed was right as a leader and being present and engaged when they were doing a less pleasant part of their job, and I got to feel what it was like to get clear engagement and genuine thanks from above, even though it was technically "just" a symbolic gesture. He could have sent that mail at 10 AM the next day, but knowing him, he was keeping tabs on the work not to check up on us or because he was petrified of a multi-day outage, but because he wanted to be able to be the first one to thank us. Side note: these experiences allowed me to better understand what Ops goes through, and have caused us to prioritize (and since complete) architectural projects that permit us to do almost every bit of periodic maintenance with the site online. We haven't taken planned downtime in several years now, because we've made those improvements. From a pure dollars-and-sense standpoint, those projects "shouldn't win" over larger business opportunities that we have, but they did win, in significant part because they were the right thing to do for our people, and having been there, I can argue for them from a position of first-hand knowledge. There is an awful lot of negativity and anti-company sentiment whenever inevitable situations come up where, on an instantaneous basis, either the employee can get what's best for them XOR the company can get what's best for it. I don't personally see these situations to be quite the same as Walmart opening at 5 PM on Thanksgiving Day, but I'm open to the idea that others see them as being exactly the same thing. I understand and respect the perspective of someone who would never do planned work on a holiday, or even those who would never do planned work on a weekend. It's a perfectly reasonable point of view, and I will happily employ those people in roles that don't require off-hours work. Those people will presumably not choose a career in operations, or at least they probably shouldn't. It is always the case that I speak for myself and not my employer, but it's worth calling that out on this post in particular.
tdurden 13y ago
> The engineers make the sacrifice because they believe it's necessary for the product/company to succeed. They feel better about it because their executives are physically present there with them Really? Your engineers are so easily swayed by having a couple execs present at the office on Christmas eve that they happily ignore their own lives?