5 ms·

> Every tech company I’ve worked for has had at least 1 manager who tries to ship features over the weekend with a ragtag team of developers who don’t understan

by zild3d 7y ago

> Every tech company I’ve worked for has had at least 1 manager who tries to ship features over the weekend with a ragtag team of developers who don’t understand why that’s a bad idea. Maybe I've been in startup land for too long, but seems super normal and fine to ship a feature over the weekend if it goes through the regular CI gates - it's tested, peer reviewed, been QAd in staging, etc. Is this not accepted outside of startups?

closeparen 7y ago
This handy flowchart hangs on the wall near my company's core infrastructure group [0]. [0] https://www.amazon.com/Should-Deploy-Friday-T-Shirt-Sizes/dp/B07HDTHPMF https://www.amazon.com/Should-Deploy-Friday-T-Shirt-Sizes/dp...
- brongondwana 7y ago
  That's almost complete... there's another three question flow which gets to yes. 1) Is stuff really badly broken? 2) Was the really bad breakage introduced recently (this could either be an earlier bad rollout, or it could be external factors changing)? 3) Is this requested deploy either a revert of a recently-made bad change or the minimal possible fix/bandaid? If all three of these are true, then you can do a deploy RIGHT NOW regardless of the calendar. (recently - because if this is something which has been broken for years then it's unlikely that it suddenly became urgent absent external change - which is already carved out above)
  - michaelt 7y ago
    So what you're saying is, it's acceptable to deploy on Friday at 5pm if and only if you're undoing the damage caused by some chump deploying on Friday at 4:55pm?
    - Piskvorrr 7y ago
      Yes. And then to find out why a DEFCON-1 level revert was necessary at all: that's a deploy that you want to be prepared for, but you never want to use.
    - brongondwana 7y ago
      Or some data corrupting bullshit that was released on Thursday and it's taken you that long to notice but it's causing ongoing damage.
nbm 7y ago
Considering the described situation is someone asking for help understanding how to achieve something (which indicates they don’t have the familiarity with the processes in place to do these things), and they’re asking the team responsible for that infrastructure for the first time on the day before launch, I don’t think one can claim that it’s been well-tested and peer reviewed.
- zild3d 7y ago
  sure, but I'd think some gates on that are standard practice no? Owners needing to approve changes, tests passing, QA'd in a staging env, etc
  - nbm 7y ago
    Sure, but we’re so far away from that scenario here. In that ideal world you describe above, the project leads would have already talked through the requirements and processes involved with the environment and/or service owners and/or release engineering groups. Not all changes are of the sort where “if the tests pass, it’s safe to ship it”, and that requires learning what is acceptable and not, and talking to the relevant people when you’re out of scope. Environments like those described often have continuous push and automated slow rollouts with health checks, so the idea of doing something on a Sunday isn’t that strange at all. That said, there’s something to be said for not trying to locally optimize. If you push bad stuff on Sunday, you’re messing up a bunch of people’s well-earned rest and recovery time from work. You push bad stuff on Monday, and everyone’s there to help you fix it without the stress of lost family or other commitment time. The difference is 24 hours, which likely isn’t going to make or break anything. It’s easy to get sucked into believing things like that matter when they don’t.
PragmaticPulp 7y ago
Shipping over the weekend isn’t always a bad thing when it’s necessary, but it needs to be an all-hands effort. You need to give people a heads up and at least collect the minimum base knowledge to do it right. In this case, someone was trying to quietly ship things to production on a Sunday without involving the owners of the front end. How would it look for you if some other team crashed your part of the website on a Sunday without even coordinating the change with you first? My point was that it’s important for companies to not reward selfish behavior from managers who want to make a name for themselves. If you genuinely need to ship a website feature on Sunday, you involve the website team for launch and follow up monitoring. You don’t try to quietly ship it out the door at the risk of breaking other parts of the business, as Rachel explained in the article.
zentiggr 7y ago
That sounds like a nice smooth pre-production process... Her story was of an untested feature trying to get injected into the frontend on Sunday to meet a Monday deadline, by someone without the proper access, and with no apparent oversight or process concerns. I'd do everything I could to block this push as well.
- op00to 7y ago
  Wouldn’t “everything” in this case be to summon the managers to drop blocks on heads? If you are yourself a manager, summon a more powerful manager that can drop more blocks.
  - mateo411 7y ago
    Here's another idea, you can ship late. There has to be a really good reason to meet the Monday deadline and it must be really important to the company. In fact it would be so important that the people and departments would know that it's important. They wouldn't find out about it on the weekend the day before it ships. Sounds like these people designed a component that wouldn't work on the current infrastructure. That happens, but if there is a big oversight like this, then you should ship late, instead of risk taking everything down.
randomdude402 7y ago
I'm at a fintech in NYC and my team generally doesn't release stuff to prod between like 10AM Friday and 10AM Monday. I haven't had specific conversations with anybody about it, but I think we have all been around the block enough times to have been burned on a few weekends when it really wasn't necessary. Not a start up at all though, and not a team of twenty somethings with anything to prove by moving fast and breaking things. Agree with you though that I have seen this at a lot of places. I did a number of phone interviews looking for a more relaxed place in order to end up here.
- pnako 7y ago
  Deploying stuff over the week-end is standard practice for trading systems, because that's when markets are closed. But obviously those deployments are discussed and planned during the week. In fact most trading platforms have this huge advantage of not being 24/7 operations.
- deleted 7y ago
  [deleted]
kodablah 7y ago
> seems super normal and fine to ship a feature over the weekend if it goes through the regular CI gates - it's tested, peer reviewed, been QAd in staging, etc. Multiple of these processes you listed often require humans. You either are asking them to do this during the weekend, which is bad, or you gave them ample time during the week meaning you're ok with steps needed on weekdays, so you can do that for the last step too, with full staff present. (bugs and urgencies notwithstanding, but that doesn't appear to be what we're discussing)
- eska 7y ago
  Depends. If you have a full CD/CI system and use feature flags, you can definitely push such an update into production, having the feature disabled by default but enabled if the requests have special attributes set, or for a specified group of people (A/B).
  - kodablah 7y ago
    My comment is unrelated to what you can do, it's about what you should do. If you can afford weekday humans in the other parts of the process, you can afford a weekday release.
peterbonney 7y ago
The hypothetical scenario you’re describing and the story I just read have almost nothing in common except the word “weekend”. Deployment over the weekend can make a lot of sense in the world of B2B, but there’s a difference between a carefully thought out plan to deploy at a quiet time and sneaking something out when no one is looking.
mifreewil 7y ago
It really depends on what it is you're shipping. Once a startup hits some level of maturity, it's unacceptable to be shipping something significant on the weekend (or whenever people aren't around to respond to an issue). Probably post product-market fit, maybe Series B. I guess it also matters how much your company values work-life balance.

by zild3d 7y ago

closeparen 7y ago

This handy flowchart hangs on the wall near my company's core infrastructure group [0]. [0] https://www.amazon.com/Should-Deploy-Friday-T-Shirt-Sizes/dp/B07HDTHPMF https://www.amazon.com/Should-Deploy-Friday-T-Shirt-Sizes/dp...

brongondwana 7y ago
That's almost complete... there's another three question flow which gets to yes. 1) Is stuff really badly broken? 2) Was the really bad breakage introduced recently (this could either be an earlier bad rollout, or it could be external factors changing)? 3) Is this requested deploy either a revert of a recently-made bad change or the minimal possible fix/bandaid? If all three of these are true, then you can do a deploy RIGHT NOW regardless of the calendar. (recently - because if this is something which has been broken for years then it's unlikely that it suddenly became urgent absent external change - which is already carved out above)
- michaelt 7y ago
  So what you're saying is, it's acceptable to deploy on Friday at 5pm if and only if you're undoing the damage caused by some chump deploying on Friday at 4:55pm?
  - Piskvorrr 7y ago
    Yes. And then to find out why a DEFCON-1 level revert was necessary at all: that's a deploy that you want to be prepared for, but you never want to use.
  - brongondwana 7y ago
    Or some data corrupting bullshit that was released on Thursday and it's taken you that long to notice but it's causing ongoing damage.

nbm 7y ago

Considering the described situation is someone asking for help understanding how to achieve something (which indicates they don’t have the familiarity with the processes in place to do these things), and they’re asking the team responsible for that infrastructure for the first time on the day before launch, I don’t think one can claim that it’s been well-tested and peer reviewed.

zild3d 7y ago
sure, but I'd think some gates on that are standard practice no? Owners needing to approve changes, tests passing, QA'd in a staging env, etc
- nbm 7y ago
  Sure, but we’re so far away from that scenario here. In that ideal world you describe above, the project leads would have already talked through the requirements and processes involved with the environment and/or service owners and/or release engineering groups. Not all changes are of the sort where “if the tests pass, it’s safe to ship it”, and that requires learning what is acceptable and not, and talking to the relevant people when you’re out of scope. Environments like those described often have continuous push and automated slow rollouts with health checks, so the idea of doing something on a Sunday isn’t that strange at all. That said, there’s something to be said for not trying to locally optimize. If you push bad stuff on Sunday, you’re messing up a bunch of people’s well-earned rest and recovery time from work. You push bad stuff on Monday, and everyone’s there to help you fix it without the stress of lost family or other commitment time. The difference is 24 hours, which likely isn’t going to make or break anything. It’s easy to get sucked into believing things like that matter when they don’t.

PragmaticPulp 7y ago

Shipping over the weekend isn’t always a bad thing when it’s necessary, but it needs to be an all-hands effort. You need to give people a heads up and at least collect the minimum base knowledge to do it right. In this case, someone was trying to quietly ship things to production on a Sunday without involving the owners of the front end. How would it look for you if some other team crashed your part of the website on a Sunday without even coordinating the change with you first? My point was that it’s important for companies to not reward selfish behavior from managers who want to make a name for themselves. If you genuinely need to ship a website feature on Sunday, you involve the website team for launch and follow up monitoring. You don’t try to quietly ship it out the door at the risk of breaking other parts of the business, as Rachel explained in the article.

zentiggr 7y ago

That sounds like a nice smooth pre-production process... Her story was of an untested feature trying to get injected into the frontend on Sunday to meet a Monday deadline, by someone without the proper access, and with no apparent oversight or process concerns. I'd do everything I could to block this push as well.

op00to 7y ago
Wouldn’t “everything” in this case be to summon the managers to drop blocks on heads? If you are yourself a manager, summon a more powerful manager that can drop more blocks.
- mateo411 7y ago
  Here's another idea, you can ship late. There has to be a really good reason to meet the Monday deadline and it must be really important to the company. In fact it would be so important that the people and departments would know that it's important. They wouldn't find out about it on the weekend the day before it ships. Sounds like these people designed a component that wouldn't work on the current infrastructure. That happens, but if there is a big oversight like this, then you should ship late, instead of risk taking everything down.

randomdude402 7y ago

I'm at a fintech in NYC and my team generally doesn't release stuff to prod between like 10AM Friday and 10AM Monday. I haven't had specific conversations with anybody about it, but I think we have all been around the block enough times to have been burned on a few weekends when it really wasn't necessary. Not a start up at all though, and not a team of twenty somethings with anything to prove by moving fast and breaking things. Agree with you though that I have seen this at a lot of places. I did a number of phone interviews looking for a more relaxed place in order to end up here.

pnako 7y ago
Deploying stuff over the week-end is standard practice for trading systems, because that's when markets are closed. But obviously those deployments are discussed and planned during the week. In fact most trading platforms have this huge advantage of not being 24/7 operations.
deleted 7y ago
[deleted]

kodablah 7y ago

> seems super normal and fine to ship a feature over the weekend if it goes through the regular CI gates - it's tested, peer reviewed, been QAd in staging, etc. Multiple of these processes you listed often require humans. You either are asking them to do this during the weekend, which is bad, or you gave them ample time during the week meaning you're ok with steps needed on weekdays, so you can do that for the last step too, with full staff present. (bugs and urgencies notwithstanding, but that doesn't appear to be what we're discussing)

eska 7y ago
Depends. If you have a full CD/CI system and use feature flags, you can definitely push such an update into production, having the feature disabled by default but enabled if the requests have special attributes set, or for a specified group of people (A/B).
- kodablah 7y ago
  My comment is unrelated to what you can do, it's about what you should do. If you can afford weekday humans in the other parts of the process, you can afford a weekday release.

peterbonney 7y ago

The hypothetical scenario you’re describing and the story I just read have almost nothing in common except the word “weekend”. Deployment over the weekend can make a lot of sense in the world of B2B, but there’s a difference between a carefully thought out plan to deploy at a quiet time and sneaking something out when no one is looking.

mifreewil 7y ago

It really depends on what it is you're shipping. Once a startup hits some level of maturity, it's unacceptable to be shipping something significant on the weekend (or whenever people aren't around to respond to an issue). Probably post product-market fit, maybe Series B. I guess it also matters how much your company values work-life balance.