7 ms·

Disappointed that one of the lessons was not “don’t deploy on Friday and then immediately run out the door.” I know most of you will say that that shouldn’t be

• sebastianmarr 8y ago
  The lesson was 'be kind'.
• saudioger 8y ago
  In an ideal world you'd never deploy on a Friday, but we've probably all done it.
  • jeremyjh 8y ago
    Deploying on a Friday is fine for many businesses - but it becomes riskier if you are leaving to a remote location with no internet access.
    • klodolph 8y ago
      Even if your automation is great, there’s often a series of events that goes like this: Error discovered -> Call person responsible -> Roll back It’s not just about whether you have the ability to fix the error, it’s about whether deploying on Friday is likely to disrupt people’s personal lives. Or, put another way, it’s not kind to deploy on Friday, it’s selfish. It looks good if you push out features quickly, and if it blows up someone else has to take time out of their weekend. On my team ops controls releases and if you miss the Thursday build you’re not getting anything in production until Monday.
  • RobertRoberts 8y ago
    We have a company policy to never deploy on Friday, and only under ideal circumstances on Thursday. This protects everyone from unneeded overtime and panicking clients during off hours. It's one of the central policies to ensure happy clients and smooth running operations that gets regular review and questions from clients when they are in a hurry. But when it was implemented, stress levels across the board plummeted. And only a small amount of client education was needed before they agreed it was a good policy. There are emergency circumstances that override this rule of course.
• peterwwillis 8y ago
  Actually, what I learned is that being afraid to deploy on Friday means you're lacking in testing, verification, auto healing and rollback processes. Also, if something goes down in a way that requires a human to work on the weekend, it should result in a postmortem, and all of the components in the deployment chain related to the failure should be evaluated, with new tasks to fix their causes. If it happens multiple times, all project work should stop until it's fixed. This of course is balanced against how much failure your business can tolerate. If the service goes down and nobody loses money, do you really need your engineers working overtime to fix it?
  • Angostura 8y ago
    > Actually, what I learned is that being afraid to deploy on Friday means you're lacking in testing, verification, auto healing and rollback processes. Or being afraid to deploy last thing on Fridays is an admission that maybe... just maybe... you're not infallible
    • protomyth 8y ago
      I always assumed that if a deploy was scheduled for Friday, it meant we were all working the weekend because no one had confidence it would go well.
      • minor3rd 8y ago
        My team deploys probably 5 times in a given day, including Friday. They are all small deploys, and none can happen at the end of the day. If shit hits the fan, we rollback and maybe people figure out the root cause over the weekend but they aren't sweating bullets.
        • rjzzleep 8y ago
          I guess all the people commenting are much more competent than the people I've worked with. Even with tests, I've seen startups do double charges on accounts and whatnot on deployments which didn't show up until the next day. I've also seen ops people updating OES where the storage service would segfault a day later. How does DevOps and OES go together in one sentence, you ask? it doesn't, but it just means not all ops people are pure wisdom either. The guy caused others to waste 72 hours of compute resources, because of this. So it's not limited to Dev. And yes, the first company did learn from the double spending bug, but why learn on a saturday? So even if your DevOps practices are amazing and you have 70% test coverage on all your components, that doesn't mean you can't deploy faulty components where the deployment itself appears successful. Now what? Things aren't failing, they appear just fine. Someone has to go in and debug the problem, it may affect multiple components, it may be critical, and a simple rollback may not cut it. Friday deployments are fine for certain components, but surely not as a general rule for everything? Friday deployments are like Monday morning or Friday meetings. You can do them, and most of the times they'll be fine, but maybe out of respect for your colleagues you shouldn't anyway.
• barbecue_sauce 8y ago
  A weekend of downtime? If you deploy on Friday, run out the door, and soon find out that your contribution to a deployment caused an outage, wouldn't you immediately return to work to at least give the appearance of personal responsibility? (on any day of the week, even) Also, wouldn't you just do a rollback to the last viable build?
  • orev 8y ago
    I guess you didn’t read the post. “Deploy then run out the door and drive 3 hours out into the woods without your laptop.”