6 ms·

Hot Takes on Code Freezes

47 points by kiyanwang 5y ago

deleted 5y ago
[deleted]
protomyth 5y ago
Also, in some climates, weather will play a role in IT activities. Even those working from home are not immune from weather related events that might affect their house / apartment. Pipes freezing will end most IT related functions from that employee. People who might have to work regardless of code freeze might have other issues. The air conditioning in the server room failing when it is -24F is a possibility.
- Jenk 5y ago
  One company that employed me has ops based in the Philippines. Doesn't matter how much you pay, if there's a typhoon the grid is going down and ops will be unavailable.
  - protomyth 5y ago
    I wonder how many of us have vendors that depend on support people in the Ukraine. I suppose that is an unexpected and unwelcome risk.
    - Jenk 5y ago
      Funnily enough the same company employ (via agency) devs in the UKR. Most of the country is stable.
sc68cal 5y ago
> Everyone who is on-call or working support during the holidays needs to be given mandatory additional time off and should receive holiday pay. Companies don't want to do that. That costs money. So instead, they developed the idea of a "code freeze" which doesn't cost money (directly). In the mind of management, if there aren't any code changes, then nothing can break. Obviously, this is not the case, but that's not the point. They're not going to pay more for overtime and the marginal cost of deploying changes over the holidays does not justify the expense.
- abernard1 5y ago
  I worked at an ecom unicorn that did this and had been doing so for about a decade. I think it saved them quite a bit of money by giving extra time off to ops in exchange for their extra work during the holidays. Code freezes (in my experience) don't actually reduce the number of bugs or incidents. I think in all probability they increase the number of incidents and cost money because of the warm up period. But they do shift the timing when incidents occur, and that can be nice if your customers are sensitive to holiday sales. It's also nice to your employees. Paying your SREs a bit more for their tour of duty to the customers and employees is worth it.
ketzo 5y ago
Reading posts like this one, I am very glad that I work at a place where we have 1) an engineer on-call rotation for every discipline - no database engineer has to troubleshoot a broken CSS rule 2) on-call engineers get paid a big chunk of change, and even more on holidays 3) all the on-call engineers are much smarter than I am (a little harder to replicate this one, unless you're dumb like me) I honestly kind of forget that it even exists because it runs so smoothly. Good opportunity for me to get some perspective and be grateful! Edit: since someone asked, here are some other cool things that my engineering org does so smoothly that I forget they exist - Very very good automated build monitoring: when you push code for review / to internal / to prod, you know what happens to it as it happens. - Business tech is something I've literally never had to think about for more than eight seconds – laptop, idm, badge, slack, outlook, etc has worked absolutely perfectly the entire time I've been here (except, weirdly, this one meeting room that refuses to stay reserved.) - Testing: we have a ridiculously competent QA apparatus. Our product is used by customers with extremely stringent UI requirements, and so regression testing is mission-critical, even for tiny UI bugs. Every team has a QA, but we also have a QA Architect position who helps guide org-wide decisions. Generally, QAs are also very respected and treated as peer engineers by feature developers, and I think it shows. I work at a big SaaS company, and there are entire teams of people devoted to developer tools, so this stuff is all """basic""", I would argue; but man, it's easy to forget how much human effort has to go into making it basic, and how often we fuck up the "easy" stuff.
- thebean11 5y ago
  Where do you work?
  - ketzo 5y ago
    I try to keep my work identity a little distant from my online presence, but feel free to hit me at the email on my HN profile if you're really curious. You could probably track it down if you really cared to. It's an excellent place to work, particularly as an engineer, and we're always hiring.
    - 5y ago
blacksmith_tb 5y ago
I personally find them frustrating because they throw off our regular release cycle. Sure, we didn't deploy any potentially buggy work (or any fixes for known bugs...), but now we have to manage a giant release when we 'thaw'. And make sure no work gets left out, which is more likely since we're off our normal cadence.
- layoric 5y ago
  Another option is (if possible) not to have a release cadence in the first place. If deployments are happening regularly everyday as a part of small PRs, you just get a backlog of code reviews. Sure some releases will require orchestration where order is important but better than one giant release. DB migrations are a different story imo, and should get more planning. I’m guessing there would be other stacks/domains where regular release intervals suit customers better though.
thrower123 5y ago
It is better to just accept that nothing is going to happen between Thanksgiving and New Years. Chill out, enjoy the holidays, and don't push so hard.
pram 5y ago
"Deploy freezes shouldn’t mean idle engineers." Why not? Thats the best part.
- rubyist5eva 5y ago
  Never allowed to have a “slow day”, gotta be grinding on at 100% effort all the time! Money to be made!
  - layoric 5y ago
    Fitter.. Healthier.. More Productive..
- paxys 5y ago
  If you have nothing to do at work during the holidays then take the time off and actually enjoy it. Sitting in front of your laptop on the clock doing nothing is pointless for everyone.
  - amflare 5y ago
    Getting paid while having low expectations sounds wonderful, actually. Perfect time to work on pet peeves or professional devlopment.
paxys 5y ago
TL;DR – Incidents can still happen despite code freezes. Developers on call should be paid extra. Account for the holiday season during quarterly planning. Time during deploy freezes can be spent on other tasks like planning, specs, tooling, testing. Are any of these really "hot takes"?
- edoceo 5y ago
  They are hot/new if one is fresh to the game. If you've been around like 10 years be prepared for many repeating patterns. My personal favourite is the flapping between Server side and Client side business-logic - and the same circular conversations.
- teh_klev 5y ago
  > Developers on call should be paid extra Sadly there are many organisations out there who don't respect employee annual leave and expect their engineering team to be on-call without compensation. The usual excuses are things like "it's the holidays, it'll be quiet".
  - toomuchtodo 5y ago
    GTFO of those shops if you can as a practitioner. Don’t take workplace abuse or provide uncompensated time.
- avidiax 5y ago
  One of the better "hot takes" was saved till the end: >A deploy freeze means that folks are not actively trying to make progress on a dev project that needs to be deployed. Deploy freezes need to equate to either time off, or time spent on things that do not need to be deployed when complete. The author does a bad job explaining why. If you allow deployable work to progress without deployment during a holiday code freeze, you are just shifting, sometimes magnifying risk post-code freeze. All that undeployed work during the code freezer will increase the rate of change tremendously post-code-freeze.
  - henryfjordan 5y ago
city41 5y ago
I don't think "preventing issues" is the reason for deploy freezes. It's more about minimizing change with the hope of minimizing the surface area the skeleton crew needs to consider when an issue does happen.
tlarkworthy 5y ago
> Hot Take: Putting a deploy freeze in place does not mean you will be incident free. I find this does not match my experience at all! The number one cause of incidents are code or configuration changes. Freeze those and the incidents DO go away 95%. Of course you need to update eventually and the problems will begin again, but freezing when there is the least amount of staff to fight fires is very sensible??? of course, nothing is 100% but let's not rock the boat so that the rare traffic based incident can be dealt with by the few staff that are there.
- delta1 5y ago
  Anecdotally, configuration changes seem to cause the biggest outages these days.
  - longcommonname 5y ago
    We call them change freezes, they are generally done when there are less people around. This allows employees to celebrate holidays. These do reduce, but don't elimate incidents. Unilateral change blocks, stink, but don't put support in the weird position of taking on something they can't.
AtNightWeCode 5y ago
Incidents are mostly caused by untested configuration changes, infrastructure failures and often known limitations of software. A “hot take” could be something like code freezes are bad cause when it comes to most software and incidents the risk with changing the ways of working and not improving the software can be bigger than the risk with new releases. I think it is ok with code freezes though.

47 points by kiyanwang 5y ago

deleted 5y ago

[deleted]

Also, in some climates, weather will play a role in IT activities. Even those working from home are not immune from weather related events that might affect their house / apartment. Pipes freezing will end most IT related functions from that employee. People who might have to work regardless of code freeze might have other issues. The air conditioning in the server room failing when it is -24F is a possibility.

Jenk 5y ago
One company that employed me has ops based in the Philippines. Doesn't matter how much you pay, if there's a typhoon the grid is going down and ops will be unavailable.
- protomyth 5y ago
  I wonder how many of us have vendors that depend on support people in the Ukraine. I suppose that is an unexpected and unwelcome risk.
  - Jenk 5y ago
    Funnily enough the same company employ (via agency) devs in the UKR. Most of the country is stable.

sc68cal 5y ago

> Everyone who is on-call or working support during the holidays needs to be given mandatory additional time off and should receive holiday pay. Companies don't want to do that. That costs money. So instead, they developed the idea of a "code freeze" which doesn't cost money (directly). In the mind of management, if there aren't any code changes, then nothing can break. Obviously, this is not the case, but that's not the point. They're not going to pay more for overtime and the marginal cost of deploying changes over the holidays does not justify the expense.

abernard1 5y ago
I worked at an ecom unicorn that did this and had been doing so for about a decade. I think it saved them quite a bit of money by giving extra time off to ops in exchange for their extra work during the holidays. Code freezes (in my experience) don't actually reduce the number of bugs or incidents. I think in all probability they increase the number of incidents and cost money because of the warm up period. But they do shift the timing when incidents occur, and that can be nice if your customers are sensitive to holiday sales. It's also nice to your employees. Paying your SREs a bit more for their tour of duty to the customers and employees is worth it.

ketzo 5y ago

Reading posts like this one, I am very glad that I work at a place where we have 1) an engineer on-call rotation for every discipline - no database engineer has to troubleshoot a broken CSS rule 2) on-call engineers get paid a big chunk of change, and even more on holidays 3) all the on-call engineers are much smarter than I am (a little harder to replicate this one, unless you're dumb like me) I honestly kind of forget that it even exists because it runs so smoothly. Good opportunity for me to get some perspective and be grateful! Edit: since someone asked, here are some other cool things that my engineering org does so smoothly that I forget they exist - Very very good automated build monitoring: when you push code for review / to internal / to prod, you know what happens to it as it happens. - Business tech is something I've literally never had to think about for more than eight seconds – laptop, idm, badge, slack, outlook, etc has worked absolutely perfectly the entire time I've been here (except, weirdly, this one meeting room that refuses to stay reserved.) - Testing: we have a ridiculously competent QA apparatus. Our product is used by customers with extremely stringent UI requirements, and so regression testing is mission-critical, even for tiny UI bugs. Every team has a QA, but we also have a QA Architect position who helps guide org-wide decisions. Generally, QAs are also very respected and treated as peer engineers by feature developers, and I think it shows. I work at a big SaaS company, and there are entire teams of people devoted to developer tools, so this stuff is all """basic""", I would argue; but man, it's easy to forget how much human effort has to go into making it basic, and how often we fuck up the "easy" stuff.

thebean11 5y ago
Where do you work?
- ketzo 5y ago
  I try to keep my work identity a little distant from my online presence, but feel free to hit me at the email on my HN profile if you're really curious. You could probably track it down if you really cared to. It's an excellent place to work, particularly as an engineer, and we're always hiring.
  - 5y ago

blacksmith_tb 5y ago

I personally find them frustrating because they throw off our regular release cycle. Sure, we didn't deploy any potentially buggy work (or any fixes for known bugs...), but now we have to manage a giant release when we 'thaw'. And make sure no work gets left out, which is more likely since we're off our normal cadence.

layoric 5y ago
Another option is (if possible) not to have a release cadence in the first place. If deployments are happening regularly everyday as a part of small PRs, you just get a backlog of code reviews. Sure some releases will require orchestration where order is important but better than one giant release. DB migrations are a different story imo, and should get more planning. I’m guessing there would be other stacks/domains where regular release intervals suit customers better though.

thrower123 5y ago

It is better to just accept that nothing is going to happen between Thanksgiving and New Years. Chill out, enjoy the holidays, and don't push so hard.

pram 5y ago

"Deploy freezes shouldn’t mean idle engineers." Why not? Thats the best part.

rubyist5eva 5y ago
Never allowed to have a “slow day”, gotta be grinding on at 100% effort all the time! Money to be made!
- layoric 5y ago
  Fitter.. Healthier.. More Productive..
paxys 5y ago
If you have nothing to do at work during the holidays then take the time off and actually enjoy it. Sitting in front of your laptop on the clock doing nothing is pointless for everyone.
- amflare 5y ago
  Getting paid while having low expectations sounds wonderful, actually. Perfect time to work on pet peeves or professional devlopment.

paxys 5y ago

TL;DR – Incidents can still happen despite code freezes. Developers on call should be paid extra. Account for the holiday season during quarterly planning. Time during deploy freezes can be spent on other tasks like planning, specs, tooling, testing. Are any of these really "hot takes"?

edoceo 5y ago
They are hot/new if one is fresh to the game. If you've been around like 10 years be prepared for many repeating patterns. My personal favourite is the flapping between Server side and Client side business-logic - and the same circular conversations.
teh_klev 5y ago
> Developers on call should be paid extra Sadly there are many organisations out there who don't respect employee annual leave and expect their engineering team to be on-call without compensation. The usual excuses are things like "it's the holidays, it'll be quiet".
- toomuchtodo 5y ago
  GTFO of those shops if you can as a practitioner. Don’t take workplace abuse or provide uncompensated time.
avidiax 5y ago
One of the better "hot takes" was saved till the end: >A deploy freeze means that folks are not actively trying to make progress on a dev project that needs to be deployed. Deploy freezes need to equate to either time off, or time spent on things that do not need to be deployed when complete. The author does a bad job explaining why. If you allow deployable work to progress without deployment during a holiday code freeze, you are just shifting, sometimes magnifying risk post-code freeze. All that undeployed work during the code freezer will increase the rate of change tremendously post-code-freeze.
- henryfjordan 5y ago

city41 5y ago

I don't think "preventing issues" is the reason for deploy freezes. It's more about minimizing change with the hope of minimizing the surface area the skeleton crew needs to consider when an issue does happen.

tlarkworthy 5y ago

> Hot Take: Putting a deploy freeze in place does not mean you will be incident free. I find this does not match my experience at all! The number one cause of incidents are code or configuration changes. Freeze those and the incidents DO go away 95%. Of course you need to update eventually and the problems will begin again, but freezing when there is the least amount of staff to fight fires is very sensible??? of course, nothing is 100% but let's not rock the boat so that the rare traffic based incident can be dealt with by the few staff that are there.

delta1 5y ago
Anecdotally, configuration changes seem to cause the biggest outages these days.
- longcommonname 5y ago
  We call them change freezes, they are generally done when there are less people around. This allows employees to celebrate holidays. These do reduce, but don't elimate incidents. Unilateral change blocks, stink, but don't put support in the weird position of taking on something they can't.

AtNightWeCode 5y ago

Incidents are mostly caused by untested configuration changes, infrastructure failures and often known limitations of software. A “hot take” could be something like code freezes are bad cause when it comes to most software and incidents the risk with changing the ways of working and not improving the software can be bigger than the risk with new releases. I think it is ok with code freezes though.