5 ms·

In the "What Is Not An Emergency?" section: > It is the end of the day on a Friday and it would just be great to get this CL in before the developer leaves for

In the "What Is Not An Emergency?" section: > It is the end of the day on a Friday and it would just be great to get this CL in before the developer leaves for the weekend. I laughed out loud because it reminded me of so many times I have seen it happen and then someone had to fix in the weekend. Who shares the same experience?

Glyptodon 7y ago
Some places I've worked explicitly won't push/deploy/similar on Friday.
- aledalgrande 7y ago
  That's a good policy. We currently don't deploy on Friday afternoons.
- atdt 7y ago
  But it's such a trivial change...! What could go wrong?
- Gibbon1 7y ago
  I've mostly worked for small industrial light manufacturers. First job I worked at they were always late with shipping stuff. And always taking stuff down to the UPS office after hours. After watching this for a few years I got the shop manager to adopt a rule. If it's not ready to be boxed up before lunch, it's not shipping today. After they started enforcing that they got a lot more productive and started shipping on time much more often. Turns out the mad rush to meet an arbitrary deadline absolutely kills productivity.
  - winrid 7y ago
    Forgive me, I don't understand your last sentence. It sounds contradictory vs "they got a lot more productive".
    - LanceH 7y ago
      They got fewer returns/defects when they stopped rushing.
      - winrid 7y ago
        I realize that now. I initially read it as they rushed to get things done before lunch everyday. Oops!
    - Gibbon1 7y ago
      What would happen is the shop would start thrashing with people switching from one task to another trying to hurry one unit through production. You eat the setup time for each production task for exactly one unit.
      - winrid 7y ago
        Yeah, I misread your comment as saying they rushed units in before lunch :D :D
  - Sammi 7y ago
    Your choices are often between these two: 1. Higher throughput, higer latency and correctness on one side. 2. Lower throughput, lower latency and some risk of faults on the other side. It's like the difference between tcp and udp. The question is: Where do you need to be? What are your requirements?
- joefkelley 7y ago
  I worked at one (horrible) place where we explicitly only deployed on Friday afternoon. The logic was we have fewer users over the weekend, so if something goes wrong, fewer people will notice.
  - fma 7y ago
    I rewrote an app that used to only deploy on weekends because we do a lot of processing on weekday afternoons and nights. I changed the deployd to the mornings. I justified as saying if something breaks we have everyone there to fix it. Plus I'm not going to spend my weekend working if I don't have to. My app has global traffic. Sometimes stuff breaks Sunday night and Asia is the first to find out.
  - Cthulhu_ 7y ago
    Ugh, I know the feeling - one client I worked for would only do deploys a few times a year if that, and only at night. I mean we'd get extra pay or free time for doing night shifts, but really, it's not good practice.
  - devnulloverflow 7y ago
    It depends on what company you are. Some who administers some SAP thing for a huge supermarket chain tells me they do all of their upgrades on weekends because their users aren't at work. Makes sense for them, but it would be a crazy thing to do the consumer-focused interweb company that I work for.
- dymk 7y ago
  But then Thursday just becomes Friday...
  - luminiferous 7y ago
    That's fine, because when something breaks the day after you've pushed, it's Friday and everyone's in the office. People are still going to rush the deadline, but putting the deadline earlier means you have time to deal with stuff that breaks after the deadline.
- twothamendment 7y ago
  We had no-deploy Friday which made support & product people feel happy and devs dread the pileup on Monday. It has evolved into "don't deploy higher impact things on Friday" and "try not to deploy at all Friday afternoon". That gives us a half day to hear about issues and fix them before the weekend.
- UK-Al05 7y ago
  This implies they don't have very good automated deploy, monitoring and automated rollback solution. The test of a good cd is that your ok with releasing on Friday at end of the day.
  - aledalgrande 7y ago
    You are assuming that monitoring and rollback solutions are bug free, always on and have 100% knowledge of what's going on. That is never true.
  - Glyptodon 7y ago
    I agree I've never worked anywhere with really heavy duty CI, as in I've never worked somewhere that automated rollbacks were either trusted in all cases or business acceptable after something was production live. (Which is to say I haven't worked at places that had really significant scale.) And CI is only as good as your test coverage, which is to say if something esoteric is overlooked in your test suite...
furyofantares 7y ago
We call it a chuck-in.
bt848 7y ago
They didn't give the two examples that are most common inside Google: Google I/O is coming up, or annual performance reviews are coming up. Strangely they also list as "not an emergency" rollbacks of clearly broken code, but that's an exception to review rules inside Google. Anyone can do a pure rollback of a change without getting the approval of the owners of the code, and there are automated tools that will roll back changes that appear to have broken at least 1000 tests, without human review. Edit: There's even a button right in the code review web UI to roll back a committed change.
- NoodleIncident 7y ago
  > Google I/O is coming up, or annual performance reviews are coming up. As long as everyone admits it, sounds like those are actual emergencies :)
- cameronbrown 7y ago
  > at least 1000 tests The threshold for this seems at least a little too high, doesn't it?
  - bt848 7y ago
    They have a lot of tests :-) Abseil-cpp alone has 2400 tests, at least in the open-source tree. https://github.com/abseil/abseil-cpp/tree/master/absl/base https://github.com/abseil/abseil-cpp/tree/master/absl/base
  - sanderjd 7y ago
    I was thinking low :)
AnimalMuppet 7y ago
Worse: The developer wants to get this checked in before they go on vacation. And they really want to, because they don't want to have to do a bunch of merges when they come back...

Glyptodon 7y ago

Some places I've worked explicitly won't push/deploy/similar on Friday.

aledalgrande 7y ago
That's a good policy. We currently don't deploy on Friday afternoons.
atdt 7y ago
But it's such a trivial change...! What could go wrong?
Gibbon1 7y ago
I've mostly worked for small industrial light manufacturers. First job I worked at they were always late with shipping stuff. And always taking stuff down to the UPS office after hours. After watching this for a few years I got the shop manager to adopt a rule. If it's not ready to be boxed up before lunch, it's not shipping today. After they started enforcing that they got a lot more productive and started shipping on time much more often. Turns out the mad rush to meet an arbitrary deadline absolutely kills productivity.
- winrid 7y ago
  Forgive me, I don't understand your last sentence. It sounds contradictory vs "they got a lot more productive".
  - LanceH 7y ago
    They got fewer returns/defects when they stopped rushing.
    - winrid 7y ago
      I realize that now. I initially read it as they rushed to get things done before lunch everyday. Oops!
  - Gibbon1 7y ago
    What would happen is the shop would start thrashing with people switching from one task to another trying to hurry one unit through production. You eat the setup time for each production task for exactly one unit.
    - winrid 7y ago
      Yeah, I misread your comment as saying they rushed units in before lunch :D :D
- Sammi 7y ago
  Your choices are often between these two: 1. Higher throughput, higer latency and correctness on one side. 2. Lower throughput, lower latency and some risk of faults on the other side. It's like the difference between tcp and udp. The question is: Where do you need to be? What are your requirements?
joefkelley 7y ago
I worked at one (horrible) place where we explicitly only deployed on Friday afternoon. The logic was we have fewer users over the weekend, so if something goes wrong, fewer people will notice.
- fma 7y ago
  I rewrote an app that used to only deploy on weekends because we do a lot of processing on weekday afternoons and nights. I changed the deployd to the mornings. I justified as saying if something breaks we have everyone there to fix it. Plus I'm not going to spend my weekend working if I don't have to. My app has global traffic. Sometimes stuff breaks Sunday night and Asia is the first to find out.
- Cthulhu_ 7y ago
  Ugh, I know the feeling - one client I worked for would only do deploys a few times a year if that, and only at night. I mean we'd get extra pay or free time for doing night shifts, but really, it's not good practice.
- devnulloverflow 7y ago
  It depends on what company you are. Some who administers some SAP thing for a huge supermarket chain tells me they do all of their upgrades on weekends because their users aren't at work. Makes sense for them, but it would be a crazy thing to do the consumer-focused interweb company that I work for.
dymk 7y ago
But then Thursday just becomes Friday...
- luminiferous 7y ago
  That's fine, because when something breaks the day after you've pushed, it's Friday and everyone's in the office. People are still going to rush the deadline, but putting the deadline earlier means you have time to deal with stuff that breaks after the deadline.
twothamendment 7y ago
We had no-deploy Friday which made support & product people feel happy and devs dread the pileup on Monday. It has evolved into "don't deploy higher impact things on Friday" and "try not to deploy at all Friday afternoon". That gives us a half day to hear about issues and fix them before the weekend.
UK-Al05 7y ago
This implies they don't have very good automated deploy, monitoring and automated rollback solution. The test of a good cd is that your ok with releasing on Friday at end of the day.
- aledalgrande 7y ago
  You are assuming that monitoring and rollback solutions are bug free, always on and have 100% knowledge of what's going on. That is never true.
- Glyptodon 7y ago
  I agree I've never worked anywhere with really heavy duty CI, as in I've never worked somewhere that automated rollbacks were either trusted in all cases or business acceptable after something was production live. (Which is to say I haven't worked at places that had really significant scale.) And CI is only as good as your test coverage, which is to say if something esoteric is overlooked in your test suite...

furyofantares 7y ago

We call it a chuck-in.

bt848 7y ago

They didn't give the two examples that are most common inside Google: Google I/O is coming up, or annual performance reviews are coming up. Strangely they also list as "not an emergency" rollbacks of clearly broken code, but that's an exception to review rules inside Google. Anyone can do a pure rollback of a change without getting the approval of the owners of the code, and there are automated tools that will roll back changes that appear to have broken at least 1000 tests, without human review. Edit: There's even a button right in the code review web UI to roll back a committed change.

NoodleIncident 7y ago
> Google I/O is coming up, or annual performance reviews are coming up. As long as everyone admits it, sounds like those are actual emergencies :)
cameronbrown 7y ago
> at least 1000 tests The threshold for this seems at least a little too high, doesn't it?
- bt848 7y ago
  They have a lot of tests :-) Abseil-cpp alone has 2400 tests, at least in the open-source tree. https://github.com/abseil/abseil-cpp/tree/master/absl/base https://github.com/abseil/abseil-cpp/tree/master/absl/base
- sanderjd 7y ago
  I was thinking low :)

AnimalMuppet 7y ago

Worse: The developer wants to get this checked in before they go on vacation. And they really want to, because they don't want to have to do a bunch of merges when they come back...