6 ms·

>Does this mean that Docker -- a major infrastructure company -- does not have any on-call engineers available to fix this? It appear to be that way. Reminds

>Does this mean that Docker -- a major infrastructure company -- does not have any on-call engineers available to fix this? It appear to be that way. Reminds me when all of the reddit admins were stuck on a plane on the way back from a wedding [1]. Remember kids, improve your bus factor. http://highscalability.com/blog/2013/8/26/reddit-lessons-learned-from-mistakes-made-scaling-to-1-billi.html http://highscalability.com/blog/2013/8/26/reddit-lessons-lea...

StavrosK 10y ago
And don't put everyone on the same bus, I guess.
- sleepychu 10y ago
  Actually since the aim is to avoid being hit by a bus, inside a bus seems like a pretty good place to put everyone.
  - deleted 10y ago
    [deleted]
  - mpnordland 10y ago
    Assume N is the total number of buses. Where N=1 your solution works. For N>1 the engineers can still be hit by a bus. This also ignores all other vehicles such as garbage trucks, semis and stealth bombers, all of which can take out your bus.
  - Sanddancer 10y ago
    You want to avoid sticking everyone on the same (air)bus too. https://en.wikipedia.org/wiki/Pacific_Southwest_Airlines_Flight_1771#Aftermath https://en.wikipedia.org/wiki/Pacific_Southwest_Airlines_Fli...
  - FireBeyond 10y ago
    Also called (by me) as "Baldrick's Bullet": https://www.youtube.com/watch?v=pKRxX3s3JlM https://www.youtube.com/watch?v=pKRxX3s3JlM Captain Blackadder: Baldrick, what are you doing out there? Private Baldrick: I'm carving something on a bullet, sir. Captain Blackadder: What are you craving? Private Baldrick: I'm carving "Baldrick", sir. Captain Blackadder: Why? Private Baldrick: It's part of a cunning plan, sir. Captain Blackadder: Of course it is. Private Baldrick: You know how they say that somewhere there's a bullet with your name on it? Captain Blackadder: Yes? Private Baldrick: Well I thought that if I owned the bullet with my name on it, I'll never get hit by it. Cause I'll never shoot myself... Captain Blackadder: Oh, shame! Private Baldrick: And the chances of there being two bullets with my name on it are very small indeed. Captain Blackadder: Yes, it's not the only thing that is "very small indeed". Your brain for example- is brain's so minute, Baldrick, that if a hungry cannibal cracked your head open, there wouldn't be enough to cover a small water biscuit.
  - StavrosK 10y ago
    The problem was that, even though Reddit had multiple engineers, they were all on the same plane, and thus unavailable at the same time.
Bromskloss 10y ago
> all of the reddit admins were stuck on a plane on the way back from a wedding Clearly, we need to operate in cells, so that no one knows everybody and will have everybody over for weddings and other parties.
- bitJericho 10y ago
  Haha. Well its not uncommon to avoid putting an entire team on one plane. Pretty risky for a company to do that!
  - madgar 10y ago
    Yeah... the entire point of an on-call rotation is to specify who is available for incident response... if everybody is on the same plane at once then by definition nobody is on-call.
    - bitJericho 10y ago
      And in the extremely unlikely event of a plane crash, the company goes belly up.
oldmanhorton 10y ago
There was a comment a bit below that suggested that those who paid for commercial support got it 24/7, but if that's true, id imagine the fix that commercial support would have given to paying customers would have fixed it for everyone else too...
- shykes 10y ago
  Disclaimer: I work at Docker. I believe commercial releases are downloaded from a separate infrastructure (to be confirmed). Either way, the availability of Docker packages, free or commercial, is critical infrastructure and we should treat it as such. IMO our primary infrastructure team should have been involved, and someone should be on call for this. We'll do a post-mortem, find the root cause, and take corrective action as needed. Apologies for the inconvenience.
  - voltagex_ 10y ago
    You may want to lock that GitHub thread soon, it's getting argumentative and not very helpful.
    - shykes 10y ago
      > You may want to lock that GitHub thread soon, it's getting argumentative and not very helpful. In open-source we call that "thursday" :)
      - zjaffee 10y ago
        What ever happened to community over code?
        shykes 10y ago
        Part of a healthy community is accepting that people disagree a lot, have different values, and communicate their ideas in very different ways. Where we draw the line is if people are being intimidated, bullied, insulted, or anything that even remotely resembles harassment. Although I personally feel that some of the comments in that thread are pretty unfair and poorly informed, they don't seem to violate the social contract.
Rapzid 10y ago
I don't think this would surprise anyone that has used Docker Hub in their CD pipeline. So many reliability issues. We moved off it last year. When we went to cancel our subscription the other month downgrades/cancellations were broken on the site as a known issue; had to open a support ticket. Most of the UI issues were still present along with some new ones.
kfrz 10y ago
There are two tiers to Docker's support, for certain, but as pointed out on the Github issue by a Docker team member here (https://github.com/docker/docker/issues/23203#issuecomment-223326996 https://github.com/docker/docker/issues/23203#issuecomment-2...), there's a definite urgency sensed by the team.

StavrosK 10y ago

And don't put everyone on the same bus, I guess.

sleepychu 10y ago
Actually since the aim is to avoid being hit by a bus, inside a bus seems like a pretty good place to put everyone.
- deleted 10y ago
  [deleted]
- mpnordland 10y ago
  Assume N is the total number of buses. Where N=1 your solution works. For N>1 the engineers can still be hit by a bus. This also ignores all other vehicles such as garbage trucks, semis and stealth bombers, all of which can take out your bus.
- Sanddancer 10y ago
  You want to avoid sticking everyone on the same (air)bus too. https://en.wikipedia.org/wiki/Pacific_Southwest_Airlines_Flight_1771#Aftermath https://en.wikipedia.org/wiki/Pacific_Southwest_Airlines_Fli...
- FireBeyond 10y ago
  Also called (by me) as "Baldrick's Bullet": https://www.youtube.com/watch?v=pKRxX3s3JlM https://www.youtube.com/watch?v=pKRxX3s3JlM Captain Blackadder: Baldrick, what are you doing out there? Private Baldrick: I'm carving something on a bullet, sir. Captain Blackadder: What are you craving? Private Baldrick: I'm carving "Baldrick", sir. Captain Blackadder: Why? Private Baldrick: It's part of a cunning plan, sir. Captain Blackadder: Of course it is. Private Baldrick: You know how they say that somewhere there's a bullet with your name on it? Captain Blackadder: Yes? Private Baldrick: Well I thought that if I owned the bullet with my name on it, I'll never get hit by it. Cause I'll never shoot myself... Captain Blackadder: Oh, shame! Private Baldrick: And the chances of there being two bullets with my name on it are very small indeed. Captain Blackadder: Yes, it's not the only thing that is "very small indeed". Your brain for example- is brain's so minute, Baldrick, that if a hungry cannibal cracked your head open, there wouldn't be enough to cover a small water biscuit.
- StavrosK 10y ago
  The problem was that, even though Reddit had multiple engineers, they were all on the same plane, and thus unavailable at the same time.

Bromskloss 10y ago

> all of the reddit admins were stuck on a plane on the way back from a wedding Clearly, we need to operate in cells, so that no one knows everybody and will have everybody over for weddings and other parties.

bitJericho 10y ago
Haha. Well its not uncommon to avoid putting an entire team on one plane. Pretty risky for a company to do that!
- madgar 10y ago
  Yeah... the entire point of an on-call rotation is to specify who is available for incident response... if everybody is on the same plane at once then by definition nobody is on-call.
  - bitJericho 10y ago
    And in the extremely unlikely event of a plane crash, the company goes belly up.

oldmanhorton 10y ago

There was a comment a bit below that suggested that those who paid for commercial support got it 24/7, but if that's true, id imagine the fix that commercial support would have given to paying customers would have fixed it for everyone else too...

shykes 10y ago
Disclaimer: I work at Docker. I believe commercial releases are downloaded from a separate infrastructure (to be confirmed). Either way, the availability of Docker packages, free or commercial, is critical infrastructure and we should treat it as such. IMO our primary infrastructure team should have been involved, and someone should be on call for this. We'll do a post-mortem, find the root cause, and take corrective action as needed. Apologies for the inconvenience.
- voltagex_ 10y ago
  You may want to lock that GitHub thread soon, it's getting argumentative and not very helpful.
  - shykes 10y ago
    > You may want to lock that GitHub thread soon, it's getting argumentative and not very helpful. In open-source we call that "thursday" :)
    - zjaffee 10y ago
      What ever happened to community over code?
      - shykes 10y ago
        Part of a healthy community is accepting that people disagree a lot, have different values, and communicate their ideas in very different ways. Where we draw the line is if people are being intimidated, bullied, insulted, or anything that even remotely resembles harassment. Although I personally feel that some of the comments in that thread are pretty unfair and poorly informed, they don't seem to violate the social contract.

Rapzid 10y ago

I don't think this would surprise anyone that has used Docker Hub in their CD pipeline. So many reliability issues. We moved off it last year. When we went to cancel our subscription the other month downgrades/cancellations were broken on the site as a known issue; had to open a support ticket. Most of the UI issues were still present along with some new ones.

kfrz 10y ago

There are two tiers to Docker's support, for certain, but as pointed out on the Github issue by a Docker team member here (https://github.com/docker/docker/issues/23203#issuecomment-223326996 https://github.com/docker/docker/issues/23203#issuecomment-2...), there's a definite urgency sensed by the team.