3 ms·

"Exhibit C: Our site works when the engineers go on holiday" Is not all so different from "Fewer patients die when heart surgeons are on vacation". Of course y

"Exhibit C: Our site works when the engineers go on holiday" Is not all so different from "Fewer patients die when heart surgeons are on vacation". Of course your site is going to be more reliable when nobody is changing it! You should be worried if reliable isn't the steady state, and it requires constant changes to stay up!

afc 10y ago
I also took issue with this part of the article; it seems obvious to me that rolling out new features would directly cause reliability issues (and the only alternative, of not rolling out new features, is, obviously, a cure worse than the disease).
- adrianratnapala 10y ago
  I am not sure that most Facebook users would actually agree about this. But then their view is slanted towards UI changes. Performance improvements and other subtleties are features too.
joshvm 10y ago
My immediate thinking was this as well: when they stop touching it, it's fine, which means that whatever bugs are checked in seem to be caught quickly. What's more, the periods of high reliability are during periods when you might expect there to be heavy load - assuming people don't check Facebook at work (har har). What counts as an incident? > Figure 1 includes data from an analysis of the timing of events severe enough to be considered an SLA (service-level agreement) violation. Each violation indicates an instance where our internal reliability goals were not met and caused an alert to be generated. Because our goals are strict most of these incidents are minor and not noticeable to users of the site. So these are minor issues. The parent article is paraphrasing at best, and jumping to conclusions at worst. From Facebook: > We believe this is not a result of carelessness on the part of people making changes but rather evidence that our infrastructure is largely self-healing in the face of non-human causes of errors such as machine failure. They then list a number of sane strategies to mitigate this. http://queue.acm.org/detail.cfm?id=2839461 http://queue.acm.org/detail.cfm?id=2839461
draw_down 10y ago
Engineers should not brag about this, in fact it's the opposite of something we should brag about. "Our site doesn't go down when we're not here devising ways to break it"
- ddacunha 10y ago
  I don't see this as bragging but as an reality/consequence of introducing change. We do a lot of things to minimise occurrence and impact of change but there always is(/will be?) that (corner-)case that break things. In other context,that is often a point against upgrading old systems that have been working and doing the job for a long time.
adrianratnapala 10y ago
Yes, I expect the graph in Exhibit C is perfectly normal for any company.

afc 10y ago

I also took issue with this part of the article; it seems obvious to me that rolling out new features would directly cause reliability issues (and the only alternative, of not rolling out new features, is, obviously, a cure worse than the disease).

adrianratnapala 10y ago
I am not sure that most Facebook users would actually agree about this. But then their view is slanted towards UI changes. Performance improvements and other subtleties are features too.

joshvm 10y ago

My immediate thinking was this as well: when they stop touching it, it's fine, which means that whatever bugs are checked in seem to be caught quickly. What's more, the periods of high reliability are during periods when you might expect there to be heavy load - assuming people don't check Facebook at work (har har). What counts as an incident? > Figure 1 includes data from an analysis of the timing of events severe enough to be considered an SLA (service-level agreement) violation. Each violation indicates an instance where our internal reliability goals were not met and caused an alert to be generated. Because our goals are strict most of these incidents are minor and not noticeable to users of the site. So these are minor issues. The parent article is paraphrasing at best, and jumping to conclusions at worst. From Facebook: > We believe this is not a result of carelessness on the part of people making changes but rather evidence that our infrastructure is largely self-healing in the face of non-human causes of errors such as machine failure. They then list a number of sane strategies to mitigate this. http://queue.acm.org/detail.cfm?id=2839461 http://queue.acm.org/detail.cfm?id=2839461

draw_down 10y ago

Engineers should not brag about this, in fact it's the opposite of something we should brag about. "Our site doesn't go down when we're not here devising ways to break it"

ddacunha 10y ago
I don't see this as bragging but as an reality/consequence of introducing change. We do a lot of things to minimise occurrence and impact of change but there always is(/will be?) that (corner-)case that break things. In other context,that is often a point against upgrading old systems that have been working and doing the job for a long time.

adrianratnapala 10y ago

Yes, I expect the graph in Exhibit C is perfectly normal for any company.