5 ms·
Taking the numbers at their face value this is a good exercise in some text book statistical test of hypothesis. Incidents: Before 89. After 126. What is the c
by srean 6y ago
Taking the numbers at their face value this is a good exercise in some text book statistical test of hypothesis.
Incidents: Before 89. After 126. What is the chance of this happening if the 'rate' of occurrence has not changed ?
Assuming an unknown but constant Poisson rate, we get the probability of observing what has been observed to be 0.00225.
A fortuitous thing about this test is that one does not need to know what that unknown constant rate is.
- visarga 6y agoMaybe Github has grown significantly since then. More complexity, more down time.
- dutchmartin 6y agoWell, Github introduced way more new features since Microsoft took over.
- paulintrognon 6y agoAlso, according to the article itself, "[it] could be all a part of coordinated effort to be more transparent about their service status".
- jcrubino 6y agoMicrosoft has every right to move fast and break things on their watch.
- jtdev 6y agoMaybe Microsoft is less averse to complexity - which would still make uptime degradation land squarely on Microsoft’s shoulders. Microsoft seems to be an organization that actually introduces too much complexity to their products imo.
- solidasparagus 6y ago> Assuming a Poisson rate That feels like a mighty big assumption. Probably big enough that trying to calculate the probability is more misleading than enlightening.
- srean 6y agoAs I mentioned, my comment is meant as an exercise. If we were to take the numbers more seriously, due diligence is necessary. That said, if we assume that one incident does not affect the other, then the Poisson nature falls out as a natural consequence of that independence and the assumption of a constant rate (our Null hypothesis). As long as the incidents are spaced out enough, that the possibility of one incident affecting the other is low, Poisson can be surprisingly realistic. Quite remarkable, given how simple it is. All in all not that bad an assumption for a back of the envelope calculation in a meeting. In practice, however, given more time, I would be looking at the statistics of inter-incident times more carefully. If those look sufficiently different from Exponentially distributed, a non-Poisson renewal process might be more appropriate than a Poisson process.
- deleted 6y ago[deleted]
- solidasparagus 6y agoBut why even start with an assumption that is so likely to be wrong? We know that incidents are frequently correlated. We know that scale and complexity add fragility. We know that GitHub has gotten bigger and more complex. The chance of the probability distribution holding constant over 5 years of major growth is basically zero.
- srean 6y agoWe are talking about different things. One is about attributing causes to an increase in failure rate, the other is about verifying whether there is any material increase in the rate at all. My comment addresses the latter as a back of the envelope calculation. Strictly speaking, when looked at through a fine toothed comb, yes the assumptions are very likely wrong. All models are wrong [0], but some of them are useful. The question is can we get some useful conclusions from such a simple model. In my experience I have been surprised by how often low failure rates are captured well by Poisson processes. Yes the assumptions could be wrong, but are they very likely to lead to wrong conclusions ? Empirical experience and math says otherwise. There are sound reasons for why this happens. If you are interested, you can pick that up from Feller. These [1] [2] links might also help. Given the data that we have, its a plenty good first cut, but that's what it is -- a first cut. With more data one can do a more refined analysis. [0] https://en.wikipedia.org/wiki/All_models_are_wrong https://en.wikipedia.org/wiki/All_models_are_wrong [1] https://en.wikipedia.org/wiki/Poisson_point_process#Approximations_with_Poisson_point_processes https://en.wikipedia.org/wiki/Poisson_point_process#Approxim... [2] https://en.wikipedia.org/wiki/Poisson_point_process#Convergence_to_a_Poisson_point_process https://en.wikipedia.org/wiki/Poisson_point_process#Converge...
- deleted 6y ago[deleted]
- wilde 6y agoWhat was the YoY change in incidents prior to the acquisition? I’d expect incidents to grow as a function of employee count at least.
- cj 6y agoThis was exactly my thought. More or less downtime (as reported by a status page) is probably affected more by changes to policies re: how / when incidents are posted publicly between "Github" and "Microsoft Github". Subjectively speaking (using Github daily) I haven't noticed a difference. In general, Github has never been extremely reliable even pre-Microsoft.