5 ms·

Lets do a quick back of the envelope calculation: Github reports 28,337,706 users by 2018-06-05 [1]. Lets assume 50% of these are active. Lets also assume that

Lets do a quick back of the envelope calculation: Github reports 28,337,706 users by 2018-06-05 [1]. Lets assume 50% of these are active. Lets also assume that, due to the unavailability of GH, around 2 usable hours per developer are lost. Another assumption is that each developer contributes around 50 US$ per hour. This means, this outage has cost us users: (28337706 * .5 * 2 * 50) = 1.351 billion US$. Perhaps not use MySQL for such critical systems? [1] https://github.com/search?q=type:user&type=Users https://github.com/search?q=type:user&type=Users

cddotdotslash 8y ago
That's a vast exaggeration. You're assuming every single one of GitHub's active users: 1) was active during the exact incident time and 2) is a business user earning US wages.
- edejong 8y ago
  How would you do the calculation of the costs of this outage? Don't forget that the 50 US$ is lost added value, not wages.
tjoff 8y ago
Average 50 percent active (as in daily usage)? 2 hours lost? I'd estimate a couple of order of magnitudes less...
- edejong 8y ago
  I don't think 2 hours is an exaggeration for an active user. The 50% might be too high, I agree.
  - tjoff 8y ago
    2 hours lost? Maybe 2 hours affected, but then just don't push/fetch during that time. Very few would have to stop and wait. Average lost I'd guess to 10 minutes (most of it being "huh, wonder what's up with github")
    - edejong 8y ago
      You lose quite a bit more. People keep checking if it's back online. The usual flow is disrupted. Questions might be asked about whether branches can be merged. Integration with CI might fail. Managers start asking questions why features cannot be deployed. Some of the tooling I wrote even would stop working because the API was unavailable. Unfortunately, it's easier to down vote, than to come with a better estimate of the total cost of a 13 hour down-time of github.
alexeiz 8y ago
The assumption that developer time is lost when Github is unavailable is wrong. The whole idea of Git being a distributed VCS that it does not require any connection to the main server (i.e. Github) to work with a local copy of the repository. If Github is down, I can still do my work locally and then push changes to Github when it's back online. The only case when I may get blocked, is when I need to fetch project dependencies hosted on Github to do the initial build of my project, which doesn't happen very often.
- goykasi 8y ago
  But you are ignoring a huge component of using github. If you only use github as a central, shared repo, sure you lost almost nothing. Their git infra seemed to be operational throughout this, but if you use the major features of github: issues, PRs, review, webhooks for CI, etc, you probably did lose out on developer time. Me and my team pushed tons of code to origin today (JST btw), but we were almost at a stand still as far as merging to master and closing out branches. Github being down had a huge affect on the process -- review, merging and CI success. Merging was our big one since we have protected branches via github -- master; CI was second since we received no webhook events. So either we wait it out (~8 hours) or we throw out our process and do something different until they fix it. The later didnt seem like a reasonable course of action. github != git Its easy to say, "well they are down, that doesnt affect git", but the reality is that a lot of orgs dont just use git. They use github. That fact envelopes a lot of process, routine, infra, schedule, money, etc. Luckily, its only been one day. But developer time has definitely beem lost. You cant reasonably say otherwise if you use github. edit: I will say I dont fully agree with the GP. Blaming the downtime on MySQL is silly and coming up with dollars lost is over the top. Things happen. That comment was a weird attack on a reasonably stable database. Github had a bad day; it happens.
  - platinium 8y ago
    Going forward, I see that the sensible solution for all small companies relying on cloud-hosted git is to always have a secondary cloud provider at all times.

cddotdotslash 8y ago

That's a vast exaggeration. You're assuming every single one of GitHub's active users: 1) was active during the exact incident time and 2) is a business user earning US wages.

edejong 8y ago
How would you do the calculation of the costs of this outage? Don't forget that the 50 US$ is lost added value, not wages.

tjoff 8y ago

Average 50 percent active (as in daily usage)? 2 hours lost? I'd estimate a couple of order of magnitudes less...

edejong 8y ago
I don't think 2 hours is an exaggeration for an active user. The 50% might be too high, I agree.
- tjoff 8y ago
  2 hours lost? Maybe 2 hours affected, but then just don't push/fetch during that time. Very few would have to stop and wait. Average lost I'd guess to 10 minutes (most of it being "huh, wonder what's up with github")
  - edejong 8y ago
    You lose quite a bit more. People keep checking if it's back online. The usual flow is disrupted. Questions might be asked about whether branches can be merged. Integration with CI might fail. Managers start asking questions why features cannot be deployed. Some of the tooling I wrote even would stop working because the API was unavailable. Unfortunately, it's easier to down vote, than to come with a better estimate of the total cost of a 13 hour down-time of github.

alexeiz 8y ago

The assumption that developer time is lost when Github is unavailable is wrong. The whole idea of Git being a distributed VCS that it does not require any connection to the main server (i.e. Github) to work with a local copy of the repository. If Github is down, I can still do my work locally and then push changes to Github when it's back online. The only case when I may get blocked, is when I need to fetch project dependencies hosted on Github to do the initial build of my project, which doesn't happen very often.

goykasi 8y ago
But you are ignoring a huge component of using github. If you only use github as a central, shared repo, sure you lost almost nothing. Their git infra seemed to be operational throughout this, but if you use the major features of github: issues, PRs, review, webhooks for CI, etc, you probably did lose out on developer time. Me and my team pushed tons of code to origin today (JST btw), but we were almost at a stand still as far as merging to master and closing out branches. Github being down had a huge affect on the process -- review, merging and CI success. Merging was our big one since we have protected branches via github -- master; CI was second since we received no webhook events. So either we wait it out (~8 hours) or we throw out our process and do something different until they fix it. The later didnt seem like a reasonable course of action. github != git Its easy to say, "well they are down, that doesnt affect git", but the reality is that a lot of orgs dont just use git. They use github. That fact envelopes a lot of process, routine, infra, schedule, money, etc. Luckily, its only been one day. But developer time has definitely beem lost. You cant reasonably say otherwise if you use github. edit: I will say I dont fully agree with the GP. Blaming the downtime on MySQL is silly and coming up with dollars lost is over the top. Things happen. That comment was a weird attack on a reasonably stable database. Github had a bad day; it happens.
- platinium 8y ago
  Going forward, I see that the sensible solution for all small companies relying on cloud-hosted git is to always have a secondary cloud provider at all times.