4 ms·

I recall reading years ago that cloud services were expected to run with a reliability of 3 or 4 '9's and that if they didn't competing services would quickly o

by s_dev 2mo ago

I recall reading years ago that cloud services were expected to run with a reliability of 3 or 4 '9's and that if they didn't competing services would quickly overtake them in adoption. The industry was supposed to be that cut throat. Has big tech reached a similar status like banks in that they are "too big to fail" i.e. when they do fail we all just look the other way and say: "well everyone else is out too". Didn't someone recently calculate that GitHub is running at 95%? For comparison the Irish Rail service which is not reliable has 80% of it's trains run on time. This seems absurd and really challenges a lot of ideas I had about big tech and cloud infrastructure. GitHub seems to have remained the dominant player relative to GitLab etc.

akmarinov 2mo ago
Competing services can't afford the data storage to compete with Big tech
- threetonesun 2mo ago
  This is the important part, we've moved away from monopolies on software to creating a vertical stack that allows a few companies to corner all cloud hardware, which let's them prevent anyone from competing on both price AND scale.
Galanwe 2mo ago
That canot happen if the market becomes monopolistic, with every bigtech out there buying every startup worth a penny. The erosion of anti trust in the US created this monstrosity.
hmry 2mo ago
https://vimeo.com/355556831 https://vimeo.com/355556831 We don't care. We don't have to. We're the phone company.
willchis 2mo ago
A lot of it is inertia I think. I hear people saying that they "just use so-and-so for version control" but really we all have tons of CI/CD build and test pipelines, config, business processes etc in Github (rightly or wrongly).
efficax 2mo ago
I think AI is going to change this. Once you've built around github you feel locked in, since moving all your CI and other actions out of there, and your workflows out of there, is labor intensive. Now you could migrate from github to another service in a week, maybe less depending on how many workflows you need to move.
- JackuB 2mo ago
  I feel you are correct. Throwing AI at this problem works reasonably well. “Rewrite this Actions workflow to run on $PROVIDER” is a sensible task. It takes a while by hand, but I recently saw a migration of old Jenkins projects to Actions with AI (yea), and it was smooth.
tokioyoyo 2mo ago
People overestimate how much they care about stuff. Moving off GitHub would be more costly for us than having 5% downtime. Obviously there’s a tipping point, but it shows people are tolerant given the price tags.
- dboreham 2mo ago
  For a reasonably common class of deployment and organization (E.g. an online service where the cost of a short outage is significant) it becomes a pain point. For instance if there's a sudden security flap and you need to re-spin your system and redeploy, but that process is gated on GitHub working, now you're screwed if that security flap happens when GitHub is in its 5% down time. Basically you've coupled your uptime to GitHub's uptime in some measure.
  - tokioyoyo 2mo ago
    I agree with you in spirit, as we have this problem from time to time as well, but number of businesses migrating off-GH shows this isn't the trigger point yet.
  - acedTrex 2mo ago
    More specifically you've coupled the likelihood of your downtime coinciding with githubs downtime. Which as its a product is likely a very small % chance.
advisedwang 2mo ago
If I'm running, say, an e-commerce business, an hour of downtime on my serving-path might cost me $100k of revenue. An hour of downtime for my development stack might cost me 30 person-hours = a few grand. As a result, I'm much more willing to have unreliable github than unreliable CDN/Compute/etc.
SkySkimmer 2mo ago
Github is a social network, not a cloud service. (only 5% /s)
deleted 2mo ago
[deleted]
frollogaston 2mo ago
That number of 9s might be a thing for the lower layer cloud services like on AWS or Azure or GCP. Even then, that us-east-1 outage was an accidental power move. It established that many are best off putting all their stuff on us-east-1 cause then their outages only happen at the best possible times, whereas multi-region complexity might cause its own kind of outage when nobody else is down.
- yurishimo 1mo ago
  Except you cannot run a business in Asia with datacenters in the US. The latency is unacceptable.

by s_dev 2mo ago

akmarinov 2mo ago

Competing services can't afford the data storage to compete with Big tech

threetonesun 2mo ago
This is the important part, we've moved away from monopolies on software to creating a vertical stack that allows a few companies to corner all cloud hardware, which let's them prevent anyone from competing on both price AND scale.

Galanwe 2mo ago

That canot happen if the market becomes monopolistic, with every bigtech out there buying every startup worth a penny. The erosion of anti trust in the US created this monstrosity.

hmry 2mo ago

https://vimeo.com/355556831 https://vimeo.com/355556831 We don't care. We don't have to. We're the phone company.

willchis 2mo ago

A lot of it is inertia I think. I hear people saying that they "just use so-and-so for version control" but really we all have tons of CI/CD build and test pipelines, config, business processes etc in Github (rightly or wrongly).

efficax 2mo ago

I think AI is going to change this. Once you've built around github you feel locked in, since moving all your CI and other actions out of there, and your workflows out of there, is labor intensive. Now you could migrate from github to another service in a week, maybe less depending on how many workflows you need to move.

JackuB 2mo ago
I feel you are correct. Throwing AI at this problem works reasonably well. “Rewrite this Actions workflow to run on $PROVIDER” is a sensible task. It takes a while by hand, but I recently saw a migration of old Jenkins projects to Actions with AI (yea), and it was smooth.

tokioyoyo 2mo ago

People overestimate how much they care about stuff. Moving off GitHub would be more costly for us than having 5% downtime. Obviously there’s a tipping point, but it shows people are tolerant given the price tags.

dboreham 2mo ago
For a reasonably common class of deployment and organization (E.g. an online service where the cost of a short outage is significant) it becomes a pain point. For instance if there's a sudden security flap and you need to re-spin your system and redeploy, but that process is gated on GitHub working, now you're screwed if that security flap happens when GitHub is in its 5% down time. Basically you've coupled your uptime to GitHub's uptime in some measure.
- tokioyoyo 2mo ago
  I agree with you in spirit, as we have this problem from time to time as well, but number of businesses migrating off-GH shows this isn't the trigger point yet.
- acedTrex 2mo ago
  More specifically you've coupled the likelihood of your downtime coinciding with githubs downtime. Which as its a product is likely a very small % chance.

advisedwang 2mo ago

If I'm running, say, an e-commerce business, an hour of downtime on my serving-path might cost me $100k of revenue. An hour of downtime for my development stack might cost me 30 person-hours = a few grand. As a result, I'm much more willing to have unreliable github than unreliable CDN/Compute/etc.

SkySkimmer 2mo ago

Github is a social network, not a cloud service. (only 5% /s)

deleted 2mo ago

[deleted]

frollogaston 2mo ago

That number of 9s might be a thing for the lower layer cloud services like on AWS or Azure or GCP. Even then, that us-east-1 outage was an accidental power move. It established that many are best off putting all their stuff on us-east-1 cause then their outages only happen at the best possible times, whereas multi-region complexity might cause its own kind of outage when nobody else is down.

yurishimo 1mo ago
Except you cannot run a business in Asia with datacenters in the US. The latency is unacceptable.