3 ms·

Please disregard the following offtopic rant; I'm not sure if I'm just getting older and jaded or if Github (and the web at large, if I might be frank) is havin

by 0des 5y ago

Please disregard the following offtopic rant; I'm not sure if I'm just getting older and jaded or if Github (and the web at large, if I might be frank) is having these issues more often as of late. Is it that what we are doing is becoming more complex, or are we, as a web, moving faster and breaking more things? I wonder sometimes if the advancement and affordability of our hardware and the systems we can run on them has decreased our reliance on optimization and correctness as a requirement of deploying on affordable hardware. Basic functions used to have to be combined with great rigor to run on the same machine for swaths of users. Now we are so far from that we have 1000 layers of abstraction, and it isn't hard to imagine how lots of tiny inefficiencies stacked on top of each other may be adding up to bigger issues. In my org we have a term for this, where you're encumbered by faults deep in the toolchain - getting 'middlewared'.

thih9 5y ago
Even more offtopic: your comment makes me think of other industries and realise how harmful the heavily processed food must be.
grumple 5y ago
While the web is quite complex now, I think the expectation of 100% uptime for anything is unreasonable. My local taco shop ran out of rice a few weeks ago. A few days ago they had some unspecified kitchen issue and had to close. They don't open on sundays. Occasional downtime is normal. We can try to prevent it, as devs, working with machines, but it is inevitable. Nothing continues working forever. All industries have downtime, it needs to be accepted and planned for. Even in the old days, with the simplest possible thing that needs a db connection, you could outscale that db, or bandwidth, or whatever, and crash. It was easier back then in a lot of ways - 1000 people visit your new site? Crash. Before auto scaling infra you had to either deal with it or pay for too much hardware. I'm sure there are many other examples.
- misnome 5y ago
  > All industries have downtime, it needs to be accepted and planned for. And those that can't - e.g. those with lives at risk over failure - have very slow, conservative and burdensome processes for changes and approval, that would never be accepted without the necessary risk.
  - grumple 5y ago
    Indeed, tech moves too quickly for those processes generally. Those who have great processes in tech - and I guess Github is one of those with great processes despite the occasional downtime - pay a lot for the uptime. Critical industries plan for failure. Pilots and astronauts plan for catastrophic failure (and it happens). Hospitals sometimes have fires or run out of oxygen or beds or have equipment failures and have to plan for that. Everybody assumes tech will just keep working in the SaaS world. At the end of the day though, costs of the average service having downtime (even if you calculate google makes 2 million per minute) are generally pretty low and are usually recouped after the service comes back up.
- prepend 5y ago
  > My local taco shop ran out of rice a few weeks ago. A few days ago they had some unspecified kitchen issue and had to close. They don't open on sundays. And mine has never run out of rice. They are closed on Sunday and Monday. I don’t monitor them enough to know how many 9’s, but I’ve probably eaten there 200 times in 9 years and never experienced an outage. Some places suck more than others. But my expectation is 100% uptime when it comes to rice at my taco shop. I think assuming competence is healthy, as long as I’m not a jerk when it happens to fail. I love my taco shop and would survive if they didn’t have an ingredient once. But they work really hard to keep everything working.
bbatha 5y ago
On the other hand, I don't look at a temporary road closure due to maintenance or an accident as an indictment of civil engineering or the highway system.
- jmount 5y ago
  Good point. But we have brought in so many systems, that the analogy might be: we are routinely using the highway system to move content form one room to another in our home.
- nightowl_games 5y ago
  > maintenance or an accident The analogy falls apart when you say 'maintenance'. Maintenance tends to have fallback detour paths, is expected, and isnt a 'deal breaker' like this kind of outage is. This is closer to an 'accident'. And if accidents are frequent at a road location, it _is_ considered an indictment of the civil engineering of that location.
kilroy123 5y ago
Microsoft forcing them to switch to Azure seems to not have helped.
- pm90 5y ago
  Is there any evidence that their issues are due to them using Azure?
- JohnWhigham 5y ago
  GitHub has these ~hour-long outages like twice a year. In the early 2010s it used to be much more frequent.
  - mgkimsal 5y ago
    twice a year? https://www.githubstatus.com/history https://www.githubstatus.com/history There have been multiple incidents longer than an hour just in the last few months this year. It's worse in some ways than, say, 5 years ago, because... GH offers more services. And more people end up relying on those, so, there's more than can go wrong. And when it does, it usually has a bigger impact. Some client projects have tied everything to GH/Actions/CI/deploy/etc. When there's any issue with GH, something is affected. Other clients have more distributed services (bitbucket/gh/gitlab/etc), which is ... sometimes less convenient, but usually means something is still up and functioning even if something else is interrupted.
undecisive 5y ago
The issue here is one of scale, not necessarily intentional complexity - though admittedly one often leads to the other. In a world of single servers - or even multiple servers load balanced over a single database - you only have a single point of failure. And while that is often touted as an terrible thing, once it goes down it's game over... in some ways, our scaled up distributed systems have a worse problem. The issue is we envision our distributed systems optimistically - as isolated little bubbles that receive and emit clean, efficient, fault tolerant packets. When one goes down, only a few others get affected, and the broken module can be easily re-deployed with very little fuss. In reality, the people with the vision quickly become horrified with what is produced. The authentication system touches everything. The stock management system gets hooked directly into the homepage. Suddenly, every single part of your system is hinged on at least 5 servers, and if any of them go down, it's game over until one of them gets brought back up, and by that point the job management queue is so backed up with critical tasks (that should probably have never been entrusted to a background jobs queue) that it takes hours for the organism to re-balance itself. And so, these systems go down more often, because instead of one single point of failure, you have 5 - while telling yourself you have none. The question is, how can we do this better? I've seen big teams and small teams similarly struggle with this. I guess Netflix with their chaos monkey strategy wins on this, but even that is not infallible - as you say, the 1000 layers of abstraction (and goodness knows how many unique combinations of interactions and potential simultaneous failures) something is bound to be missed from time to time. So I don't think the problem is we're moving faster or breaking more things. It's just that we don't know how broken our systems are until they break.
jdavis703 5y ago
I’ve been using GitHub for nearly a decade now. It used to be down all the time. Outages are now comparatively rare.
- phendrenad2 5y ago
  Compared to 2010? Maybe. Compared to 2014? Nope
oliwarner 5y ago
Not ignoring the truth in the other comments or yours, the CI/CD is also in the midst of a battle with mining. While we allow people to turn energy and CPU cycles into cash, people are going to abuse any run-what-you-like service like GitHub Actions.
contingencies 5y ago
Nice one. Added to https://github.com/globalcitizen/taoup https://github.com/globalcitizen/taoup

by 0des 5y ago

thih9 5y ago

Even more offtopic: your comment makes me think of other industries and realise how harmful the heavily processed food must be.

grumple 5y ago

While the web is quite complex now, I think the expectation of 100% uptime for anything is unreasonable. My local taco shop ran out of rice a few weeks ago. A few days ago they had some unspecified kitchen issue and had to close. They don't open on sundays. Occasional downtime is normal. We can try to prevent it, as devs, working with machines, but it is inevitable. Nothing continues working forever. All industries have downtime, it needs to be accepted and planned for. Even in the old days, with the simplest possible thing that needs a db connection, you could outscale that db, or bandwidth, or whatever, and crash. It was easier back then in a lot of ways - 1000 people visit your new site? Crash. Before auto scaling infra you had to either deal with it or pay for too much hardware. I'm sure there are many other examples.

misnome 5y ago
> All industries have downtime, it needs to be accepted and planned for. And those that can't - e.g. those with lives at risk over failure - have very slow, conservative and burdensome processes for changes and approval, that would never be accepted without the necessary risk.
- grumple 5y ago
  Indeed, tech moves too quickly for those processes generally. Those who have great processes in tech - and I guess Github is one of those with great processes despite the occasional downtime - pay a lot for the uptime. Critical industries plan for failure. Pilots and astronauts plan for catastrophic failure (and it happens). Hospitals sometimes have fires or run out of oxygen or beds or have equipment failures and have to plan for that. Everybody assumes tech will just keep working in the SaaS world. At the end of the day though, costs of the average service having downtime (even if you calculate google makes 2 million per minute) are generally pretty low and are usually recouped after the service comes back up.
prepend 5y ago
> My local taco shop ran out of rice a few weeks ago. A few days ago they had some unspecified kitchen issue and had to close. They don't open on sundays. And mine has never run out of rice. They are closed on Sunday and Monday. I don’t monitor them enough to know how many 9’s, but I’ve probably eaten there 200 times in 9 years and never experienced an outage. Some places suck more than others. But my expectation is 100% uptime when it comes to rice at my taco shop. I think assuming competence is healthy, as long as I’m not a jerk when it happens to fail. I love my taco shop and would survive if they didn’t have an ingredient once. But they work really hard to keep everything working.

bbatha 5y ago

On the other hand, I don't look at a temporary road closure due to maintenance or an accident as an indictment of civil engineering or the highway system.

jmount 5y ago
Good point. But we have brought in so many systems, that the analogy might be: we are routinely using the highway system to move content form one room to another in our home.
nightowl_games 5y ago
> maintenance or an accident The analogy falls apart when you say 'maintenance'. Maintenance tends to have fallback detour paths, is expected, and isnt a 'deal breaker' like this kind of outage is. This is closer to an 'accident'. And if accidents are frequent at a road location, it _is_ considered an indictment of the civil engineering of that location.

kilroy123 5y ago

Microsoft forcing them to switch to Azure seems to not have helped.

pm90 5y ago
Is there any evidence that their issues are due to them using Azure?
JohnWhigham 5y ago
GitHub has these ~hour-long outages like twice a year. In the early 2010s it used to be much more frequent.
- mgkimsal 5y ago
  twice a year? https://www.githubstatus.com/history https://www.githubstatus.com/history There have been multiple incidents longer than an hour just in the last few months this year. It's worse in some ways than, say, 5 years ago, because... GH offers more services. And more people end up relying on those, so, there's more than can go wrong. And when it does, it usually has a bigger impact. Some client projects have tied everything to GH/Actions/CI/deploy/etc. When there's any issue with GH, something is affected. Other clients have more distributed services (bitbucket/gh/gitlab/etc), which is ... sometimes less convenient, but usually means something is still up and functioning even if something else is interrupted.

undecisive 5y ago

The issue here is one of scale, not necessarily intentional complexity - though admittedly one often leads to the other. In a world of single servers - or even multiple servers load balanced over a single database - you only have a single point of failure. And while that is often touted as an terrible thing, once it goes down it's game over... in some ways, our scaled up distributed systems have a worse problem. The issue is we envision our distributed systems optimistically - as isolated little bubbles that receive and emit clean, efficient, fault tolerant packets. When one goes down, only a few others get affected, and the broken module can be easily re-deployed with very little fuss. In reality, the people with the vision quickly become horrified with what is produced. The authentication system touches everything. The stock management system gets hooked directly into the homepage. Suddenly, every single part of your system is hinged on at least 5 servers, and if any of them go down, it's game over until one of them gets brought back up, and by that point the job management queue is so backed up with critical tasks (that should probably have never been entrusted to a background jobs queue) that it takes hours for the organism to re-balance itself. And so, these systems go down more often, because instead of one single point of failure, you have 5 - while telling yourself you have none. The question is, how can we do this better? I've seen big teams and small teams similarly struggle with this. I guess Netflix with their chaos monkey strategy wins on this, but even that is not infallible - as you say, the 1000 layers of abstraction (and goodness knows how many unique combinations of interactions and potential simultaneous failures) something is bound to be missed from time to time. So I don't think the problem is we're moving faster or breaking more things. It's just that we don't know how broken our systems are until they break.

jdavis703 5y ago

I’ve been using GitHub for nearly a decade now. It used to be down all the time. Outages are now comparatively rare.

phendrenad2 5y ago
Compared to 2010? Maybe. Compared to 2014? Nope

oliwarner 5y ago

Not ignoring the truth in the other comments or yours, the CI/CD is also in the midst of a battle with mining. While we allow people to turn energy and CPU cycles into cash, people are going to abuse any run-what-you-like service like GitHub Actions.

contingencies 5y ago

Nice one. Added to https://github.com/globalcitizen/taoup https://github.com/globalcitizen/taoup