3 ms·

I know this a greybeard's fantasy and that most people working today were trained not to bother, but: important things should not have GitHub as a failure point

I know this a greybeard's fantasy and that most people working today were trained not to bother, but: important things should not have GitHub as a failure point. Hobby projects and today's work? Sure. Point straight at GitHub and hack away. And when it goes down, get yourself a coffee. But everything that's anywhere near production should have already pointed those github requests to a mirror or other tool in your own controlled ecosystem. The status of GitHub should have nothing to do with whether you're customers are getting what they're paying you for. Same goes for Docker containers and every other kind of remotely distributed dependency.

willsmith72 3y ago
It's not that black and white. Where do you draw the line on what can and can't be a failure point? My cloud provider is probably an acceptable point. If every AWS region goes down I'm not going to have a spare cloud provider. What about an auth provider? Do I need a backup there? What about CI, do I need multiple CI systems? 3rd party search services, realtime messaging services, the list goes on. For 1% of systems, you need backups for all of these (or to not use anything external). The other 99%, building backups for every one of these systems is a losing business strategy. Some of them sure, but which those are will vary based on the context. It's not as simple as having a backup for "every other kind of remotely distributed dependency."
- GrinningFool 3y ago
  That's all besides the point isn't it? Everything you mentioned can be replaced. Your source code can't.
  - wongarsu 3y ago
    It depends on the Recovery Time Objective we are talking about. For example at our company there are daily backups of both our dependencies and those used in the build process. If github had a prolonged outage or accidentally deleted all their data we could set up a gitea with all our code within hours, and get a replacement for Github Actions working within days. But that takes longer than the expected duration of this outage, and is a lot of work. It's not like we have a standby gitea we can just seamlessly switch to, so we are still hit by this outage. On the other hand for build dependencies we do have a standby mirror.
- Animats 3y ago
  > What about an auth provider? Do I need a backup there? Relying on Google for that would probably be a mistake. With cancelled projects and cancelled users, that's high risk.
- callalex 3y ago
  I draw that line at around 99.9% availability.
  - BHSPitMonkey 3y ago
    Availability is a measured outcome. How's it gonna help you quantify your various risks _before_ they become a problem so you can spend your mitigation time wisely?
    - callalex 3y ago
      I was too brief, I meant I would add redundancy reactively where there was none there before if a service I’m using can’t consistently keep 4 9’s. You are correct that being proactive is more complicated.
daxaxelrod 3y ago
Interesting perspective, disagree on one central tenant. Even though Github holds production code, it is not production. Built artifacts and the machines running those artifacts is production. When Github goes down, which it rarely does, it just means developers cant sync for a couple hours, no different than if someone works offline. The temptation to increase internal devops complexity should not be an automatic immune response when a service goes down, it comes with all sorts of hidden costs.
andrewaylett 3y ago
It's not entirely clear to me whether you're talking about using GitHub for your own production tooling, or as a source for some arbitraty third party component. If it's the latter, then I completely agree with you. Use a read-through proxying package repository. I don't care if you run it yourself or if you pay a provider, but don't pull stuff from the origin every time you build. In the general case, adoping an external system will bring with it greater reliability than trying to run stuff oneself. The differences are that you don't get to choose your maintenance windows, and you can't do anything to fix it yourself. Take care about who you pick, and own the depencency, because you've put a part of your own reputation in the hands of your provider. Now, if you pick GitHub as a part of your controlled ecosystem -- which is totally reasonable, if it fits your use-case -- then you still shouldn't be pulling arbitrary stuff from places outwith your control. GitHub has package repository tooling that you can use :). Although it's not entirely clear to me that it's as suitable for third-party dependencies as tools like Artifactory or Nexus.
- swatcoder 3y ago
  Yeah, I was talking about access to third-party dependencies, which I had assumed was the crisis the OP had in mind. I've since read many sibling replies fret over build automation or source code storage, but some downtime in build automation will rarely damage customers and (my goodness) I hope people aren't trusting the only copy of all their IP to a business partner and that they always have a very recent backup somewhere they own themselves.
  - andrewaylett 3y ago
    Some things are disappointingly hard to back up properly. Source code isn't really in that category. There's still a question of how strong an ownership we need -- for example, my mail server is owned by OVH, and its backups are sent to rsync.net. I consider that to be sufficent ownership of backups. I'm much more likely to lose my files than they are. I'm sure that GitHub also have backups, but they're not for the benefit of their customers.

willsmith72 3y ago

It's not that black and white. Where do you draw the line on what can and can't be a failure point? My cloud provider is probably an acceptable point. If every AWS region goes down I'm not going to have a spare cloud provider. What about an auth provider? Do I need a backup there? What about CI, do I need multiple CI systems? 3rd party search services, realtime messaging services, the list goes on. For 1% of systems, you need backups for all of these (or to not use anything external). The other 99%, building backups for every one of these systems is a losing business strategy. Some of them sure, but which those are will vary based on the context. It's not as simple as having a backup for "every other kind of remotely distributed dependency."

GrinningFool 3y ago
That's all besides the point isn't it? Everything you mentioned can be replaced. Your source code can't.
- wongarsu 3y ago
  It depends on the Recovery Time Objective we are talking about. For example at our company there are daily backups of both our dependencies and those used in the build process. If github had a prolonged outage or accidentally deleted all their data we could set up a gitea with all our code within hours, and get a replacement for Github Actions working within days. But that takes longer than the expected duration of this outage, and is a lot of work. It's not like we have a standby gitea we can just seamlessly switch to, so we are still hit by this outage. On the other hand for build dependencies we do have a standby mirror.
Animats 3y ago
> What about an auth provider? Do I need a backup there? Relying on Google for that would probably be a mistake. With cancelled projects and cancelled users, that's high risk.
callalex 3y ago
I draw that line at around 99.9% availability.
- BHSPitMonkey 3y ago
  Availability is a measured outcome. How's it gonna help you quantify your various risks _before_ they become a problem so you can spend your mitigation time wisely?
  - callalex 3y ago
    I was too brief, I meant I would add redundancy reactively where there was none there before if a service I’m using can’t consistently keep 4 9’s. You are correct that being proactive is more complicated.

daxaxelrod 3y ago

Interesting perspective, disagree on one central tenant. Even though Github holds production code, it is not production. Built artifacts and the machines running those artifacts is production. When Github goes down, which it rarely does, it just means developers cant sync for a couple hours, no different than if someone works offline. The temptation to increase internal devops complexity should not be an automatic immune response when a service goes down, it comes with all sorts of hidden costs.

andrewaylett 3y ago

It's not entirely clear to me whether you're talking about using GitHub for your own production tooling, or as a source for some arbitraty third party component. If it's the latter, then I completely agree with you. Use a read-through proxying package repository. I don't care if you run it yourself or if you pay a provider, but don't pull stuff from the origin every time you build. In the general case, adoping an external system will bring with it greater reliability than trying to run stuff oneself. The differences are that you don't get to choose your maintenance windows, and you can't do anything to fix it yourself. Take care about who you pick, and own the depencency, because you've put a part of your own reputation in the hands of your provider. Now, if you pick GitHub as a part of your controlled ecosystem -- which is totally reasonable, if it fits your use-case -- then you still shouldn't be pulling arbitrary stuff from places outwith your control. GitHub has package repository tooling that you can use :). Although it's not entirely clear to me that it's as suitable for third-party dependencies as tools like Artifactory or Nexus.

swatcoder 3y ago
Yeah, I was talking about access to third-party dependencies, which I had assumed was the crisis the OP had in mind. I've since read many sibling replies fret over build automation or source code storage, but some downtime in build automation will rarely damage customers and (my goodness) I hope people aren't trusting the only copy of all their IP to a business partner and that they always have a very recent backup somewhere they own themselves.
- andrewaylett 3y ago
  Some things are disappointingly hard to back up properly. Source code isn't really in that category. There's still a question of how strong an ownership we need -- for example, my mail server is owned by OVH, and its backups are sent to rsync.net. I consider that to be sufficent ownership of backups. I'm much more likely to lose my files than they are. I'm sure that GitHub also have backups, but they're not for the benefit of their customers.