8 ms·

What do people do to get around this? Run my own git server like the good 'ol days? Github has become a central source of failure for us now...

• H1Supreme 9y ago
  Gitlab?
  • Robin_H 9y ago
    Does anyone know an alternative to githost.io? We don't want to use gitlab.org as it's painfully slow but also don't want to manage our own gitlab instance. edit: We don't like the recent changes made in githost. ( Replaced the $35 option with $149 ) We're a team of 15 people. edit2: should have clarified; I'm hooked to GitLab, just looking for another service that does managed hosting. Not looking for GitLab alternatives.
    • ShakataGaNai 9y ago
      Gitlab.com? BitBucket is the other major player that comes to mind, but googling "hosting git" returns enough results to melt ones mind. If you're really concerned, you run it yourself. GitLab insider a docker container isn't hard to manage. You can even 1-click Gitlab onto a DO Droplet in just a few minutes: https://www.digitalocean.com/community/tutorials/how-to-use-the-gitlab-one-click-install-image-to-manage-git-repositories https://www.digitalocean.com/community/tutorials/how-to-use-...
    • m_sahaf 9y ago
      Gitea[0] is extremely simple to host, if you want a self-hosted solution that's not as heavy as gitlab. [0] https://gitea.io https://gitea.io
      • kuschku 9y ago
        Gitea sadly has some issues (for example, the "Public Activity" tab on user profiles 500's atm: https://try.gitea.io/crackedmind?tab=activity; https://try.gitea.io/crackedmind?tab=activity; it throws 500's everywhere once you throw on a modern Xeon more than 30 users/second at it, because it doesn't cache database accesses, at all, etc).
        • m_sahaf 9y ago
          Thanks for mentioning that. It seems like this bug wasn't reported yet, so I have filed a bug report.
          • kuschku 9y ago
            Also, I don’t seem to be able to properly run it in docker (I’m trying to deploy it on kubernetes), do you, by random chance, have experience with deploying gitea?
            • m_sahaf 9y ago
              Sorry, I don't know about running it in docker. My usecase is personal lightweight repos management, so I just download the binary and run it backed by SQLite. Never faced any issues with that. I never needed to go fancy with docker.
    • gfosco 9y ago
      Consider running your own Gitlab instance.. I run one for $40/Mo on DigitalOcean, very happy with it.
• DougBTX 9y ago
  Samba surely, wanna cry about having to set up file:// remotes?
• batbomb 9y ago
  Work locally? It's Git.
  • keithnz 9y ago
    sure, but if your Build System / CI is based around github, then you are a bit poked. thank goodness I'm on bitbucket :) One of the things they offer is your own local server synched with their cloud servers for redundancy.
    • trustfundbaby 9y ago
      > sure, but if your Build System / CI is based around github, then you are a bit poked. then architect your build system be able to switch to another service with the flip of a switch, if its that important to you.
      • flukus 9y ago
        Github has added all sorts of proprietary integration points to prevent it from being this easy, like push based checkin notifications. Then developers get lure by the drag'n'drop CI servers that integrate with them. Unfortunately people only care about API's now instead of standards.
• ben174 9y ago
  With each GitHub failure people learn some good lessons. Redundancy is the key. Push to multiple remotes and fall back to backup remotes in your scripts.
• pcmonk 9y ago
  If everyone has SSH access to a server somewhere, it's easy to just put a git repo there to let everyone keep working. Obviously, this doesn't trivially scale to many repos/users/etc, which is why Github exists in the first place.
  • Silhouette 9y ago
    It's obviously possible to manage very large projects stored in Git repos with no dependency on a site like GitHub at all. Just look at what Git was designed for, after all. However, I'd say one of the biggest practical problems with most of the popular DVCS tools is still that they don't also have a simple, out-of-the-box way to set up a centralised repo. As you say, using Git+SSH works up to a point, but it's still irritating and somewhat time-consuming to set up if you work with a lot of projects that might each have different contributors. Last time I checked, none of the main alternatives was any better.
    • vacri 9y ago
      It depends on what your deployment system is; it's not always trivial to just change the target git origin to a new server. For someone who knows what they're doing and has the time to focus on doing it, it won't be too hard, but Murphy's Law gets in the way.
      • Silhouette 9y ago
        It depends on what your deployment system is; it's not always trivial to just change the target git origin to a new server. This seems to be a fundamental weakness in whatever deployment system is used, then. Relying on a single point of failure outside of your organisation's own control for anything critical is always risky, and in this case it seems to be entirely avoidable.
• jonaf 9y ago
  Your question seems to indicate that you have a runtime dependency on github. If true, this is a problem no matter where your git repo resides. You need to architect a solution wherein you do not have a runtime dependency, especially if it is a single point of failure, or accept that your reliability will never be better than the sum of your services' reli abilities (there was an article on the "calculus of availability" or something like that recently). What I personally do in AWS is bake my artifacts or other git-sourced data into AMI's. If you want a middle ground, you can instead push your artifacts to an s3 target -- s3 has better reliability / track record than github for this purpose.
  • avaer 9y ago
    Thing is, runtime dependencies on Github is something that Github is promoting and selling now (https://github.com/marketplace/ https://github.com/marketplace/). Maybe this incident puts that value proposition in a more visceral context.
    • aisofteng 9y ago
      Just because they sell it doesn't mean people shouldn't know better.
    • paxy 9y ago
      How is the GitHub marketplace a runtime dependency? Code review, project management, build bots or whatever else should NOT affect the availability of your website.
      • deleted 9y ago
        [deleted]
      • avaer 9y ago
        Sure, if you measure uptime of your business and dev team by whether you're serving pages. I wouldn't. If Github being down means you're not fixing your site today, I'd call it a runtime dependency.
• deleted 9y ago
  [deleted]
• deleted 9y ago
  [deleted]
• btown 9y ago
  Others will say "work locally," but the problem is more in deployment. There's still a lot of app and CI servers out there where the workflow is "push to Github, then some automated script will tell the cluster to pull from Github" rather than "push the changes directly to the cluster." Containerization, especially projects like Kubernetes, make the latter much more viable; your cluster handles its own orchestration and deployment. It's still early days, though, and there's a lot that a new user can get wrong in setting such a system up. Ansible telling servers to git pull from Github may give you better uptime once you factor in your own mistakes.
  • adekok 9y ago
    I agree. Github is an off-site backup, and a nice interface for pull requests and code browsing. There's just no reason to make github a critical part of your infrastructure.
  • twistedpair 9y ago
    > then some automated script will tell the cluster to pull from Github GitHub isn't a deployment tool. It's a source control tool. Keep your deployment artifacts somewhere else (e.g. S3/GCS) more durable so you can still deploy/rollback during a GH outage.
  • stevegore 9y ago
    Could you elaborate on how Kubernetes makes it easier to deploy code changes directly to the cluster? Are you talking about the Docker/container image build?
• fishtoaster 9y ago
  A common solution is to decide that github is down rarely enough that this isn't a problem. Eg if your only dependency on github is at deploy time, it might be ok if you can't deploy for a few hours a year when github is down. If you have a bigger (eg runtime) dependency or if being unable to deploy for a few hours is a problem, then yes– you run your own git server.
  • gtirloni 9y ago
    Exactly. There are no right/wrong or absolutes. It's a trade-off.. if it makes sense in someone's situation, it makes sense.
  • twistedpair 9y ago
    > github is down rarely enough Our stats show a 1:239K request error rate for S3, and a near 100% availability. There has only been one outage of note in years. GH on the other hand is down with shocking frequency like today and yesterday, though not as much as BitBucket. Some companies I know write to GCS and S3 in parallel, which is easy given many tools and API's work directly with both (e.g. AWSCLI/gsutil).
• cheeze 9y ago
  AWS CodeCommit has worked pretty well for me. I generally trust AWS/GCP/Azure over others regarding availability.
• __jal 9y ago
  You need to either ensure you control the resources you depend on, or plan for outages. At my gig, we use GH Enterprise (on-prem) as well as some smaller plain-vanilla git repos (mainly for CM and other critical services). Any third-party code and all other dependencies live in an artifact manager or private mirror repos. Personally, I don't use any cloud provider to store my data. Cliche, but true: if you don't control the hardware, you don't control the data on it.
• omarforgotpwd 9y ago
  just set up a server via webhook that pulls the repo everytime someone pushes to it. When GitHub goes down you can use that server.
• zbobet2012 9y ago
  Most items of ours that have a production dependency on GitHub actually point at a simpler (and much stupider) mirror. When commits are made pushes are mirrored to those repos. These repos can even be naively replicated since git's nature allows for this. If you _need_ Github to be up, mirror it. If your build or deployment tools _need_ Github to be up, and you _need_ them to be always available make simple dumb mirrors. Also make sure your tools don't lock on only one item.
• ransom1538 9y ago
  At this point Github is a national security risk. The amount of passwords, keys, hashes, code, data uploaded to that system is massive. I am sure that government agencies, contractors, multinational defense departments, etc are dumping things on there hourly. Will it get hacked? Get your popcorn out. It will happen.
  • ams6110 9y ago
    Is GitHub even allowed for govt contractor use? I'd be surprised, but don't know.
    • WorldMaker 9y ago
      For some projects, yes. The biggest well known US examples are the 18F open source efforts and http://code.gov http://code.gov / http://code.mil http://code.mil open source efforts, most of which are public, open source on GitHub.
  • CapnCrunchie 9y ago
    I'm pretty sure most, if not all, government agencies require on-premises hosting of code
    • bpicolo 9y ago
      There's e.g. https://aws.amazon.com/govcloud-us/details/ https://aws.amazon.com/govcloud-us/details/ these days. Azure and Google have government offerings as well. Safe to say neither would be bothering if the government didn't use cloud products these days. http://fortune.com/2016/09/02/us-government-embraces-cloud/ http://fortune.com/2016/09/02/us-government-embraces-cloud/ Looks like even the CIA is using some AWS. "U.S. government agencies are moving to cloud computing and away from their own data centers faster than private corporations"
• jensvdh 9y ago
  Github Enterprise
• shiftoutbox 9y ago
  Use fossil --> https://www.fossil-scm.org/index.html/doc/trunk/www/index.wiki https://www.fossil-scm.org/index.html/doc/trunk/www/index.wi...
• vmware513 9y ago
  It was in lunchtime in New Zealand, so doesnt really matter. ;) I am just back with our world famous wellingtonian flat white, and looks everything perfect. :)
• nickpsecurity 9y ago
  Reinvent VMS-style clustering on BSD/Linux boxes or just buy some VMS servers. They measured uptime in years to decades for tech that started in the 80's. There should be several, modern solutions like that given companies were developing them a long time ago when I first looked into such things. I just haven't surveyed to see what exists, is reliable, and is affordable. They could also have a base version of the site that's read-only for source or binaries to allow customers access to the data to fall back to whenever problems happen in more complex functions. Keep it running in some form while they fix things. They might also similarly sell Git appliances w/ high-availability that themselves plug into multiple colo's or cloud providers mirroring changes in the repo to the Github site. Just a wild idea as I have no idea if that's marketable.
• nathanvanfleet 9y ago
  You can run a local copy on a server of something like Github. For example BitBucket Server (which used to be called Stash). I worked somewhere that was on GitHub but cried when it was down, then went to BitBucket and cried when it was down. It's a big point of failure.
• swah 9y ago
  Sublime Text also is depending a lot on github.com these days with Package Manager...
• wyqydsyq 9y ago
  Use Docker. When you build a docker image, you copy all your runtime dependencies (`node_modules/`) and build artifacts into the image. When you push a deployment you are deploying that static image with the dependencies baked in, instead of trying to install the dependencies at startup.
• jimwalsh 9y ago
  Self hosted bitbucket and the like really. Or AWS CodeCommit. We are currently using both of those in our build pipelines.
• gumby 9y ago
  > Run my own git server like the good 'ol days? Isn't every git client a server? Since git is peer-peer, when GitHub is inaccessible we just merge back and forth (or just commit locally -- it's never been down for days).