13 ms·

GitHub Incident

• abluecloud 5y ago
  Github broken. CircleCI broken. I'm going for a coffee.
  • trutannus 5y ago
    Its the snow-day of the tech world.
• relieferator 5y ago
  went to downdetector first but this is betta
• kiloreux 5y ago
  This is getting too much.I wonder what effects this will have on the reputation of Github. If I am blocked by Github every few days and can't properly operate. I will start looking for a more reliable alternative. And I am just a solo dev that pays $4, what happens to companies for whom time is worth a lot of money? Pretty disappointing.
  • chockchocschoir 5y ago
    How do you as a solo developer get blocked on anything related to GitHub? Are you doing all project planning on GitHub without any other copies elsewhere? Most if not everything you do on GitHub could be replicated locally, one way or another. I can understand the problem for larger development teams, as a lot of communication and workflows can happen via GitHub Pull Requests or similar.
    • tyingq 5y ago
      Maybe using their CI/CD?
      • chockchocschoir 5y ago
        CI/CD should you perform tasks that you could execute on any machine, besides the deployment, which should work differently locally VS via CI service. But it should still be possible to deploy without CI, otherwise you set yourself up for being unable to deploy when you really need it.
        • kiloreux 5y ago
          Yeah, but theory and practice are very different things. If CI always worked for me and my experience was great with it. I will not invest time to do independent CI for everything. Hence depending on Github Actions. Clearly not the best approach, but it's about being pragmatic also.
          • chockchocschoir 5y ago
            What do you mean "independent CI"? You start with a `Makefile` with the commands for testing, building and so on, and CI is just calling those commands but on another instances, that likely gets triggered for each commit/tag/pull request/whatever. When you first setup CI, you don't actually just develop for the CI environment first, that's the second step after you know how to perform the same things locally. So not sure why there would be "independent CI for everything".
            • tyingq 5y ago
              It's easy enough to unintentionally box yourself into a hard 3rd party dependency. Like the github secrets store, conditional logic or triggers in the ci.yml files, pipeline driven semantic versioning, etc.
        • DogLover_ 5y ago
          Github-actions is not created to encourage that though. It uses syntax that only works on Github. Look at their workflow-examples and you will see.
    • kiloreux 5y ago
      I am a solo dev, but I work in a company and now have been working on pushing something to test it on Github actions. If it was something different, maybe. But I understand how I might have miscommunicated that.
      • misnome 5y ago
        I'm also in a company, but today am building up something on Github Actions. Every single time I start doing development on Actions, GitHub goes down or actions start failing. I'm not sure if this is saying something about me, or GitHub.
    • bilekas 5y ago
      > Most if not everything you do on GitHub could be replicated locally, one way or another That's not really the point though.. If you have services in github etc, why would you replicate all of them locally too for unplanned downtime ?
      • chockchocschoir 5y ago
        Well, usually you figure out how to run your tests, check code coverage and build your binaries locally first, then figuring out how to get it to work in the CI environment, not the other way around. So the replication has already been done, but in the direction of local environment > remote environment.
        • bilekas 5y ago
          I don't disagree, really local environments should be able to manage dev and most daily tasks, but large integration pipelines, test automations etc. I just don't feel the need to defend GitHub here for the reason of : "Well you should be able to manage locally anyway"
      • hunter2_ 5y ago
        This is quite similar to the somewhat philosophical question of having backups: if your job is to make sure your company has backups of their data, and you entrust a single vendor with that data because they take care of backups so you don't have to, are you really doing your job? Said another way: if you (or your immediate team) are not the party that turns one thing into two things, then there is still a SPOF in your wheelhouse, and the whole point is for that to not be the case.
        • kayodelycaon 5y ago
          Maintaining an entire secondary deployment and CI system just seems unrealistic to me. I'd get so much flack about the extra spend and when the systems get out of sync and causes downtime, that investment ends up with net negative for all involved.
        • 5y ago
    • kayodelycaon 5y ago
      Personally, I can't deploy if GitHub is down. CI pulls from GitHub. One of my projects includes libraries as submodules, with GitHub as the origin. The deployment configuration has everything cloning from GitHub.
    • coldpie 5y ago
      Try building any Rust project while GitHub is down.
      • chockchocschoir 5y ago
        That's only applicable if you use crates directly from GitHub repositories right? I've been sitting and developing+building a Rust project for the last +10 hours (except the minutes when I was here arguing on HN) and never hit any issues, but I'm using crates either directly from disk (cloned from GitHub initially) or from crates.io.
        • coldpie 5y ago
          I could be wrong, but try a fresh build with a clean CARGO_HOME (or maybe just a cargo update?). It seems to query a lot if you don't have a full cache.
          • steveklabnik 5y ago
            While it's true that Cargo will attempt to check on some things, you can pass flags to tell it to not bother. Specifically, https://doc.rust-lang.org/stable/cargo/commands/cargo.html#option-cargo---offline https://doc.rust-lang.org/stable/cargo/commands/cargo.html#o...
• zthrowaway 5y ago
  Anyone think the warnings issued of Russia's cyber attacks has any correlation here?
  • fishnchips 5y ago
    I'd personally correlate it with their Azure migration. Either that, or recent launches. Stability aside, they've been shipping a lot of changes recently. And more often than not, changes lead to outages.
    • JonChesterfield 5y ago
      Seems likely that migrating to Azure is related to being acquired by Microsoft. I blame the new ownership for the decreased stability.
      • pm90 5y ago
        A migration of that scale is difficult to execute on. I wouldn't be so quick to blame the whole Azure platform, unless it really is something in Azure that's responsible for these outages.
    • zthrowaway 5y ago
      Yeah I could see that... with just them going to Azure alone lol. Azure was very unstable at a previous company I was at.
  • dgellow 5y ago
    No
  • time_to_smile 5y ago
    I would expect any serious Russian cyber attack would more closely resemble "polonium tea" in the sense that it would be: incredibly destructive, quite obviously from a state actor, and yet officially not connectable to Russia. Strategically cyber attacks are the most effective when they damage your adversary heavily, and yet at the same time make it so that politically retaliation looks like aggression since there is no unquestionable link between the actor and the action. Consider Stuxnet as an example. Widely understood to be a joint US/Israeli cyber attack on Iran, and at the same time difficult to retaliate against. While annoying, these outages we've been seeing are hardly doing any serious harm. Very little money is ultimately lost because of them, and it's not remotely clear that this is a cyber attack. So if they were cyber attacks it wouldn't make much sense. Not saying that it's impossible they are, but it seems there are many other more plausible explainations.
• eutropia 5y ago
  As someone who is responsible for CI/CD... welp. I'm thankful that at least I'm not on security investigating the severity of the okta hack.
• 5e92cb50239222b 5y ago
  This has been happening pretty much every day for a long time. Maybe this will push more people towards some decentralization. A self-hosted Gitea instance my organization has been using had exactly zero downtime in the past year. None. Same with our CI system. Yes, I know, GitHub is 10⁹⁹⁹ times larger than our puny Gitea instance, and that's why they're having issues, but why should I care when one is working, and the other one isn't?
  • chockchocschoir 5y ago
    It's not about being larger, it's about doing changes. GitHub used to be pretty stable before, when they didn't really do any (user facing) changes. It stood still in terms of features, but was also pretty stable overall. Compared to GitHub, how often does your Gitea instance go through code changes? How many major features have been switched on in that instance during the past year without any hiccups?
    • freedomben 5y ago
      I tried to look at the Gitea repo to see how fast they get commits, but github is 500ing :-( https://github.com/go-gitea/gitea https://github.com/go-gitea/gitea
      • chockchocschoir 5y ago
        Rest assure, a open source project with ~100 contributors (whereas maybe 10% regularly contributes) has less activity and changes than a multi-international company focusing a SaaS with "1000 to 5000" employees worldwide.
        • judge2020 5y ago
          For perspective, GitHub has 4,000 employees who list them as their employer on LinkedIn and GitLab has 1,700.
    • Deukhoofd 5y ago
• uwuemu 5y ago
• ksnll 5y ago
  Cannot really rely on Github lately. Wondering if they have some sort of post mortem for the incidents of last month. I would like to see why we should trust it to not go down again.
  • chockchocschoir 5y ago
    > I would like to see why we should trust it to not go down again. You shouldn't ever trust anything to never go down again. It's just inevitable that services go down from time to time, you can't change that unless we have a major breakthrough in software engineering where somehow people stop making mistakes. So instead of figuring out if you should "trust it to go down again or not", assume it will at one point in the future and plan accordingly. This is how you build resilient systems.
• ahmadrosid 5y ago
  Today, I saw a beta timeline feature. Could it be this incident related to this release?
• rvz 5y ago
  From [0] > Until the next time GitHub goes down again (hopefully that won't be in another month's time). Oh dear. [0] Never expected it to be this soon or that bad. Expected it to be down in a few weeks at least. Last time this happened was just 5 days ago. [1] Yet I see complaints here of GitHub being unreliable as predicted and even some considering to eventually self-host despite paying $4 a month even. I don't think it make sense to go 'all in' on GitHub, does it? [0] https://news.ycombinator.com/item?id=30714123 https://news.ycombinator.com/item?id=30714123 [1] https://news.ycombinator.com/item?id=30711442 https://news.ycombinator.com/item?id=30711442
• duckmasterflex 5y ago
  My boss threw his hands up and said we a switching to gitlab. I love github, why you doing this to me Microsoft!
• judge2020 5y ago
  It seems the pre-MS GitHub's approach of "we have maybe 1 new feature a year in the pursuit of keeping our duct-taped service afloat" might have been the right call. Nowadays tons of new features are pushed out every week[0]. 0: https://github.blog/changelog https://github.blog/changelog
  • darknavi 5y ago
    I get what you're saying but I cracked up at this "GitHub is releasing too many features and breaking things!" because when I clicked your link the latest entry is: > Diffing Adobe Photoshop documents is no longer supported
    • raxxorrax 5y ago
      How can a system be stable if you don't at least scrap two features a year?
• EMIRELADERO 5y ago
  It would be nice if GitHub started operating the way GitLab does. Free community edition (open-source) and paid proprietary Enterprise Edition (source-available). Developement being completely public (employees push their commits, do code review and work on issue boards on the official GitLab repo[1] which is publicly available) [1] https://gitlab.com/gitlab-org/gitlab https://gitlab.com/gitlab-org/gitlab
• mbesto 5y ago
  Can we please stop posting major system incidents and status pages for major cloud/SaaS providers? What's the point? Post mortems - TOTALLY fine. But just linking to a specific incident is IMHO not valuable to HN and just takes unwarranted frontpage space.
  • disgruntledphd2 5y ago
    I think it's so people realise it's not just them.
    • mbesto 5y ago
      Makes no sense. If you're having issues wouldn't you just go directly to <insert service here>'s status page first? Why HN first?
      • 0x0000000 5y ago
        Because status pages lie.
      • astrosi 5y ago
        In many cases the status page of a service is an unreliable indicator (e.g. AWS's) so I find HN / Twitter to be a much better signal that an incident that affects many people is happening.
      • crent 5y ago
        Status pages are just not reliable unfortunately.
  • aerovistae 5y ago
    I find this valuable. -Sometimes for conversations about alternatives -Sometimes for conversations about the product's stability history -Sometimes just for solidarity in frustration
    • mbesto 5y ago
• FpUser 5y ago
  I just got bunch of errors - error: failed to push some refs to 'github.com'. Luckily after a while the error disappeared. I was going nuts. For my own products I use gitea and host repos myself. Some clients of mine however use github so in this case I have no choice
• hirako2000 5y ago
  'Be what's next', or how to ruin the brand of a 7.5B acquisition.
  • colinmhayes 5y ago
    Microsoft has big plans for github. Maybe the brand is a bit tarnished by the time they realize those plans, but if they succeed in fully integrating the developer productivity stack into both the github app and site they'll see a very large amount of revenue.
    • JonChesterfield 5y ago
      What GitHub app? edit: turns out to be a flavour of vendor lock in, https://docs.github.com/en/developers/apps/getting-started-with-apps/about-apps https://docs.github.com/en/developers/apps/getting-started-w...
      • colinmhayes 5y ago
        Visual studio will be rebranded as the github app.
• lolspace 5y ago
  Thanks! Just had problem
• arnonejoe 5y ago
  can't open a PR right now.
• thadjo 5y ago
  pretty wild how these outages are basically just tolerated now. folks just get some coffee refresh HN a dozen times and then resume work when it comes back online. no huge uproar, no exodus from GH
  • dogleash 5y ago
    It was a foreseeable inevitability that once SaaS took over from on-prem that the quality would degrade to what we were getting from on-prem. ...but if you said anything you were just interrupting people playing with their new toys. The shrug whatever coffee break while another department fixes things is the same now as it was back then.
  • corobo 5y ago
    Everyone with projects small enough to kneejerk move around migrated to GitLab when Microsoft bought GitHub
• higeorge13 5y ago
  Instead of releasing unnecessary social-like stuff they should have focused on their core git service. Seriously, what’s their uptime? Do they even have a number somewhere? Actions must be down every second day or so.
• intunderflow 5y ago
  With all the recent incidents have they broken their SLAs yet?
  • ProAm 5y ago
    A SAAS company never breaks an SLA. SLA's are written by the legal team. Uptime is maintained by the engineering team. They aren't even in the same building.
    • AlexandrB 5y ago
      Yup. After all this isn't downtime, its "degraded performance": > Git Operations is experiencing degraded performance. We are continuing to investigate.
• bob1029 5y ago
  Ah the semi-daily github fuckup. For this one, I found myself rage-screaming at my monitor as their issue markdown preview API was timing out ~80% of the time. Github enterprise is probably happening for us this year. I don't give a shit about the new features in the public build. I just want it stable and working at this point.
• grumple 5y ago
  Was wondering about multi-remote git workflow and found this: https://jigarius.com/blog/multiple-git-remote-repositories https://jigarius.com/blog/multiple-git-remote-repositories This gets tougher with CI/CD though. How urgent is it to run your tests or deploy right now as opposed to when the outage is resolved later, or tomorrow? In emergencies I think you need ways to deploy without the CD pipeline (for those that can't currently). In my experience there's always more work to do and downtime is a temporary inconvenience. Of course... my customers would be blowing up our phones and emails if we were down, threatening to quit, etc. I assume Github doesn't have quite the same level of pain when they have downtime.
• teaearlgraycold 5y ago
  Has anyone made a good SaaS product for self-hosted Github Actions runners? I'd gladly pay for beefier VMs from a 3rd party provider. Actually self-hosting is a pain in the ass because Github gives you only the runner software but none of the containerization/stateless wrapping I'd expect. Without some containers and scripting you'll get permanent changes to your VM's filesytem as you run actions. And the VMs GitHub provides are far too low spec to run Cypress in. There's no option to pay for more resources.
• yuuta 5y ago
  Microsoft SRE always sucks. GitHub is just another victim.
• tokumei 5y ago
  It seems incidents like this with the big providers are becoming a lot more common. Wonder why that is
• nrvn 5y ago
  Github is a platform that comes with all platform-related risks, i.e. operational risk of having the whole development and SDLC lifecycle blocked, creating a vendor lock due to using some unique github features, inability to rely on third-party dependencies hosted on github. There are a couple of strategies one could rely on to mitigate the risk of github downtime: - create a map of features between github and competitors, find the common denominator and risk accept anything that is unique to github but viral for your business/product; - maintain a primary self hosted instance of gitea/gitlab/sourcehut and use github as a mirror; - use github as a primary platform, maintain mirror with public gitlab and switch to gitlab in case of outage; - if github is used beyond its primary purpose(git hosting) put some efforts to maintain the same features in gitlab/gitea/sourcehut, i.e. use CI features of both platoforms and push releases and artifacts to both of them so that end users have the choice. - separate concerns and not go all in with github as a platform for the whole SDLC and project management and instead use different tools/platforms for different purposes. Yes, it takes more time for the initial setup but it quickly pays back. The above applies to those cases where your products need to be hosted publicly.
• KronisLV 5y ago
  Someone who decided to self-host their own source control system here, figured i'd comment about some of my experience due to some people mentioning it as an alternative. I've personally run GitLab in my own homelab for a few years but after some updates going wrong, i eventually moved over to Gitea, Nexus and Drone, also self-hosted: https://blog.kronis.dev/articles/goodbye-gitlab-hello-gitea-nexus-and-drone https://blog.kronis.dev/articles/goodbye-gitlab-hello-gitea-... GitLab is a large and complicated piece of software, which is a double edged sword: you do get an integrated issue tracking solution, Docker Registry with integrated access controls, as well as a pretty decent CI solution, which is great. At the same time, however, there's a chance for one of those subsystems to start misbehaving and bring down the whole thing (e.g. Omnibus install with default configuration for where to store container images --> running out of space; especially relevant if you don't find a good way to clean up old images), which may or may not be harder to solve than having 3 separate integrated systems. Also, GitLab is pretty resource hungry and migrating to another solution actually improved the performance of certain systems/parts of functionality bunches: Gitea is really fast, Drone CI is about on par with GitLab CI and Nexus is a bit faster than GitLab Registry, however also eats too much RAM (e.g. the same problem within both Ruby and Java, Nexus seems to fail to start up below having approx. 1 GB of RAM, GitLab needed at least 2 - 3 GB of RAM). However, running GitLab at work is still a pretty good idea, as long as keeping the instance up and running, as well as updated is someone else's responsibility and you can focus on just being the end user. At the same time, however, the update paths can be problematic sometimes, as was the case with my container based Omnibus installation that was multiple major versions out of date. But for personal use? The Gitea, Nexus and Drone setup is perfectly sufficient, alongside maybe some issue tracking software (be it Kanboard, OpenProject or something else). In short, to the folks who are considering self-hosting in the comments: you can definitely try it out and have it be a learning experience, maybe on an affordable VPS or a device you have laying around! Furthermore, there's nothing really preventing you from mirroring your repos over and keeping them in multiple systems, alongside any backups that you might be making!
• AtNightWeCode 5y ago
  What surprises me with many of the outages from big companies is that it seems like they have a bad setup when it comes to zone redundancy. Fact is many of the smaller companies I worked with are much better at this. Facebook, Fastly, Github. How can companies like that fail at such basic concepts? Really mind boggling. There must be something fundamentally flawed in their development process.
  • benmanns 5y ago
    Perhaps they've all gotten big enough to run their own DNS?