7 ms·

Tangential at best but I work in a regulated industry and we're starting to get some heavy scrutiny from external auditors and regulators on the way we plan to

• dangero 2y ago
  That's a lot of overhead. Do you know how they are calculating the possible risk of these events? It feels like there are a million rabbit holes like this you could go down when modern infrastructure is so cloud connected. Is the risk of your git repos higher than a chip shortage causing you to lose access to the infrastructure you need? So many factors to consider. A chip shortage doesn't seem that unlikely with geopolitics. The list of scenarios you mitigate for seem like they could very easily be an arbitrary list of the of scenarios a single person came up with.
  • jcims 2y ago
    It's an astonishing amount of overhead that slows everything to a veritable crawl. It also creates downstream issues because you need to build in contingencies for your service consumers until you are able to fully abstract the dependency to a point where rehoming doesn't impact them. The risk calculations are very primitive at the moment, I'm guessing they will be refined over time as industry feedback starts to resonate.
• cheeze 2y ago
  And the ultimate problem that you don't need a solution for, until you _need_ it. Which often leads to "yeah we have backups" but without regular testing, can you trust them?
• parpfish 2y ago
  but if you self host, how do you anticipate power outages and natural disasters? you shift it from a problem of software reliability to a problem of physical infrastructure. at some point in the chain somebody has to do that, but i'd prefer the person doing that was somebody with DEEP experience in it that could give you some nice confident assurances.
  • 0xbadcafebee 2y ago
    Basically you rent space and machines in datacenters that are geographically distributed and designed to resist natural disasters
    • Gigachad 2y ago
      That's exactly what Github does already. I wouldn't bet on my own org being better at hosting github than github.
      • mrweasel 2y ago
        Perhaps not better at hosting Github, but some sort of code repository. We've been running on-prem Bitbucket for years and our uptime is easily better than Githubs. The feature set is small, and the CI/CD pipelines are a separate issue, but I still think we come out on top. Depending on your size your requirements may be much lower and easier to manage. Github has to be all things to all people and that comes with complexity and that can make things more fragile.
      • 0xbadcafebee 2y ago
        Agreed. I have worked for several companies who tried to self-host their VCS etc, and their uptime was way worse, the overall cost was higher, and there were less features. Having a backup is always a good idea though, and an emergency plan for how to onboard to a competitor.
  • jcims 2y ago
    Most of the companies exposed to this are already going to be running a number of geographically diverse datacenters and have been exposed to regulations around basic DR/BCP for decades. >at some point in the chain somebody has to do that, but i'd prefer the person doing that was somebody with DEEP experience in it that could give you some nice confident assurances. Yes. This is part of the reason why services like git are being moved outside the datacenter. Most of the product offerings on the market don't scale well, have terrible reliability and are still very expensive to run.
• 0xbadcafebee 2y ago
  First pay a consultancy a fortune to tell you which SaaS you are able to use based on compliance requirements. Then run your own IdP, use service providers that have a data takeout mechanism, take regular backups, and use standards-based technology. Line up a self-hosted fallback for each non-self-hosted option for if you're forced to exit them. Basically you line up the exit strategy for the auditors but hope you never have to use it.
• macintux 2y ago
  The nice thing about git, from my perspective, is that if your entire hosted service vanishes, you can still reconstruct what you need from your users’ working directories. All of the important branches should be there. Somewhere. And any important integration branches that aren’t cached can be reconstructed. Of all the many dependencies on cloud services, git is by far the last I’d worry overly much about.
  • jcims 2y ago
    I think my point is we are losing the option to worry about it or not. There needs to be an answer with a runbook to restore services within a given window of time should we lose that vendor.
  • dangoldin 2y ago
    Problem is that often you also end up relying on GitHub for CI/CD so not as easy of a change. Imagine GH being down and you need to deploy a hotfix. How do you handle that? Especially, if you followed best practices and set up a system where all PRs need to go through code review.
    • hamandcheese 2y ago
      Systems like these should have an escape hatch of some sort. The key part is that it needs to be auditable. Anything you do in CI should be possible outside of CI, at least by some subset of users.
      • dangoldin 2y ago
        Yea - definitely. Just not ideal and something that needs to be built out, tested, etc.
        • hamandcheese 2y ago
          Yes, it is easier said than done. At my company we use buildkite and many people wrote scripts that simply fail outside of buildkite. GitHub actions is even worse, it seems like it was designed from the ground up to create lock in. Nix helps a bit on the bootstrapping and dependency management problem, but won't save you from writing a script that is too tightly coupled to its runtime environment.
• st3fan 2y ago
  GitHub not being available for 45 minutes is not an "extended outage".
  • jcims 2y ago
    Agreed, that's why i said 'tangential at best'.
• Over2Chars 2y ago
  I don't see how self-hosting solves the problem of 3rd party vendors unless you're standing up a self-hosting solution as a hot/cold backup to your 3rd party vendor "in the event of an extended outage". Kinda eliminates all those pennies saved (in theory) for outsourcing to "the cloud" if you have to duplicate your infra. Hybrid has always seemed the most optimal approach, but there's always someone in charge who thinks spending money on safety nets is just wasted money.
  • theteapot 2y ago
    Doesn't it exactly solve "extended outages and forced exits from third party vendors."?
    • Over2Chars 2y ago
      Self hosting as an alternative (excluding the 3rd party) or self-hosting as a method of redundancy (in addition to the third party)? If you self host as an ALTERNATIVE to the 3rd party you have all of the same problems - more because you know about them, and the 3rd party can make all these claims you can't verify until they fall over with a "load balancer misconfig" story you also can't verify. If you self-host redundantly to a 3rd party you have no special benefit (it does the same thing) AND the additional cost of a redundant infrastructure. Why not just have redundant 3rd parties (so-called "multi-cloud") if you can't or won't trust your 3rd party.
      • theteapot 2y ago
        I believe the point was protecting against risk of 3rd party falling over. If you self host that reduces the risk. At least you have visibility into the falling over process.
        • Over2Chars 2y ago
          It reduces the risk only if your self-hosted solution also doesn't fall over. It's like saying "I can reduce the risk of my rental car failure by owning my own car", assuming your own car you keep undriven in your garage, doesn't have a dead battery, no gas, flat tires, and proves to be unusable for hauling. The "cloud" was touted as the fix for all that nuisance in self-hosting. Magically Jeff's bit barn would work to five 9s of uptime, and you could sit back and write your code, unshackled to infra. Until Jeff's bit barn went tits up. I say the "cloud" is just another guys data center behind an API. You wanna cloud experience? Put an API in front of your own servers and burn a $100 bill.
• outworlder 2y ago
  My company is replacing the entire system we built with Github actions. Our previous DR plan involved running our automation scripts to reprovision the infrastructure that manages the rest of the infrastructure (runs jenkins jobs, etc). They are replacing everything with Github actions. I wonder what they are going to do when Github is down.
  • deleted 2y ago
    [deleted]
• globnomulous 2y ago
  I work on one part of a huge suite of interconneted services that have extremely strict SLAs for up-time. It really is an interesting problem, and the quantity of engineering resources devoted to ensuring availability and avoiding downtime -- basically fighting to make inherently unstable, complex systems stable and reliable -- is dumbfounding. When I'm trying to explain to people what it's like to work on this kind of software, I like to use an analogy: it's as though I have my own personal brick, or group of bricks, in the great pyramids of Egypt, just a a tiny piece of a stupefyingly, inconceivably larger whole -- and when I twist my chisel and strike my block just so, at exactly the right (or rather the wrong) angle, I can shake the very foundations of Egypt.
• thayne 2y ago
  Let's say you self-host Github. Now you are responsible for maintaining uptime, and you have less expertise with the service and less resources to dedicate to keeping it up, so it's going to be hard to match, much less exceed the uptime of Github cloud. And it doesn't protect you from a "forced exit" either. Github could terminate your contract, and change the terms of the license in a way that you found unacceptable, or even go out of business, and being self hosted would leave you in no better position than if you had used cloud with external backups. You can somewhat mitigate this risk by self hosting an open source solution, so that in the worst case scenario, you can fork the project and maintain it yourself, but there is still risk that the project could be abandoned, or possibly have the license changed in future versions. To be clear, I'm not saying that you shouldn't self host and SaaS is always better. But it isn't a magic bullet that solves these problems.
  • bgrainger 2y ago
    We self-host GitHub using GitHub Enterprise Server. It is a mature product that requires next-to-no maintenance and is remarkably stable. (We did have a period of downtime caused by running it on an underprovisioned VM for our needs, but since resolving that it hasn't had problems.) Of course we have a small and mostly unchanging number of users, don't have to deal with DDoS attacks, and can schedule the fairly-infrequent updates during maintenance windows that are convenient for us (since we don't need 100% availability outside of US working hours). I don't have the metrics in front of me, but I would say we've easily exceeded github.com's uptime in the last 12 months.
    • jcims 2y ago
      Things start to go sideways when you have tens of thousands of users.
      • randmeerkat 2y ago
        > Things start to go sideways when you have tens of thousands of users. If that’s really the case, run another GitHub instance then. Not all tens of thousands of users need access to the same codebases. In the kind of environment described someone would want identity boundaries established around each project anyway…