5 ms·

I saw some comments on reddit which highlighted a pretty serious problem - many orgs rely on github as a fully integrated CD platform, with everything from code

I saw some comments on reddit which highlighted a pretty serious problem - many orgs rely on github as a fully integrated CD platform, with everything from code hosting, to running CI hooks, to pushing to staging or prod. It seems very unwise to have essentially your whole deployment process manager in the hands of an entity which you don't only have no control over, but which has regularly been targeted in attacks by nation-state-level actors because of their role as a code hosting platform. EDIT: GH hasn't been targeted regularly, but it has been so historically, so this is a plausible thing which might happen again.

Klathmon 8y ago
It's a tradeoff, just like anything else. We use Github heavily at my work, and at past jobs as well. But at the same time it's not the ONLY way we can work. If github has an outage, our CD will shut down, but we can still run all tests locally, and we can still push to the server directly, and we can still push code between ourselves manually and review it. Sure, it's a hiccup in our day when it goes down, but it's not like the entire company grinds to a halt. And the alternative of maintaining servers and systems to replicate all of that would take significantly more time, potentially cost more, and is probably more likely to go down.
- colonelpopcorn 8y ago
  On premises git hosting with Gittea or GitLab with mirrors to GitHub seems like a smart idea going forward.
  - elliotlarson 8y ago
    But then you have to host it and maintain it. It's a slippery slope. How many 3rd party services do you in-house with hosted OS software. Pretty soon you're spending a huge chunk of your time doing ops work. And, where do you host it? On AWS, which can also go down, or on hardware hosted at your office. With on premise hosting, now you're in the hardware game too.
    - module0000 8y ago
      Well, your code is your companies' IP. It might be prudent to have that IP on-prem(eg Gitlab). Every business has varying requirements, but I've yet to be employed at one using 3rd party hosting for source control without an on-site mirror. Disclaimer: my employment has been at megacorps so far, smaller shops may not do this.
      - Klathmon 8y ago
        with git, you have a mirror on every single developer machine (kinda... depending on what you consider "all" of the code). We also have a full copy on our CI server (which is hosted on another service, so still not "in-house"), and we have a copy of at least the master branch (and all it's history) on our production boxes (which also gets pulled into our whole backup system there). In a disaster recovery scenario, that's more than enough for me. Sure, if github blinked out of existence we would probably be at a fraction of our normal productivity for a while until we fully recover everything and find new workflows, but the risk vs reward there is well within the margins of what I'd consider acceptable for a company like us.
        ams6110 8y ago
        > In a disaster recovery scenario That's not really enough. You need more than "all the code is around here somewhere" you need a plan with specific steps that have been tested.
        twunde 8y ago
        Honestly for git, it probably is enough. We're talking about someone deleting your github account or github closing overnight with no warning (it's been acquired by Microsoft, so it's much more likely that the company you're working for will shutter). It should take ~30 minutes to push your repo to another provider including looking up instructions. Unlike database backups, there is rarely any data loss and any data loss should be recoverable. It's also not client-facing, but is a temporary problem similar to wifi going down at your office. An inconvenience and hassle, yes. Long-term problem, no. Furthermore, the problem with these disaster scenarios are that there are much more dangerous problems than your account being deleted. Someone with admin access, could insert a back door or sell your source code to someone else. That's honestly scarier.
  - deleted 8y ago
    [deleted]
  - deleted 8y ago
    [deleted]
- bschwindHN 8y ago
  > and is probably more likely to go down. People often say this but I wonder if it's true. Your replication wouldn't be handling the same load as GitHub itself, so perhaps the issues that GitHub experiences would rarely happen to a self hosted version.
ProAm 8y ago
>It seems very unwise to have essentially your whole deployment process manager in the hands of an entity which you don't only have no control over, Welcome to the SAAS world. Where you offload institutional knowledge and hiring to a 3rd party and roll the dice.
nineteen999 8y ago
This is why we built and manage our own git servers internally. We run emergency services infrastructure so we can not afford to be disconnected from our version control. Ever. Same reasons we don't run our application and supporting infrastructure in a cloud provider really - apart from the complexity of our particular network which has a heavy dependency on mobile and satellite links - we cannot guarantee our customer of our availability if we don't control the compute, storage, SAN fabric, and as much of the network infrastructure as is possible.
h1d 8y ago
Why would state actor level attacker target for being a code hosing platform? At best they can temporarily suspend operations.

Klathmon 8y ago

It's a tradeoff, just like anything else. We use Github heavily at my work, and at past jobs as well. But at the same time it's not the ONLY way we can work. If github has an outage, our CD will shut down, but we can still run all tests locally, and we can still push to the server directly, and we can still push code between ourselves manually and review it. Sure, it's a hiccup in our day when it goes down, but it's not like the entire company grinds to a halt. And the alternative of maintaining servers and systems to replicate all of that would take significantly more time, potentially cost more, and is probably more likely to go down.

colonelpopcorn 8y ago
On premises git hosting with Gittea or GitLab with mirrors to GitHub seems like a smart idea going forward.
- elliotlarson 8y ago
  But then you have to host it and maintain it. It's a slippery slope. How many 3rd party services do you in-house with hosted OS software. Pretty soon you're spending a huge chunk of your time doing ops work. And, where do you host it? On AWS, which can also go down, or on hardware hosted at your office. With on premise hosting, now you're in the hardware game too.
  - module0000 8y ago
    Well, your code is your companies' IP. It might be prudent to have that IP on-prem(eg Gitlab). Every business has varying requirements, but I've yet to be employed at one using 3rd party hosting for source control without an on-site mirror. Disclaimer: my employment has been at megacorps so far, smaller shops may not do this.
    - Klathmon 8y ago
      with git, you have a mirror on every single developer machine (kinda... depending on what you consider "all" of the code). We also have a full copy on our CI server (which is hosted on another service, so still not "in-house"), and we have a copy of at least the master branch (and all it's history) on our production boxes (which also gets pulled into our whole backup system there). In a disaster recovery scenario, that's more than enough for me. Sure, if github blinked out of existence we would probably be at a fraction of our normal productivity for a while until we fully recover everything and find new workflows, but the risk vs reward there is well within the margins of what I'd consider acceptable for a company like us.
      - ams6110 8y ago
        > In a disaster recovery scenario That's not really enough. You need more than "all the code is around here somewhere" you need a plan with specific steps that have been tested.
        twunde 8y ago
        Honestly for git, it probably is enough. We're talking about someone deleting your github account or github closing overnight with no warning (it's been acquired by Microsoft, so it's much more likely that the company you're working for will shutter). It should take ~30 minutes to push your repo to another provider including looking up instructions. Unlike database backups, there is rarely any data loss and any data loss should be recoverable. It's also not client-facing, but is a temporary problem similar to wifi going down at your office. An inconvenience and hassle, yes. Long-term problem, no. Furthermore, the problem with these disaster scenarios are that there are much more dangerous problems than your account being deleted. Someone with admin access, could insert a back door or sell your source code to someone else. That's honestly scarier.
- deleted 8y ago
  [deleted]
- deleted 8y ago
  [deleted]
bschwindHN 8y ago
> and is probably more likely to go down. People often say this but I wonder if it's true. Your replication wouldn't be handling the same load as GitHub itself, so perhaps the issues that GitHub experiences would rarely happen to a self hosted version.

ProAm 8y ago

>It seems very unwise to have essentially your whole deployment process manager in the hands of an entity which you don't only have no control over, Welcome to the SAAS world. Where you offload institutional knowledge and hiring to a 3rd party and roll the dice.

nineteen999 8y ago

This is why we built and manage our own git servers internally. We run emergency services infrastructure so we can not afford to be disconnected from our version control. Ever. Same reasons we don't run our application and supporting infrastructure in a cloud provider really - apart from the complexity of our particular network which has a heavy dependency on mobile and satellite links - we cannot guarantee our customer of our availability if we don't control the compute, storage, SAN fabric, and as much of the network infrastructure as is possible.

h1d 8y ago

Why would state actor level attacker target for being a code hosing platform? At best they can temporarily suspend operations.