9 ms·

Your question seems to indicate that you have a runtime dependency on github. If true, this is a problem no matter where your git repo resides. You need to arch

by jonaf 9y ago

Your question seems to indicate that you have a runtime dependency on github. If true, this is a problem no matter where your git repo resides. You need to architect a solution wherein you do not have a runtime dependency, especially if it is a single point of failure, or accept that your reliability will never be better than the sum of your services' reli abilities (there was an article on the "calculus of availability" or something like that recently). What I personally do in AWS is bake my artifacts or other git-sourced data into AMI's. If you want a middle ground, you can instead push your artifacts to an s3 target -- s3 has better reliability / track record than github for this purpose.

avaer 9y ago
Thing is, runtime dependencies on Github is something that Github is promoting and selling now (https://github.com/marketplace/ https://github.com/marketplace/). Maybe this incident puts that value proposition in a more visceral context.
- aisofteng 9y ago
  Just because they sell it doesn't mean people shouldn't know better.
- paxy 9y ago
  How is the GitHub marketplace a runtime dependency? Code review, project management, build bots or whatever else should NOT affect the availability of your website.
  - deleted 9y ago
    [deleted]
  - avaer 9y ago
    Sure, if you measure uptime of your business and dev team by whether you're serving pages. I wouldn't. If Github being down means you're not fixing your site today, I'd call it a runtime dependency.
    - mehwoot 9y ago
      I kind of see your point but it's the wrong phrase. It's not a runtime dependency in a technical sense.
      - mygo 9y ago
        see now you're just splitting hairs and getting into philosophy. if your website will be stale tomorrow because you can't update it today, then that is a runtime dependency imo.
        paulddraper 9y ago
        Why it is down tomorrow? (Hint: sans hug of death, because a runtime dependency brought it down.)
    - paulddraper 9y ago
      Incorrect. You need build-time dependencies to change an application, and runtime dependencies to run it. Deploying/scaling is somewhere in the middle.
      - jonaf 9y ago
        Yeah, I should have clarified I was talking with my operations hat on. From a development perspective, you might call it a "deploy-time" or "scale up" dependency.
- delias_ 9y ago
  naive tip: give your app that is relying on github a SKIPGIT environment variable to ignore the bits around git operations, and just make local changes as an interim hack.
todd3834 9y ago
Is it still considered a runtime dependency if all you do is merge code and deploy while relying on GitHub? I assume most apps will not go down if GitHub goes down, but their ability to move code to production gets stalled.
- tux1968 9y ago
  Git is inherently distributed. There's really no reason that Github has to stand between development and production.
  - noobermin 9y ago
    I think the point is git is becoming the new svn for many people.
    - JonathonW 9y ago
      "Becoming"? Can you point to any large project (at any point in the history of Git) that doesn't use some centralized service as a single point of distribution/source of truth? The beauty of Git's design, though, is that you've still got everything to use it distributed even if you, under normal circumstances, just use it centralized like a fancy SVN.
  - gtirloni 9y ago
    There's the convenience reason. Teams making that trade-off (not having a local mirror) need to understand they made GitHub itself a runtime dependency.
    - bananaboy 9y ago
      But in theory in a pinch you could spin up a git server somewhere publicly accessible and switch your site to pull from there, and push to there yourself.
      - gtirloni 9y ago
ProAm 9y ago
> Your question seems to indicate that you have a runtime dependency on github. Really he is speaking to the problem with SAAS. Sure it's cheaper to rely on someone else to do the heavy lifting for you and they can do this because they 'make it up in volume'. But the other side of that coin is no one really knows how to do that anymore. If you wanted to roll your own it's going to be very hard for your company to do the basics because you've become dependent upon the cloud, and someone else paying employees to do the basics for you... The basics have never been hard.
- Twisell 9y ago
  But a positiv counter argument would be that github/gitlab also does the sharing part of the jobs and you (hopefully) pay them to handle all the security aspect of that public sharing. Opening a port to your local instance of git implies more thinking and security consideration. Of course you could host a mirror on a cloud instance, but then it's saas anyway and you have even more work to do before ever starting coding. So yeah github/gitlab does the heavy lifting for you, but for small organisations that might be cheaper overall than to pay somebody mastering all required stack to implement and maintain your own instance(s). For big company no question that an in house git team is probably more efficient.
vacri 9y ago
> you can instead push your artifacts to an s3 target This is our tactic. I tried to do baked AMIs at one point, but the 10-15 minute turnaround in registering them meant that we couldn't use them for staging or testing (too long to iterate changes). Previously we were capistrano-deploying with git from bitbucket - every server had to individually git pull from servers on the other side of the continent, over the public internet. Susceptible to all sorts of problems.
- jonaf 9y ago
  You can speed up AMI baking by using chroot + a mounted EBS volume on a running instance. It should drastically reduce the time it takes to produce a new image. Also be sure that you're building EBS-backed AMI's, as S3-backed AMI's will require a lot of time for copy.

by jonaf 9y ago

avaer 9y ago

Thing is, runtime dependencies on Github is something that Github is promoting and selling now (https://github.com/marketplace/ https://github.com/marketplace/). Maybe this incident puts that value proposition in a more visceral context.

aisofteng 9y ago
Just because they sell it doesn't mean people shouldn't know better.
paxy 9y ago
How is the GitHub marketplace a runtime dependency? Code review, project management, build bots or whatever else should NOT affect the availability of your website.
- deleted 9y ago
  [deleted]
- avaer 9y ago
  Sure, if you measure uptime of your business and dev team by whether you're serving pages. I wouldn't. If Github being down means you're not fixing your site today, I'd call it a runtime dependency.
  - mehwoot 9y ago
    I kind of see your point but it's the wrong phrase. It's not a runtime dependency in a technical sense.
    - mygo 9y ago
      see now you're just splitting hairs and getting into philosophy. if your website will be stale tomorrow because you can't update it today, then that is a runtime dependency imo.
      - paulddraper 9y ago
        Why it is down tomorrow? (Hint: sans hug of death, because a runtime dependency brought it down.)
  - paulddraper 9y ago
    Incorrect. You need build-time dependencies to change an application, and runtime dependencies to run it. Deploying/scaling is somewhere in the middle.
    - jonaf 9y ago
      Yeah, I should have clarified I was talking with my operations hat on. From a development perspective, you might call it a "deploy-time" or "scale up" dependency.
delias_ 9y ago
naive tip: give your app that is relying on github a SKIPGIT environment variable to ignore the bits around git operations, and just make local changes as an interim hack.

todd3834 9y ago

Is it still considered a runtime dependency if all you do is merge code and deploy while relying on GitHub? I assume most apps will not go down if GitHub goes down, but their ability to move code to production gets stalled.

tux1968 9y ago
Git is inherently distributed. There's really no reason that Github has to stand between development and production.
- noobermin 9y ago
  I think the point is git is becoming the new svn for many people.
  - JonathonW 9y ago
    "Becoming"? Can you point to any large project (at any point in the history of Git) that doesn't use some centralized service as a single point of distribution/source of truth? The beauty of Git's design, though, is that you've still got everything to use it distributed even if you, under normal circumstances, just use it centralized like a fancy SVN.
- gtirloni 9y ago
  There's the convenience reason. Teams making that trade-off (not having a local mirror) need to understand they made GitHub itself a runtime dependency.
  - bananaboy 9y ago
    But in theory in a pinch you could spin up a git server somewhere publicly accessible and switch your site to pull from there, and push to there yourself.
    - gtirloni 9y ago

ProAm 9y ago

> Your question seems to indicate that you have a runtime dependency on github. Really he is speaking to the problem with SAAS. Sure it's cheaper to rely on someone else to do the heavy lifting for you and they can do this because they 'make it up in volume'. But the other side of that coin is no one really knows how to do that anymore. If you wanted to roll your own it's going to be very hard for your company to do the basics because you've become dependent upon the cloud, and someone else paying employees to do the basics for you... The basics have never been hard.

Twisell 9y ago
But a positiv counter argument would be that github/gitlab also does the sharing part of the jobs and you (hopefully) pay them to handle all the security aspect of that public sharing. Opening a port to your local instance of git implies more thinking and security consideration. Of course you could host a mirror on a cloud instance, but then it's saas anyway and you have even more work to do before ever starting coding. So yeah github/gitlab does the heavy lifting for you, but for small organisations that might be cheaper overall than to pay somebody mastering all required stack to implement and maintain your own instance(s). For big company no question that an in house git team is probably more efficient.

vacri 9y ago

> you can instead push your artifacts to an s3 target This is our tactic. I tried to do baked AMIs at one point, but the 10-15 minute turnaround in registering them meant that we couldn't use them for staging or testing (too long to iterate changes). Previously we were capistrano-deploying with git from bitbucket - every server had to individually git pull from servers on the other side of the continent, over the public internet. Susceptible to all sorts of problems.

jonaf 9y ago
You can speed up AMI baking by using chroot + a mounted EBS volume on a running instance. It should drastically reduce the time it takes to produce a new image. Also be sure that you're building EBS-backed AMI's, as S3-backed AMI's will require a lot of time for copy.