4 ms·

GitHub as a sysadmin/system engineer I feel your pain, I understand it completely and know the horrors of failures leading into multi-hour recovery. That said,

by chuhnk 14y ago

GitHub as a sysadmin/system engineer I feel your pain, I understand it completely and know the horrors of failures leading into multi-hour recovery. That said, you need to do better. This is a heavily relied upon resource for the open source community. Perhaps you didn't estimate this sort of growth but now you are here I'm sorry but the weight does fall on your shoulders. I heavily commend you guys on the service you've provided thus far and what you've done to actually pull all the varying language/library communities together. I honestly want to see this scale and serve 5 9s year round. You need to take a good hard look at the architecture of the stack and find a way to get it multi-homed.

thaumaturgy 14y ago
Please don't take this personally, but I'd like to propose a new rule of etiquette on HN: it's unacceptably lazy to say that some business "needs to do better" without explaining exactly what they should do better, or how. GitHub is doing better by proactively upgrading their network. During the upgrade, they have run in to some technical difficulties. There are maybe, what, a thousand people in the U.S. that have worked on network administration for something GitHub's size? And far fewer who could have predicted this kind of trouble? If we're going to shit on someone, we should at least make it nutrient-rich: include recommendations based on experience from dealing with problems of that nature and magnitude.
- dsl 14y ago
  I understand they have a lot of nerd/hacker cred in and around the tech hotspots of the country, but GitHub is by no measure "big". The most recent number I could find puts them at 33 servers total. I've worked for non-tech companies that have that many as hot standbys. One big recommendation I'd offer is to have a secondary network with as few moving pieces as possible (think a dumb unmanaged Netgear switch or two) to run things like heartbeats and DRBD over. That sort of thing should not be living on the same segment as your production traffic.
  - imbriaco 14y ago
    We have significantly more than 33 servers.
    - dsl 14y ago
      Feel free to share. All of the major vendors MLAG implementations are limited to two agg switches, assuming you have a pair of 24 ports and a 48 port rack level switch hanging off each, that still puts total servers at less than 1000.
      - nixgeek 14y ago
        By that simplistic logic then Arista 7500-series [1] would give you a 384-port aggregation (per device) with 48-port rack switches hung off it for something like 8000+ servers. What stops someone from having multiple zones within a datacenter interconnected with suitably large LAGs thereby avoiding being limited by the port capacity of any one device? Why is the exact number of servers important to the topic, or is it just to satisfy your curiosity? :) [1] http://www.aristanetworks.com/en/products/7500series http://www.aristanetworks.com/en/products/7500series
        dsl 14y ago
        I'm specifically addressing the original parents straw man. > There are maybe, what, a thousand people in the U.S. that have worked on network administration for something GitHub's size? I didn't intend to deflate any nerd egos, just putting things in realistic terms. GitHub is cool. It's the biggest in its industry. It's definitely an interesting problem. But it's still hosted within Rackspace, which in itself isn't that big.
- chuhnk 14y ago
  Actually you are right it is a bit lazy. Ok so the first question is, are they multi-homed (as in multiple datacenters)? If not, why not? Because that should be the first step to take. We could talk about all the ways in which you can build redundancy and automated recovery into a cluster of machines in a single datacenter but let me tell you the obvious point that the datacenter is a single point of failure. Rackspace are an extremely solid hosting provider, we used them in my old company for 5 years but even they can suffer fiber cuts, power cuts, UPS's dying, generators not working, someone performing network maintenance and crapping out the routing for the entire dc. My first recommendation would be setting up a warm-standby or even a hot-standby if possible in another datacenter. It all comes down to state, meaning the data/repos. They are currently using drbd for block level replication of disks in their file servers, this can be done over wan using something like drbd proxy. Another option is to write something in house which replicates write requests to git repos hosted elsewhere. They wrote a collection of services to access git repos on the file servers via rpc, its entirely plausible to extend this. If they are using mysql anywhere thats a pretty simple task for master-slave replication or semi-sync replication in mysql 5.5. Redis also supports replication. I'm sort of hinting at high availability with read/write but starting out with a read-only solution hosted in another datacenter on the order of 60-120 secs out of sync is not impossible. I think we would all benefit from that kind of continuity right? Replicate your data to a secondary site, keep it in read-only mode, set low ttl's in dns. Take it a step further and start looking at globally distributed file systems if you are feeling adventurous.
  - robryan 14y ago
    That wouldn't have really helped them here though? If the closely connected storage nodes were unable to detect whether state was consistent, how would have the storage node in another data center known? I guess what your saying is in an error making the site entirely read only from another data center. That could be of some use but I would imagine that would require a complete rearchitecture to separate out all writes and also have a method of recombining stuff like logging which will still be written later on. On the git level though it would be useful for a major use case of github which is to grab code.
    - 14y ago
Ricapar 14y ago
Sure, GitHub is "heavily relied upon resource for the open source community" - but what is really the impact of their downtime? You have to wait a little longer to do your push/pull/merge/etc? Give them a break.
- chuhnk 14y ago
  Remember that individuals and organisations pay for private repo hosting. Aren't they entitled to an adequate SLA of say 4 9s a year?
  - nixgeek 14y ago
    As with any purchasing decision you can read the T&Cs when making up your mind and are free to vote with your wallet if you disagree. Another factoid is that Amazon Web Services (AWS) only offers a 99.95% SLA so where did the 99.99% for GitHub come from?
    - chuhnk 14y ago
      Why does it matter that AWS is 99.95%? "Cloud" based services like AWS and AppEngine have more relaxed SLAs. GitHub is not a cloud service, its a centralized version control repository. And the 4 9s, well that's just something I've always strived for when providing a service but I do believe we should all hold ourselves to it.
- gcr 14y ago
  When seen in that light, lots of nonessential services seem equally unimportant. What's really the impact of Facebook downtime? You have to wait a little longer to "poke" your crush's profile/look at their picture/etc? Give them a break.
- kawsper 14y ago
  Homebrew on OSX is unusable without Github. Rubygems is kind of unusable without Github. Several ppensource projects documentation is unavailable without Github.

by chuhnk 14y ago

thaumaturgy 14y ago

Please don't take this personally, but I'd like to propose a new rule of etiquette on HN: it's unacceptably lazy to say that some business "needs to do better" without explaining exactly what they should do better, or how. GitHub is doing better by proactively upgrading their network. During the upgrade, they have run in to some technical difficulties. There are maybe, what, a thousand people in the U.S. that have worked on network administration for something GitHub's size? And far fewer who could have predicted this kind of trouble? If we're going to shit on someone, we should at least make it nutrient-rich: include recommendations based on experience from dealing with problems of that nature and magnitude.

dsl 14y ago
I understand they have a lot of nerd/hacker cred in and around the tech hotspots of the country, but GitHub is by no measure "big". The most recent number I could find puts them at 33 servers total. I've worked for non-tech companies that have that many as hot standbys. One big recommendation I'd offer is to have a secondary network with as few moving pieces as possible (think a dumb unmanaged Netgear switch or two) to run things like heartbeats and DRBD over. That sort of thing should not be living on the same segment as your production traffic.
- imbriaco 14y ago
  We have significantly more than 33 servers.
  - dsl 14y ago
    Feel free to share. All of the major vendors MLAG implementations are limited to two agg switches, assuming you have a pair of 24 ports and a 48 port rack level switch hanging off each, that still puts total servers at less than 1000.
    - nixgeek 14y ago
      By that simplistic logic then Arista 7500-series [1] would give you a 384-port aggregation (per device) with 48-port rack switches hung off it for something like 8000+ servers. What stops someone from having multiple zones within a datacenter interconnected with suitably large LAGs thereby avoiding being limited by the port capacity of any one device? Why is the exact number of servers important to the topic, or is it just to satisfy your curiosity? :) [1] http://www.aristanetworks.com/en/products/7500series http://www.aristanetworks.com/en/products/7500series
      - dsl 14y ago
        I'm specifically addressing the original parents straw man. > There are maybe, what, a thousand people in the U.S. that have worked on network administration for something GitHub's size? I didn't intend to deflate any nerd egos, just putting things in realistic terms. GitHub is cool. It's the biggest in its industry. It's definitely an interesting problem. But it's still hosted within Rackspace, which in itself isn't that big.
chuhnk 14y ago
Actually you are right it is a bit lazy. Ok so the first question is, are they multi-homed (as in multiple datacenters)? If not, why not? Because that should be the first step to take. We could talk about all the ways in which you can build redundancy and automated recovery into a cluster of machines in a single datacenter but let me tell you the obvious point that the datacenter is a single point of failure. Rackspace are an extremely solid hosting provider, we used them in my old company for 5 years but even they can suffer fiber cuts, power cuts, UPS's dying, generators not working, someone performing network maintenance and crapping out the routing for the entire dc. My first recommendation would be setting up a warm-standby or even a hot-standby if possible in another datacenter. It all comes down to state, meaning the data/repos. They are currently using drbd for block level replication of disks in their file servers, this can be done over wan using something like drbd proxy. Another option is to write something in house which replicates write requests to git repos hosted elsewhere. They wrote a collection of services to access git repos on the file servers via rpc, its entirely plausible to extend this. If they are using mysql anywhere thats a pretty simple task for master-slave replication or semi-sync replication in mysql 5.5. Redis also supports replication. I'm sort of hinting at high availability with read/write but starting out with a read-only solution hosted in another datacenter on the order of 60-120 secs out of sync is not impossible. I think we would all benefit from that kind of continuity right? Replicate your data to a secondary site, keep it in read-only mode, set low ttl's in dns. Take it a step further and start looking at globally distributed file systems if you are feeling adventurous.
- robryan 14y ago
  That wouldn't have really helped them here though? If the closely connected storage nodes were unable to detect whether state was consistent, how would have the storage node in another data center known? I guess what your saying is in an error making the site entirely read only from another data center. That could be of some use but I would imagine that would require a complete rearchitecture to separate out all writes and also have a method of recombining stuff like logging which will still be written later on. On the git level though it would be useful for a major use case of github which is to grab code.
  - 14y ago

Ricapar 14y ago

Sure, GitHub is "heavily relied upon resource for the open source community" - but what is really the impact of their downtime? You have to wait a little longer to do your push/pull/merge/etc? Give them a break.

chuhnk 14y ago
Remember that individuals and organisations pay for private repo hosting. Aren't they entitled to an adequate SLA of say 4 9s a year?
- nixgeek 14y ago
  As with any purchasing decision you can read the T&Cs when making up your mind and are free to vote with your wallet if you disagree. Another factoid is that Amazon Web Services (AWS) only offers a 99.95% SLA so where did the 99.99% for GitHub come from?
  - chuhnk 14y ago
    Why does it matter that AWS is 99.95%? "Cloud" based services like AWS and AppEngine have more relaxed SLAs. GitHub is not a cloud service, its a centralized version control repository. And the 4 9s, well that's just something I've always strived for when providing a service but I do believe we should all hold ourselves to it.
gcr 14y ago
When seen in that light, lots of nonessential services seem equally unimportant. What's really the impact of Facebook downtime? You have to wait a little longer to "poke" your crush's profile/look at their picture/etc? Give them a break.
kawsper 14y ago
Homebrew on OSX is unusable without Github. Rubygems is kind of unusable without Github. Several ppensource projects documentation is unavailable without Github.