6 ms·

Can we please use some of that 100 million investment to buy an infrastructure that is more resilient to these kinds of DDoS attacks? Pretty please? This is a

Can we please use some of that 100 million investment to buy an infrastructure that is more resilient to these kinds of DDoS attacks? Pretty please? This is a service I pay for and my business relies on. Having it down three times in three days impacts our work.

Caballera 14y ago
I'm sure that's what they are doing now. However it takes time to setup new servers as well as writing code (that's been throughly audited) to help protect their existing and future servers.
bkanber 14y ago
Did it really impact your work, other than slightly inconveniencing you? "UGH GitHub down again, I guess I have to go work on something equally as important for upwards of an hour" I call shenanigans on you, good sir.
- codinghorror 14y ago
  Since when is it OK for the services I rely on and pay for to even slightly inconvenience me ... for three days in a row? Since never. At least for businesses that want to remain an ongoing concern.
  - bkanber 14y ago
    Devil's avocado here: let's say you pay $100/mo for a gym membership and they shut down three days in a row because somebody called in a threat. How upset would you be at the gym? A malicious attack by a third party is different from, say, the gym allowing black mold to grow in the locker room. I'd quit a gym if they had black mold. That's mismanagement. I wouldn't quit a gym if malicious third party intervention inconvenienced me. Besides, GitHub is obviously more concerned about this than you or I could ever be. And having money doesn't make infrastructure magically appear. I pay GitHub too. My company relies on it. I, too, was slightly inconvenienced this week. I was also slightly inconvenienced when I had to make a u-turn because the Battery Tunnel southbound on-ramp was closed. So what? In summary: shenanigans! Good day sir!
    - Tobu 14y ago
      What makes DDOSes different from black mold? Both are expected risks and should be mitigated. Yeah, there are sentient actors behind the DDOS, but GitHub has to deal with it at the level of their infrastructure either way.
      - bkanber 14y ago
        The fact that there are sentient actors behind the DDoS _is_ the difference. You can reliably predict and protect against things like network outages, server failures, full datacenter failures (black mold)--you can directly measure their impact and plan failover paths. A DB server goes out? Whatever! That's why you have a hot backup or two online and ready to go. What you can't predict is exactly how far a malicious third party will go to hurt you. You can't predict how many dollars they'll spend on their botnet minutes. You don't know if they're going to attack your infrastructure or the DNS. Can buying more bandwidth fix the problem? If so, how much more? And will the attacker simply up the ante when they see that you're recovering? Can filtering requests fix the problem? If so, will the attacker provision different resources to attack you with? This isn't simply a matter of infrastructure, buying the right equipment, or setting things up "just right" precisely because there is a sentient actor trying to hurt you. It's more like a game of chess.
  - jiggy2011 14y ago
    If high availability git is so critical then why not run your own git servers instead of or in addition to github? I find it slightly ironic that the entire point of git is that it is distributed version control but 90% of git use seems to be focused around a product from a single company.
    - Serplat 14y ago
      Github isn't solely a git server. They also have issues, wikis, etc, that are harder to keep in multiple locations.
      - Tobu 14y ago
        The wikis are git repos as well. Sync them periodically and you're good. Plus it's useful as a backup, because I don't think github archives your reflog. Issues are normally mirrored to e-mails (caveat: you don't get mail for your own comments), so you can mostly pick up existing threads if your e-mail address book can find the github users involved. If they didn't obscure recipients (at least within an organisation — because I don't think address-book lock-in is worth inconveniencing paying clients), and made an auto self-bcc of your activity, issues would be entirely disaster resistant.
      - 27182818284 14y ago
        If you have GitHub enterprise, it is run on your stuff and you get your own Wikis, issues, unlimited repos, etc. Even your own Gist that you can wrap behind your own CAS.
- adgar2 14y ago
  Because high availability is for legacy waterfall chumps. Github's too busy being rockstars.
Firehed 14y ago
Easier said than done. We've had DDOS issues in the past as well, and getting it resolved - even by throwing money at the problem - is nontrivial. What amounts to throwing a massive amount of hardware at the problem (i.e., boxes that can handle 10-100+gbps of traffic, filter out the attacks, and pass only legit stuff down to your servers) is expensive[1], and casuses all sorts of unexpected behavior: API clients mysteriously break, good traffic gets mistakenly dropped, latency is added to the whole process, etc. It gets even weirder on SSL-protected sites. And it's all dependent on attackers not getting the IP of your actual servers which they could then just attack directly. [1] For sites with even not a whole lot of traffic, you're talking a one-year contract easily in the range of an engineer's salary. I wouldn't be surprised if the cost to protect sites with as much traffic as Github exceeded $1m/year. Even if you have plenty of cash in the bank, that's one hell of a pill to swallow.
- nnnnnnnn 14y ago
  Github can easily afford to use someone like Prolexic. And they should. When you say things like "And it's all dependent on attackers not getting the IP of your actual servers" this makes me wonder how much you understand the subject matter. There are many, many options.
  - Firehed 14y ago
    Prolexic's servers don't take the load if the attackers know where the computers behind the scrubbers are. Configuring iptables to ignore all traffic not coming from prolexic's IPs doesn't come close to fending off a DDOS. I know this because I was told this by prolexic while configuring our servers to sit behind their scrubbing servers while we're under an equally crippling DDOS (one that took down half the customers in our datacenter, not just us). So while I haven't examined their tech stack under a magnifying glass, I'm not exactly talking out of my ass here. Yes, there are other options but those don't take an hour to implement like signing a contract and changing a few DNS entries does. And when these conditions exist, you need an answer that can be implemented in an hour.
    - 14y ago
mylittlepony 14y ago
Just move to bitbucket. You can even pay to them if you want to.
- mattdeboard 14y ago
  Yeah, bitbucket never gets DDOS'd! Well, except when they do http://blog.bitbucket.org/2009/10/04/on-our-extended-downtime-amazon-and-whats-coming/ http://blog.bitbucket.org/2009/10/04/on-our-extended-downtim...

Caballera 14y ago

I'm sure that's what they are doing now. However it takes time to setup new servers as well as writing code (that's been throughly audited) to help protect their existing and future servers.

bkanber 14y ago

Did it really impact your work, other than slightly inconveniencing you? "UGH GitHub down again, I guess I have to go work on something equally as important for upwards of an hour" I call shenanigans on you, good sir.

codinghorror 14y ago
Since when is it OK for the services I rely on and pay for to even slightly inconvenience me ... for three days in a row? Since never. At least for businesses that want to remain an ongoing concern.
- bkanber 14y ago
  Devil's avocado here: let's say you pay $100/mo for a gym membership and they shut down three days in a row because somebody called in a threat. How upset would you be at the gym? A malicious attack by a third party is different from, say, the gym allowing black mold to grow in the locker room. I'd quit a gym if they had black mold. That's mismanagement. I wouldn't quit a gym if malicious third party intervention inconvenienced me. Besides, GitHub is obviously more concerned about this than you or I could ever be. And having money doesn't make infrastructure magically appear. I pay GitHub too. My company relies on it. I, too, was slightly inconvenienced this week. I was also slightly inconvenienced when I had to make a u-turn because the Battery Tunnel southbound on-ramp was closed. So what? In summary: shenanigans! Good day sir!
  - Tobu 14y ago
    What makes DDOSes different from black mold? Both are expected risks and should be mitigated. Yeah, there are sentient actors behind the DDOS, but GitHub has to deal with it at the level of their infrastructure either way.
    - bkanber 14y ago
      The fact that there are sentient actors behind the DDoS _is_ the difference. You can reliably predict and protect against things like network outages, server failures, full datacenter failures (black mold)--you can directly measure their impact and plan failover paths. A DB server goes out? Whatever! That's why you have a hot backup or two online and ready to go. What you can't predict is exactly how far a malicious third party will go to hurt you. You can't predict how many dollars they'll spend on their botnet minutes. You don't know if they're going to attack your infrastructure or the DNS. Can buying more bandwidth fix the problem? If so, how much more? And will the attacker simply up the ante when they see that you're recovering? Can filtering requests fix the problem? If so, will the attacker provision different resources to attack you with? This isn't simply a matter of infrastructure, buying the right equipment, or setting things up "just right" precisely because there is a sentient actor trying to hurt you. It's more like a game of chess.
- jiggy2011 14y ago
  If high availability git is so critical then why not run your own git servers instead of or in addition to github? I find it slightly ironic that the entire point of git is that it is distributed version control but 90% of git use seems to be focused around a product from a single company.
  - Serplat 14y ago
    Github isn't solely a git server. They also have issues, wikis, etc, that are harder to keep in multiple locations.
    - Tobu 14y ago
      The wikis are git repos as well. Sync them periodically and you're good. Plus it's useful as a backup, because I don't think github archives your reflog. Issues are normally mirrored to e-mails (caveat: you don't get mail for your own comments), so you can mostly pick up existing threads if your e-mail address book can find the github users involved. If they didn't obscure recipients (at least within an organisation — because I don't think address-book lock-in is worth inconveniencing paying clients), and made an auto self-bcc of your activity, issues would be entirely disaster resistant.
    - 27182818284 14y ago
      If you have GitHub enterprise, it is run on your stuff and you get your own Wikis, issues, unlimited repos, etc. Even your own Gist that you can wrap behind your own CAS.
adgar2 14y ago
Because high availability is for legacy waterfall chumps. Github's too busy being rockstars.

Firehed 14y ago

Easier said than done. We've had DDOS issues in the past as well, and getting it resolved - even by throwing money at the problem - is nontrivial. What amounts to throwing a massive amount of hardware at the problem (i.e., boxes that can handle 10-100+gbps of traffic, filter out the attacks, and pass only legit stuff down to your servers) is expensive[1], and casuses all sorts of unexpected behavior: API clients mysteriously break, good traffic gets mistakenly dropped, latency is added to the whole process, etc. It gets even weirder on SSL-protected sites. And it's all dependent on attackers not getting the IP of your actual servers which they could then just attack directly. [1] For sites with even not a whole lot of traffic, you're talking a one-year contract easily in the range of an engineer's salary. I wouldn't be surprised if the cost to protect sites with as much traffic as Github exceeded $1m/year. Even if you have plenty of cash in the bank, that's one hell of a pill to swallow.

nnnnnnnn 14y ago
Github can easily afford to use someone like Prolexic. And they should. When you say things like "And it's all dependent on attackers not getting the IP of your actual servers" this makes me wonder how much you understand the subject matter. There are many, many options.
- Firehed 14y ago
  Prolexic's servers don't take the load if the attackers know where the computers behind the scrubbers are. Configuring iptables to ignore all traffic not coming from prolexic's IPs doesn't come close to fending off a DDOS. I know this because I was told this by prolexic while configuring our servers to sit behind their scrubbing servers while we're under an equally crippling DDOS (one that took down half the customers in our datacenter, not just us). So while I haven't examined their tech stack under a magnifying glass, I'm not exactly talking out of my ass here. Yes, there are other options but those don't take an hour to implement like signing a contract and changing a few DNS entries does. And when these conditions exist, you need an answer that can be implemented in an hour.
  - 14y ago

mylittlepony 14y ago

Just move to bitbucket. You can even pay to them if you want to.

mattdeboard 14y ago
Yeah, bitbucket never gets DDOS'd! Well, except when they do http://blog.bitbucket.org/2009/10/04/on-our-extended-downtime-amazon-and-whats-coming/ http://blog.bitbucket.org/2009/10/04/on-our-extended-downtim...