5 ms·

The second in two days, github is getting increasingly unreliable - https://status.github.com/messages https://status.github.com/messages. Also, I'd be interes

The second in two days, github is getting increasingly unreliable - https://status.github.com/messages https://status.github.com/messages. Also, I'd be interested to know how complete lack of service is 'mitigating a DDOS attack' - to me it sounds like 'successful DDOS attack'.

kevingadd 13y ago
For a large DDOS attack there aren't any easy ways to only drop the DDOS traffic - especially if it's hard to identify the DDOS traffic in the first place. If they're getting more 'bad' incoming traffic than their connections can handle I don't know how they'd drop that stuff - they have to receive the packets before they can filter them. Maybe their bandwidth provider has tricks for this...
- regecks 13y ago
  There are certainly providers that have networks you can hide behind which can filter+absorb DDoS traffic for you.
  - daenney 13y ago
    That doesn't work in all cases. If you can't distinguish between good and bad traffic or if it isn't specifically targeting an entity it becomes much more difficult to handle. It also depends on the ingenuity of the DDoS attack, none of which are known to the public so you can't really say anything sensible about it. If the anti-DDoS mitigating tools they are using aren't working nor is using other services like Prolexic that's usually hint enough that this isn't particularly common or easily filtered out. You can gripe on all the companies and tools if you will but a good DDoS is quite a bit more complex than 'just filter away the bad crap'.
    - Terretta 13y ago
      > you can't really say anything sensible about it See #4 of the 12 timeless networking truths: (4) Some things in life can never be fully appreciated nor understood unless experienced firsthand. Some things in networking can never be fully understood by someone who neither builds commercial networking equipment nor runs an operational network. -- http://highscalability.com/blog/2013/10/2/rfc-1925-the-twelve-timeless-networking-truths.html http://highscalability.com/blog/2013/10/2/rfc-1925-the-twelv...
- corobo 13y ago
  They're using Prolexic by the looks of things.. You'd think a company that specializes in mitigating DDoS attacks would be able to mitigate DDoS attacks. Maybe I'm just misunderstanding the word mitigate
  - spongle 13y ago
    Security companies are all snakeoil. They never deliver from experience. They always bill though.
    - api 13y ago
      Infosec in general is a ghetto of navel-gazers.
      - freehunter 13y ago
        In response to your comment and the parent comment, from someone in InfoSec: security can never be completed in the way a product can be. It's an ongoing war, and sometimes your opponent gets the upper hand for a while. The problem with being the "good guys" in security is that you have to make sure every hole is closed while still letting the business run. It's easier to be the bad guy, because you just have to find one thing the security team missed. Security doesn't exist without the business and the business doesn't exist without security, but the business tends to trump security for the sake of features and convenience. It's a very delicate see-saw, and all you can really do is trying to run back and forth from side to side hoping that the other end doesn't hit the ground before you can get over there again. Attitudes like yours don't help a damn thing.
        spongle 13y ago
        Actually as a solution architect I have to deal with all sides of the problems: people attacking, audit companies, penetration testing companies and software engineers leaving gaping holes. The only people who deliver little value are the paid up consultants. When a full penetration and code review misses 4 purposely placed obvious vulnerabilities (by myself) they get told to fuck off. Application firewalls which are circumvented trivially. QoS solutions that don't work. So far, four well known, well respected companies offering certification and testing have missed the holes and have been fired. That's the problem: no delivery. My attitude might be wrong in your eyes but I refuse to employ box tickers which is what the entire white hat side of the industry is about. Canned report, where's my cheque? No seesaw other than a bent twisted one that sucks up cash in exchange for a half arsed job.
- alistairjcbrown 13y ago
  Couldn't they use something like cloudflare to have the IP point to local servers? Then the traffic is split on location, with each edge server taking only local requests. That should greatly reduce the incoming traffic, at which point they can try to filter out the 'bad'.
  - TheSwordsman 13y ago
    You assume that the attack was limited to their web stack. For all we know it could be the systems that handle the git-over-ssh connections. I may not be too versed in CloudFlare, but I didn't think they would be able to protect a service like SSH.
    - lrem 13y ago
      Getting things through https definitely did not work for some time. I thought I broke my vundle.
    - druiid 13y ago
      Indeed you would not be able to. CloudFlare only does HTTP/HTTPS right now. Technically since Nginx can also support SMTP they should be able to do that as well but it's not implemented currently. Basically if you want to protect SSH it would have to be a provider that does layer-level protection like Prolexic.
    - tedunangst 13y ago
      Well, cloud flare can certainly forward (or not forward, in the case of bad) ssh traffic. But they would need to dedicate an IP to your account, or provide you with a port number to use.
- utnick 13y ago
  Is it feasable to firewall whitelist any IP that has ever pushed to a repo?
  - jcfiala 13y ago
    You could, but now you're just forcing the attackers to fork an existing project and push some nonsense code to it before they can attack.
  - astrodust 13y ago
    Why would you want to cut off all people that use only the web front-end?
nevinera 13y ago
Two hours ago, they were down constantly, but now they seem to be up. Github seems to be getting increasingly reliable! You can always pick an appropriate window of time, point to it, and say "See, there's a trend!". That doesn't make it so. Github used to go down much more often than they do now - calling them 'increasing unreliable' really just shows that you have begun depending on it more heavily.
- tedunangst 13y ago
  Maybe increasingly unreliable is the wrong term. What about sufficiently unreliable?
  - Cthulhu_ 13y ago
    Their status page [1] still indicates a 99.85% uptime in the past month, and before this and yesterday's problem, their status page was mostly green across the board for a couple of weeks. It depends on your requirements, really. Nobody can guarantee 100% uptime. [1] https://status.github.com/graphs/past_month https://status.github.com/graphs/past_month
  - nevinera 13y ago
    'Sufficiently' is meaningless without a qualifier. Sufficiently for what purpose? There is a value proposition involved - you can run your own source code hosting and anything else, but it costs money and time to do it. Especially if you need six nines of uptime.
chernevik 13y ago
Would it help if they allowed users to specify some port other than the default, perhaps only as a fallback? I am presuming that it would be easier for them to prioritize traffic to a given range of ports -- in this instance the NON standard ones -- but perhaps that's wrong. If so, I wonder if they could counter this now by 1) implementing that 2) opening and publishing a not-standard port for login solely for that purpose 3) maybe moving that change-the-default port around if the DDOS shifts on to it. Even if not-perfect they'd force the attack to spread resources.

kevingadd 13y ago

For a large DDOS attack there aren't any easy ways to only drop the DDOS traffic - especially if it's hard to identify the DDOS traffic in the first place. If they're getting more 'bad' incoming traffic than their connections can handle I don't know how they'd drop that stuff - they have to receive the packets before they can filter them. Maybe their bandwidth provider has tricks for this...

regecks 13y ago
There are certainly providers that have networks you can hide behind which can filter+absorb DDoS traffic for you.
- daenney 13y ago
  That doesn't work in all cases. If you can't distinguish between good and bad traffic or if it isn't specifically targeting an entity it becomes much more difficult to handle. It also depends on the ingenuity of the DDoS attack, none of which are known to the public so you can't really say anything sensible about it. If the anti-DDoS mitigating tools they are using aren't working nor is using other services like Prolexic that's usually hint enough that this isn't particularly common or easily filtered out. You can gripe on all the companies and tools if you will but a good DDoS is quite a bit more complex than 'just filter away the bad crap'.
  - Terretta 13y ago
    > you can't really say anything sensible about it See #4 of the 12 timeless networking truths: (4) Some things in life can never be fully appreciated nor understood unless experienced firsthand. Some things in networking can never be fully understood by someone who neither builds commercial networking equipment nor runs an operational network. -- http://highscalability.com/blog/2013/10/2/rfc-1925-the-twelve-timeless-networking-truths.html http://highscalability.com/blog/2013/10/2/rfc-1925-the-twelv...
corobo 13y ago
They're using Prolexic by the looks of things.. You'd think a company that specializes in mitigating DDoS attacks would be able to mitigate DDoS attacks. Maybe I'm just misunderstanding the word mitigate
- spongle 13y ago
  Security companies are all snakeoil. They never deliver from experience. They always bill though.
  - api 13y ago
    Infosec in general is a ghetto of navel-gazers.
    - freehunter 13y ago
      In response to your comment and the parent comment, from someone in InfoSec: security can never be completed in the way a product can be. It's an ongoing war, and sometimes your opponent gets the upper hand for a while. The problem with being the "good guys" in security is that you have to make sure every hole is closed while still letting the business run. It's easier to be the bad guy, because you just have to find one thing the security team missed. Security doesn't exist without the business and the business doesn't exist without security, but the business tends to trump security for the sake of features and convenience. It's a very delicate see-saw, and all you can really do is trying to run back and forth from side to side hoping that the other end doesn't hit the ground before you can get over there again. Attitudes like yours don't help a damn thing.
      - spongle 13y ago
        Actually as a solution architect I have to deal with all sides of the problems: people attacking, audit companies, penetration testing companies and software engineers leaving gaping holes. The only people who deliver little value are the paid up consultants. When a full penetration and code review misses 4 purposely placed obvious vulnerabilities (by myself) they get told to fuck off. Application firewalls which are circumvented trivially. QoS solutions that don't work. So far, four well known, well respected companies offering certification and testing have missed the holes and have been fired. That's the problem: no delivery. My attitude might be wrong in your eyes but I refuse to employ box tickers which is what the entire white hat side of the industry is about. Canned report, where's my cheque? No seesaw other than a bent twisted one that sucks up cash in exchange for a half arsed job.
alistairjcbrown 13y ago
Couldn't they use something like cloudflare to have the IP point to local servers? Then the traffic is split on location, with each edge server taking only local requests. That should greatly reduce the incoming traffic, at which point they can try to filter out the 'bad'.
- TheSwordsman 13y ago
  You assume that the attack was limited to their web stack. For all we know it could be the systems that handle the git-over-ssh connections. I may not be too versed in CloudFlare, but I didn't think they would be able to protect a service like SSH.
  - lrem 13y ago
    Getting things through https definitely did not work for some time. I thought I broke my vundle.
  - druiid 13y ago
    Indeed you would not be able to. CloudFlare only does HTTP/HTTPS right now. Technically since Nginx can also support SMTP they should be able to do that as well but it's not implemented currently. Basically if you want to protect SSH it would have to be a provider that does layer-level protection like Prolexic.
  - tedunangst 13y ago
    Well, cloud flare can certainly forward (or not forward, in the case of bad) ssh traffic. But they would need to dedicate an IP to your account, or provide you with a port number to use.
utnick 13y ago
Is it feasable to firewall whitelist any IP that has ever pushed to a repo?
- jcfiala 13y ago
  You could, but now you're just forcing the attackers to fork an existing project and push some nonsense code to it before they can attack.
- astrodust 13y ago
  Why would you want to cut off all people that use only the web front-end?

nevinera 13y ago

Two hours ago, they were down constantly, but now they seem to be up. Github seems to be getting increasingly reliable! You can always pick an appropriate window of time, point to it, and say "See, there's a trend!". That doesn't make it so. Github used to go down much more often than they do now - calling them 'increasing unreliable' really just shows that you have begun depending on it more heavily.

tedunangst 13y ago
Maybe increasingly unreliable is the wrong term. What about sufficiently unreliable?
- Cthulhu_ 13y ago
  Their status page [1] still indicates a 99.85% uptime in the past month, and before this and yesterday's problem, their status page was mostly green across the board for a couple of weeks. It depends on your requirements, really. Nobody can guarantee 100% uptime. [1] https://status.github.com/graphs/past_month https://status.github.com/graphs/past_month
- nevinera 13y ago
  'Sufficiently' is meaningless without a qualifier. Sufficiently for what purpose? There is a value proposition involved - you can run your own source code hosting and anything else, but it costs money and time to do it. Especially if you need six nines of uptime.

chernevik 13y ago

Would it help if they allowed users to specify some port other than the default, perhaps only as a fallback? I am presuming that it would be easier for them to prioritize traffic to a given range of ports -- in this instance the NON standard ones -- but perhaps that's wrong. If so, I wonder if they could counter this now by 1) implementing that 2) opening and publishing a not-standard port for login solely for that purpose 3) maybe moving that change-the-default port around if the DDOS shifts on to it. Even if not-perfect they'd force the attack to spread resources.