12 ms·
February 28th DDoS Incident Report
- hyperpower 9y agoWow, 1.35Tbps? That's a lot for a DoS attack, right?
- flafla2 9y agoAccording to [0] that is around 1/400th of total internet traffic per second. This begs the question: who has that kind of botnet at their disposal and why are they targeting Github? Edit: The attacker didn't need nearly that kind of bandwidth to execute this attack. See [1] Edit: 1/50th -> 1/400th (bits vs bytes) [0] http://www.internetlivestats.com/one-second/#traffic-band http://www.internetlivestats.com/one-second/#traffic-band [1] https://news.ycombinator.com/item?id=16493497 https://news.ycombinator.com/item?id=16493497
- voltagex_ 9y agomemcached reflection: https://blogs.akamai.com/2018/03/memcached-fueled-13-tbps-attacks.html https://blogs.akamai.com/2018/03/memcached-fueled-13-tbps-at...
- toomuchtodo 9y agoWho is exposing their memcached instances on the public internet? Who is not filtering outbound UDP traffic from their memcached instances? Rhetorical of course. Akamai should have logs of their offenders. Off to scan for offenders and notify their providers!
- theojulienne 9y agoDuring some analysis we did notice that at least some cloud providers default to having instances with public IPs (with no network-level ACLs) by default, and some Linux distributions default to having memcached listening for UDP traffic and binding to `0.0.0.0` by default as soon as it's installed. The unfortunate combination of these result in the machine being vulnerable to being used as an amplification vector in these attacks.
- toomuchtodo 9y agoDid you receive any cooperation from those cloud providers in using ACLs at their network edge to drop that traffic (that they should’ve been blocking in the first place)?
- snuxoll 9y agoThis is one area where I really disagree with Debian and derivatives default behavior of starting a service immediately after installation and not having a firewall enabled by default. If I install a service on CentOS/RHEL/Fedora it is disabled by default, if I start the service firewalld will block traffic until I have explicitly enabled a rule to allow it (or explicitly stopped and disabled the firewalld service). Does this prevent people from making poor decisions, like just blindly starting the service without reading the configuration file, or disabling firewalld/enabling a rule without checking the configuration first? No, it doesn't - but that small hurdle at least prevents people from inadvertently turning on a service and opening it up to the world just by installing a package.
- vbernat 9y agoAs a compromise, Debian also comes with a secure-by-default configuration. In the case of memcached, the service is configured to listen on 127.0.0.1.
- snuxoll 9y agoThankfully that is one thing they do pretty well in most cases, though there are some (apache2, for example) that do listen on all interfaces by default. While even services like Apache may have secure configurations by default, it can often be installed by other programs that link or copy their own configuration files into the apache config directory - and then all you need is a PHP vulnerability or whatever.
- toast0 9y ago> Who is not filtering outbound UDP traffic from their memcached instances? This is of course the wrong way to do it -- you need to filter inbound UDP to your memcached instances so you don't waste your resources generating the responses, and also so you don't accidentally fragment the responses and only drop the first fragment outbound.
- toomuchtodo 9y agoI disagree, due to seperation of responsibilities. Having run both an ISP and a hosting company, you have to filter traffic at your edge that can impact external resources (just as ISPs block outbound NetBios and SMTP traffic on port 25/tcp). Yes, the server or instance customer should be doing this. But they’re not, because poor security practices are an externality, not a cost they sustain. Security is more important than developer velocity, but users pay the bills.
- hueving 9y agoYour confused if you think it's the clouds that are misconfigured here. The issue is the ISPs allowing the spoofed traffic going towards the memcached servers.
- toomuchtodo 9y agoIf you’re a service provider allowing your equipment to participate in an amplification attack, you’re the fool trashing the commons. It’s an ISPs job to filter outbound udp on arbitrary ports? Shall we only let 443 tcp outbound from eyeball networks?
- hueving 9y agoAny UDP service can be used in an amplification attack. It's not the responsibility of AWS or other hosting companies to shut down all UDP traffic. The problem is the ISPs allowing spoofed IPs.
- 9y ago
- flafla2 9y agoI see. For anyone else who doesn't have any background in this attack: memcached is an open source general purpose cache that uses sockets to cache data. From what I gather, the attack here was possible because Github engineers accidentally left the memcached port open. So the attackers were able to spam memcached with large requests, and memcached responds immediately with the full contents of the cached memory (assuming, of course, that the client is localhost). > The memcache protocol was never meant to be exposed to the Internet, but there are currently more than 50,000 known vulnerable systems exposed at the time of this writing. By default, memcached listens on localhost on TCP and UDP port 11211 on most versions of Linux, but in some distributions it is configured to listen to this port on all interfaces by default. Yikes!
- tlunter 9y agoI don't think it was GitHub's memcached instances. It was other public instances that with spoofed network requests ended up sending traffic back towards GitHub's network.
- kevinconaway 9y ago> From what I gather, the attack here was possible because Github engineers accidentally left the memcached port open. That is incorrect. The attackers made requests that were forged to have the sender IP address of Github to multiple public memcached instances. Memcached then responds back to Github instead of the attacker. This is documented in more detail in the Cloudflare vulnerability report[0] https://blog.cloudflare.com/memcrashed-major-amplification-attacks-from-port-11211/ https://blog.cloudflare.com/memcrashed-major-amplification-a...
- sarreph 9y agoI assume this explains the apparent hugeness of the attack: > The vulnerability via misconfiguration described in the post is somewhat unique amongst that class of attacks because the amplification factor is up to 51,000, meaning that for each byte sent by the attacker, up to 51KB is sent toward the target.
- Thaxll 9y agoIt's in bit/sec not byte/sec so 8x lower.
- randall 9y agoProbably to get press. If you have a giant botnet and get press for it, it's probably easier to get clients.
- deleted 9y ago[deleted]
- sneak 9y agohttps://begthequestion.info https://begthequestion.info
- gjtorikian 9y agoIt's the largest recorded attack: https://www.wired.com/story/github-ddos-memcached https://www.wired.com/story/github-ddos-memcached
- jgrahamc 9y agoYes and there are a lot of attacks of very, very large sizes going on. Over the last few days we've mitigated some huge attacks. Luckily, everyone is working together to rate limit and clean up this problem.
- paulie_a 9y agoThis is honestly why I have no problem with things like brickerbot. I consider it community service.
- crescentfresh 9y agoLinked article for the memcached explanation: https://blog.cloudflare.com/memcrashed-major-amplification-attacks-from-port-11211/ https://blog.cloudflare.com/memcrashed-major-amplification-a... From what I understand the attack originates from publicly exposed memcached servers configured to support udp and that have no authentication requirements: - put a large object in a key - construct a memcached "get" request for that key - forge the IP address of the udp request to point to that of the target/victim server - memcached sends the large object to the target/victim Multiply times thousands of exposed memcached servers. That about right?
- Proven 9y agoLast time a major DDoS disruption happened it seems the likely reason was anti China gov content on Github. Did someone host popular anti-Xi content on Github?
- r1ch 9y agoThis is a great example of why it's important to pick secure defaults when writing software, especially software that is often deployed on high bandwidth servers or cloud instances. If no listening interfaces are specified then the default should be to exit with an error, not listen on everything! I also wonder if you can store something in a memcached cache that looks like a valid request, then reflect that with the source IP of another memcached server and let them burn each other out...
- arkadiyt 9y agoShortly after Cloudflare's blog post, memcached pushed a commit that disabled UDP by default: https://github.com/memcached/memcached/commit/dbb7a8af90054bf4ef51f5814ef7ceb17d83d974 https://github.com/memcached/memcached/commit/dbb7a8af90054b...
- chrisper 9y agoThis is why I dislike that Ubuntu starts services by default after installing them.
- corndoge 9y agoVery cool that they are able to change BGP advertisements from ChatOps, achieve convergence and mitigate the attack in all of 4 minutes, that is some insane engineering.
- giodamelio 9y agoI had a similar reaction. I had to double check the timestamps when I first read them. That this was all handled so fast is extremely impressive to me.
- scurvy 9y agoMeh, or just block UDP to your networks that have no reason to run UDP. Every carrier will do upstream ACL's these days. 5 years ago that wasn't the case. These days, they all do. Some free, some charge. re: chat ops vs a web page. It's just a single BGP advertisement -- big whoop. Chatops is just hipster famous right now.
- mirashii 9y agoA snarky reply like this comes up every time there's discussion of a DDOS, but it ignores the fact that there is some point that has to filter that UDP traffic, and if that point is saturated, the DDOS still worked. Mitigating attacks of this size isn't a firewall rule or a support ticket with your ISP.
- scurvy 9y agoSnark or not, the traffic is filtered upstream before your handoff. If you pick your carriers well, there's not a problem. Many carriers have turned upstream filters into a product. NTT's DPS Lite springs to mind. This just comes down to experience and knowing how to build a network. I'd think that Github would have people knowing how to architect this. They've been through a few DDoS before. Edit: It looks like Github uses NTT for traffic. Hello Github Netops person, you need to call your sales rep and turn on DPS Lite. It's like $100 per 10gig port and you get full ACLs. Telia, another one of your carriers, will do this too. At least they have for me. Level3 though? Lol kick that sorry network to the curb Also, get another /22 allocation so you can at least separate out your DC-origin traffic from your customer traffic.
- always_good 9y agoDDoS is a reminder of how broken the internet is. How many times are we going to see the HN comment that says "lol why do so many people use Cloudflare? I don't need it for my blog!" Naive decentralization (naive trust) doesn't work.
- Santosh83 9y agoNow that we are moving away from net neutrality, can we not get ISPs to do DDOS protection so that we don't need specialised services like Cloudflare to be layered on top of simple sites?
- tachyoff 9y agoISPs absolutely could, but having worked near this space previously, it really isn't as easy as it sounds, both the detection and the mitigation, and ISPs are not particularly equipped to handle it themselves right now. There's a lot of money to be made there, though.
- sofaofthedamned 9y agoOVH do, though i've never used it as I ceased being a customer a while ago. There are still a zillion low-end bottom feeding web hosts who wouldn't do anything about this, though.
- always_good 9y agoOVH only employs some heuristics. Which are next to useless because a percentage of a volumetric attack is still a volumetric attack.
- codefined 9y agoBeing naive here, wouldn't a massive help be to not focus on detection of DoS/DDoS attacks but instead to focus on validating that IP addresses come from within the range of addresses being served by the ISP? It strikes me that this would prevent a massive number of amplification attacks.
- dec0dedab0de 9y agoIs there any legitimate reason to spoof a source IP? I don't think there is, why don't ISPs block any traffic with a source IP that isn't in their network. And then the rest of us block any ISPs that don't do that.
- uw_rob 9y agoTragedy of the commons situation. It is advantageous for an individual ISP to wait out for the other ISPs to block source IP spoofing.
- deathanatos 9y agoNote everyone seems to agree: https://www.internetsociety.org/blog/2014/07/anti-spoofing-bcp-38-and-the-tragedy-of-the-commons/ https://www.internetsociety.org/blog/2014/07/anti-spoofing-b... If I understand the article's point, essentially, carriers pay for the egress traffic that causes DDoSes, that cost and the cost of the generated ill-will outweighs that of filtering, whose price has fallen and continues to fall. Personally, I think that if the article author is correct, then I wonder if this is one of those high-level long-term decisions that companies appear absolutely incapable of making. (In my experience, short-term gains are way overvalued at the cost of long-term loss, generally, especially when it is hard to directly determine the costs/benefits involved.)
- TimWolla 9y agoThere even is a dedicated homepage for that problem of providers not filtering their egress traffic: http://www.bcp38.info/index.php/Main_Page http://www.bcp38.info/index.php/Main_Page
- dboreham 9y agoA couple of reasons: 1. It may be difficult/expensive to arrange for the correct set of source subnets to be available at the points where filtering needs to be done. Motivation to perform egress filtering fails to overcome this cost threshold. 2. Fear that some customers are actually (probably without realizing) relying on alien source address traffic being routed. Therefore filtering that traffic would result in unhappy customers and support workload. In our network over the years I've come across several instances where it turned out we were (erroneously) relying on one of our upstream providers routing traffic with source IP from another provider's network. Since policy-based source IP selection on outbound traffic is quite tricky to setup and get right, I can imagine that ISPs would take the easy way out and just pass the traffic.
- yinyang_in 9y agoAwesome engineering work (y)
- koolba 9y agoWhat’s the point of a DDoS on GitHub anyway? Extortion or pure malice? I don’t see them paying a ransom to stop it so why bother?
- shuntress 9y agoAs another comment points out: If you are trying to sell your DDoS services "Hey watch me bring down github" is a pretty powerful sales tool.
- deleted 9y ago[deleted]
- paulgrimes1 9y agoSide observation: kudos to Sam Kottler for level-headed acknowledgement of the business impact of an incident like this to Github’s clientele, and appearing to own it. Well done, sir
- _RPM 9y agoHere is the commit to disable UDP by default https://github.com/memcached/memcached/commit/dbb7a8af90054bf4ef51f5814ef7ceb17d83d974 https://github.com/memcached/memcached/commit/dbb7a8af90054b... The default should also be changed to only listen on the loopback device.
- bhauer 9y agoAm I old-fashioned to raise an eyebrow when I discover that Memcached servers are running visible to the public Internet? This strikes me as approximately as bizarre as having a database server that accepts connections from the public Internet. In my day, such back-end services were either simply not connected to the Internet (connected via a private network to the application services), firewalled, or at the very least, configured to listen for and respond exclusively to connections from known front-end or application services. Is this sort of deployment architecture falling out of favor? My casual observation is that cloud architectures—at least the ones I've seen employed by small organizations—are more comfortable than I am with services running with public IPs. What is going on? Am I misunderstanding this in some way?
- smudgymcscmudge 9y agoIt is very much out of favor. But it just takes a few dozen misconfigured servers with big pipes to launch an attack like this.
- tetha 9y agoIt's simpler to just click services on AWS and get a public IP to connect to. Drop-policy Firewalls like AWS security groups are hard to configure and debug. Managing network interfaces and binding to specific interfaces instead of others is hard and causes hanging connections. Those are the excuses I dealt with when I took over the current IT department. By now, only haproxy accepts public connections. Everything else is firewalled to the office at most.
- smsm42 9y agoI wonder if it's time for providers like Amazon to provide configs by default that block all ports besides TCP 22, 80 and 443. You want to do other stuff? Configure a firewall. Don't know how? Hire somebody who does. This scenario with cheap insecure things being put out on the internet repeats again and again. IoT, PaaS, etc.
- deoxxa 9y ago
- vannevar 9y agoThese attacks are often described as denial of service attacks, but I wonder if many of them aren't employed as cover for an intrusion attempt. Is it possible that intrusive traffic could be mixed in with such an attack?
- Benjammer 9y agoA DoS attack is, by literal definition, an attempt to overwhelm a host until it is forced to _deny service_ to valid user requests. Are there intrusion techniques that both bring down the server and break into it at the same time? I'm not a security expert, but that doesn't seem like it makes a whole lot of sense to me.
- vannevar 9y agoMaybe in the milliseconds between packet swarms, or immediately before or after? Just seems like a lot of resources to pour into an attack that was defeated in a few minutes. To what end?
- w0rd-driven 9y agoThere's validity to the approach of sending packet swarms to cover intrusion attempts but the traffic levels were more than a small amount of cover. It is possible that it was designed as a smokescreen and someone's calculations were wildly incorrect. No one's safe from off by one errors :>
- vlan0 9y agoI’d be willing to bet it was a “test”. Like what we saw with the mirai botnet against Kreb’s blog.
- erikrothoff 9y agoWhat does an incident like this cost to Github in terms of the extra capacity added? I guess the potential loss of business is way higher, but still very curious about the magnitude.
- r4um 9y agoBGP Sim during the changes https://stat.ripe.net/widget/bgplay#w.ignoreReannouncements=false&w.resource=AS36459&w.starttime=1519838365&w.endtime=1519839625&w.rrcs=0%2C13%2C16&w.instant=null&w.type=bgp https://stat.ripe.net/widget/bgplay#w.ignoreReannouncements=...