14 ms·

IT Pro confession: I contributed to the DDOS attack against Spamhaus

79 points by esalazar 14y ago

laumars 14y ago
I have a couple of name servers I've inherited since starting my job. How would I go about testing these servers to see if they're set up correctly (obviously I'm not interested in the forged UDP packets side of things, only testing to make sure that recursive look ups are disabled).
- jethro_tell 14y ago
  you could do an nslookup google.com <your.name.server.ip> from off network, or check your blocks on http://openresolverproject.org/ http://openresolverproject.org/
- deleted 14y ago
  [deleted]
sciurus 14y ago
"""Let's say that you leave your recursive server open to the internet. Now not only can you ask your DNS server for information about other DNS servers on the internet, so can anyone else. If someone asks your server "where is www.google.com" a whole bunch of times then your server starts flooding google.com's DNS servers. For every 1 byte of data sent to your DNS server 50 bytes of traffic end up directed at the target.""" This explanation is skipping a key component of a DNS reflection attack. When the attacker makes a DNS request, they spoof their source address so it is the address of the host they want to attack. Thus they send a small request to your DNS server, and your DNS server returns a large response not to them, but to the host they're attacking.
- laumars 14y ago
  I think that was implied. However you're right that the author should have made that a little more clear.
- bradleyjg 14y ago
  Why does it need to be recursive then? Couldn't you perform the same attack by querying a whole bunch of authoritative name servers for zones they serve with forged source addresses?
  - entropy_ 14y ago
    The attack as specified by the article would be pretty ineffective as recursive DNS servers tend to have a cache. So only the first request would hit the target DNS server. Also, you can definitely do this with authoritative servers only, which is why only egress filtering by ISPs is a permanent solution to the problem. However, there are way fewer authoritative DNS servers out there than there are open recursors and they are better managed. So an attack would never grow to the scale this one has grown to using only authoritative servers.
    - ricardobeat 14y ago
      You got me confused for a minute. So I assume in an actual attack you'll rotate the domains being requested?
      - entropy_ 14y ago
        No, the attack does not work by overloading target DNS servers. There is no benefit in asking a recursive DNS server to make a DNS request for you to overload another DNS server, you could just make that request yourself(or with whatever botnet you're using). The attack works by sending a recursive DNS server a request with a spoofed source IP. Namely, you make the recursive DNS server think your target is making the request. While a typical DNS query consists of a 64byte UDP packet, a reply can be much much lengthier(it can go well over 1KB). So say you have a botnet with a total bandwidth of 1Gb/s. Each request you make(64bytes) will result in, say, 1KB being sent by the DNS server to your target although the server thinks it is sending it to you. That results in a 16x amplification of the amount of data you are sending the target's way. So instead of flooding your target with 1Gb/s of data, you are flooding it with 16Gb/s of DNS replies. The only permanent solution to this problem(though it is discussed elsewhere in this thread why this is impractical) is for all(or almost all) ISPs to have egress filtering. That is, that they would drop all packets sent from their networks with a source IP that is not on their networks. This would make it impossible to fool a recursive DNS server into sending the reply to the wrong IP. Since this is very hard to do(ISPs have zero incentive to do egress filtering, and we can't even locate the ones from whose networks these attacks are originating to shame them into doing it) the pursued solution is the easier one of locating and closing publicly open DNS recursors. This would still allow DNS amplification attacks using authoritative servers, but they would be much more limited in scope.
- dsl 14y ago
  You should also note that DNSSEC is just as much of a problem as open resolvers. A normal A record lookup results in 1-2x amplification $ dig www.ripe.net. in a | grep SIZE ;; MSG SIZE rcvd: 46 Asking for DNSSEC records specifically yields a 10x+ amplification $ dig www.ripe.net. in RRSIG | grep SIZE ;; MSG SIZE rcvd: 534 According to research by DJB[1] over 2000 DNSSEC enabled zones provide >30x amplification for incoming UDP queries. 1. cr.yp.to/talks/2012.06.04/slides.pdf
  - lambda 14y ago
    But you can easily force regular DNS to give you high amplification. Just query a domain which gives you a larger response, such as multiple A records, MXes, SPF records, and so on. For example: $ dig google.com. ANY | grep SIZE ;; MSG SIZE rcvd: 546 If you want more amplification than that gives, just host one yourself. The recursive resolver will hit your DNS server once, then send out replies based on the cache. There seems to be lots of fearmongering about DNSSEC amplification, but you can get just the same amount of amplification out of regular DNS, so it seems that fixing DNS amplification in other ways would be more effective than trying to avoid adopting DNSSEC.
    - dsl 14y ago
      > There seems to be lots of fearmongering about DNSSEC amplification There is also a 300+ Gbps DDoS attack making use of it right now. This was foreseen as a huge amplification vector in a stateless protocol during the design phase, but was ignored. Now we get to reap the benefits of that decision. Normal DNS responses don't often grow to the size of google.com/IN/ANY. You have a very limited number of authoritative sources to use (and hosting yourself creates a bottleneck as well as a path back to you). With widespread DNSSEC adoption every zone becomes a good amplification source, which nullifies the current best practices for mitigation (rate limiting responses per zone/source). If DNSSEC adoption becomes the norm, open recursive resolvers no longer become the problem and direct to authoritative becomes a viable attack vector.
      - pyre 14y ago
- matt_heimer 14y ago
  Thanks, I was wondering about their explanation. Every DNS server I setup will temporary cache the results of its recursive lookups so I didn't get how this was going to work.
mindstab 14y ago
Whats a simple way to confirm by test your DNS server isn't doing recursion?
- rhizome 14y ago
  A little Googling can often help: http://dnsknowledge.com/bind/howto-test-bind-open-recursive-dns-queries/ http://dnsknowledge.com/bind/howto-test-bind-open-recursive-...
  - mindstab 14y ago
    fantastic, thanks!
- tquai 14y ago
  The Open DNS Resolver Project has a list of 25 million open resolvers. You can query their database for your IP address or up to a /24. Their site also has information on how to reduce or eliminate the problem via a couple options (RRL, BCP38). If you run a BIND resolver, consider switching to unbound. Part of this problem is rooted in BIND combining resolver and authoritative service in one daemon, which IMO "mis-educated" a lot of people. http://openresolverproject.org/ http://openresolverproject.org/
SageRaven 14y ago
How disappointing. I thought it was going to be the story of a fed-up email admin breaking down and DoS'ing one of the scourges of the internet. Blacklists are pure evil, and nothing will ever change my opinion of that. They cause far more problems than they solve. Granted, it's usually by idiot, over-zealous mail admins who block on merely being listed anywhere, rather than by weighted score.
- jff 14y ago
  Agreed. If I had known at the time that there was a DDoS against Spamhaus, I'd have probably joined in against the self-righteous pricks. Block my home server, will you?
  - freehunter 14y ago
    As far as I know, Spamhaus doesn't block anything. They just maintain a list of IPs they see spam-like activity coming from. The actual blocking is implemented by whoever is using Spamhaus for their blacklist. Your beef is with the admins who block everything based on a single blacklist.
    - SageRaven 14y ago
      That's kinda like saying the MPAA ratings systems is ok because my beef is really with large theater chains that refuse to carry non-rated films. Large, entrenched authorities who provide ratings about [insert noun here] have historically been a problem.
      - freehunter 14y ago
        I believe your analogy is correct, I see no problem with it. Theaters do not have to follow the rating system. There are local theaters that will play unrated films, and film festivals that play unrated films. Likewise, while you can find some ESRB ratings on the Apple App Store, Apple doesn't really have to pay attention to those and are free to set their own standards. If a theater chain refuses to play a film because it is unrated, that's not the MPAA's fault. And I never thought I would be defending the MPAA... Sure, large and entrenched authorities pose risks. There's not much that can be done about that as long as they're large and entrenched. The best way to ensure their power stays in check is to try to have their customers put pressure on them to clean up their act. And the end-user (you and me) is not their customer. We are customers of the businesses and organizations implementing their block lists. I understand your frustration, and as security professional I have my own beef with Spamhaus ratings, but the answer to that problem lies in comparing their ratings with those of other organizations and a bit of common sense.
brokentone 14y ago
The post slug is the best part of this article by far: "i_accidentally_the_internet" Full current URL in case it gets updated: http://www.theregister.co.uk/2013/03/28/i_accidentally_the_internet/ http://www.theregister.co.uk/2013/03/28/i_accidentally_the_i...
- alxndr 14y ago
  You'll like the opening image in this, then; #10 on HN home page right now: http://blog.tinfoilsecurity.com/building-a-browser-extension-be-careful-not-t-17787 http://blog.tinfoilsecurity.com/building-a-browser-extension...
  - brokentone 14y ago
    I'm familiar with the meme. The humor isn't in the meme, but in the usage by a professional journalism establishment that is unrelated to the title of the piece.
    - alxndr 14y ago
      Right.
sunyc 14y ago
one of my server got exposed too, it was being queried for ripe.net
ajross 14y ago
This really is a real issue. My home machine was an open recursor for a while too. I set up a dnsmasq installation and forgot to set an "except-interface" to restrict it to the internal network. I even like to think I know this stuff well, but still got burned. I'm sure at the time my security analysis (if I even thought of the externally-facing issue) was "who cares if I expose a caching nameserver with no sensitive content to the rest of the internet?".
unethical_ban 14y ago
Some have suggested that DNS move to TCP, but I don't think that's proper. The nature of DNS lends itself to connectionless, lightweight communication. That said, could the next iteration of DNS implement application-level handshaking? The reason not to do this at layer 4 is because I, in the several minutes of pondering it, think it could break lots of security devices that track connection state across lots of computers in a network. Make some kind of C -> S request C <- S ack C -> S yes C <- S lots of data done C -> S request C <- S ack C -> S no done
- drdaeman 14y ago
  Unfortunately, round-trip time is still important, too. I suspect, almost doubling the DNS request time may cause problems in some cases.
metalruler 14y ago
I don't understand why it's necessary for the server to be open, and have recursion enabled. I run a couple of authoritative name servers and have seen them used for amplification attacks. Sure, it's not as easy as querying every open recursive DNS server you can find for <single_domain_with_huge_sized_reply>.com, but there's still (literally) billions of unique hostnames on the internet which can be resolved "legitimately" via their authoritative name servers. There is no magical config option to prevent this; the only way to block this type of activity is to analyze traffic to find IPs that are repeatedly sending the same [spoofed] request.

79 points by esalazar 14y ago

I have a couple of name servers I've inherited since starting my job. How would I go about testing these servers to see if they're set up correctly (obviously I'm not interested in the forged UDP packets side of things, only testing to make sure that recursive look ups are disabled).

jethro_tell 14y ago
you could do an nslookup google.com <your.name.server.ip> from off network, or check your blocks on http://openresolverproject.org/ http://openresolverproject.org/
deleted 14y ago
[deleted]

sciurus 14y ago

"""Let's say that you leave your recursive server open to the internet. Now not only can you ask your DNS server for information about other DNS servers on the internet, so can anyone else. If someone asks your server "where is www.google.com" a whole bunch of times then your server starts flooding google.com's DNS servers. For every 1 byte of data sent to your DNS server 50 bytes of traffic end up directed at the target.""" This explanation is skipping a key component of a DNS reflection attack. When the attacker makes a DNS request, they spoof their source address so it is the address of the host they want to attack. Thus they send a small request to your DNS server, and your DNS server returns a large response not to them, but to the host they're attacking.

laumars 14y ago
I think that was implied. However you're right that the author should have made that a little more clear.
bradleyjg 14y ago
Why does it need to be recursive then? Couldn't you perform the same attack by querying a whole bunch of authoritative name servers for zones they serve with forged source addresses?
- entropy_ 14y ago
  The attack as specified by the article would be pretty ineffective as recursive DNS servers tend to have a cache. So only the first request would hit the target DNS server. Also, you can definitely do this with authoritative servers only, which is why only egress filtering by ISPs is a permanent solution to the problem. However, there are way fewer authoritative DNS servers out there than there are open recursors and they are better managed. So an attack would never grow to the scale this one has grown to using only authoritative servers.
  - ricardobeat 14y ago
    You got me confused for a minute. So I assume in an actual attack you'll rotate the domains being requested?
    - entropy_ 14y ago
      No, the attack does not work by overloading target DNS servers. There is no benefit in asking a recursive DNS server to make a DNS request for you to overload another DNS server, you could just make that request yourself(or with whatever botnet you're using). The attack works by sending a recursive DNS server a request with a spoofed source IP. Namely, you make the recursive DNS server think your target is making the request. While a typical DNS query consists of a 64byte UDP packet, a reply can be much much lengthier(it can go well over 1KB). So say you have a botnet with a total bandwidth of 1Gb/s. Each request you make(64bytes) will result in, say, 1KB being sent by the DNS server to your target although the server thinks it is sending it to you. That results in a 16x amplification of the amount of data you are sending the target's way. So instead of flooding your target with 1Gb/s of data, you are flooding it with 16Gb/s of DNS replies. The only permanent solution to this problem(though it is discussed elsewhere in this thread why this is impractical) is for all(or almost all) ISPs to have egress filtering. That is, that they would drop all packets sent from their networks with a source IP that is not on their networks. This would make it impossible to fool a recursive DNS server into sending the reply to the wrong IP. Since this is very hard to do(ISPs have zero incentive to do egress filtering, and we can't even locate the ones from whose networks these attacks are originating to shame them into doing it) the pursued solution is the easier one of locating and closing publicly open DNS recursors. This would still allow DNS amplification attacks using authoritative servers, but they would be much more limited in scope.
dsl 14y ago
You should also note that DNSSEC is just as much of a problem as open resolvers. A normal A record lookup results in 1-2x amplification $ dig www.ripe.net. in a | grep SIZE ;; MSG SIZE rcvd: 46 Asking for DNSSEC records specifically yields a 10x+ amplification $ dig www.ripe.net. in RRSIG | grep SIZE ;; MSG SIZE rcvd: 534 According to research by DJB[1] over 2000 DNSSEC enabled zones provide >30x amplification for incoming UDP queries. 1. cr.yp.to/talks/2012.06.04/slides.pdf
- lambda 14y ago
  But you can easily force regular DNS to give you high amplification. Just query a domain which gives you a larger response, such as multiple A records, MXes, SPF records, and so on. For example: $ dig google.com. ANY | grep SIZE ;; MSG SIZE rcvd: 546 If you want more amplification than that gives, just host one yourself. The recursive resolver will hit your DNS server once, then send out replies based on the cache. There seems to be lots of fearmongering about DNSSEC amplification, but you can get just the same amount of amplification out of regular DNS, so it seems that fixing DNS amplification in other ways would be more effective than trying to avoid adopting DNSSEC.
  - dsl 14y ago
    > There seems to be lots of fearmongering about DNSSEC amplification There is also a 300+ Gbps DDoS attack making use of it right now. This was foreseen as a huge amplification vector in a stateless protocol during the design phase, but was ignored. Now we get to reap the benefits of that decision. Normal DNS responses don't often grow to the size of google.com/IN/ANY. You have a very limited number of authoritative sources to use (and hosting yourself creates a bottleneck as well as a path back to you). With widespread DNSSEC adoption every zone becomes a good amplification source, which nullifies the current best practices for mitigation (rate limiting responses per zone/source). If DNSSEC adoption becomes the norm, open recursive resolvers no longer become the problem and direct to authoritative becomes a viable attack vector.
    - pyre 14y ago
matt_heimer 14y ago
Thanks, I was wondering about their explanation. Every DNS server I setup will temporary cache the results of its recursive lookups so I didn't get how this was going to work.

mindstab 14y ago

Whats a simple way to confirm by test your DNS server isn't doing recursion?

rhizome 14y ago
A little Googling can often help: http://dnsknowledge.com/bind/howto-test-bind-open-recursive-dns-queries/ http://dnsknowledge.com/bind/howto-test-bind-open-recursive-...
- mindstab 14y ago
  fantastic, thanks!
tquai 14y ago
The Open DNS Resolver Project has a list of 25 million open resolvers. You can query their database for your IP address or up to a /24. Their site also has information on how to reduce or eliminate the problem via a couple options (RRL, BCP38). If you run a BIND resolver, consider switching to unbound. Part of this problem is rooted in BIND combining resolver and authoritative service in one daemon, which IMO "mis-educated" a lot of people. http://openresolverproject.org/ http://openresolverproject.org/

SageRaven 14y ago

How disappointing. I thought it was going to be the story of a fed-up email admin breaking down and DoS'ing one of the scourges of the internet. Blacklists are pure evil, and nothing will ever change my opinion of that. They cause far more problems than they solve. Granted, it's usually by idiot, over-zealous mail admins who block on merely being listed anywhere, rather than by weighted score.

jff 14y ago
Agreed. If I had known at the time that there was a DDoS against Spamhaus, I'd have probably joined in against the self-righteous pricks. Block my home server, will you?
- freehunter 14y ago
  As far as I know, Spamhaus doesn't block anything. They just maintain a list of IPs they see spam-like activity coming from. The actual blocking is implemented by whoever is using Spamhaus for their blacklist. Your beef is with the admins who block everything based on a single blacklist.
  - SageRaven 14y ago
    That's kinda like saying the MPAA ratings systems is ok because my beef is really with large theater chains that refuse to carry non-rated films. Large, entrenched authorities who provide ratings about [insert noun here] have historically been a problem.
    - freehunter 14y ago
      I believe your analogy is correct, I see no problem with it. Theaters do not have to follow the rating system. There are local theaters that will play unrated films, and film festivals that play unrated films. Likewise, while you can find some ESRB ratings on the Apple App Store, Apple doesn't really have to pay attention to those and are free to set their own standards. If a theater chain refuses to play a film because it is unrated, that's not the MPAA's fault. And I never thought I would be defending the MPAA... Sure, large and entrenched authorities pose risks. There's not much that can be done about that as long as they're large and entrenched. The best way to ensure their power stays in check is to try to have their customers put pressure on them to clean up their act. And the end-user (you and me) is not their customer. We are customers of the businesses and organizations implementing their block lists. I understand your frustration, and as security professional I have my own beef with Spamhaus ratings, but the answer to that problem lies in comparing their ratings with those of other organizations and a bit of common sense.

brokentone 14y ago

The post slug is the best part of this article by far: "i_accidentally_the_internet" Full current URL in case it gets updated: http://www.theregister.co.uk/2013/03/28/i_accidentally_the_internet/ http://www.theregister.co.uk/2013/03/28/i_accidentally_the_i...

alxndr 14y ago
You'll like the opening image in this, then; #10 on HN home page right now: http://blog.tinfoilsecurity.com/building-a-browser-extension-be-careful-not-t-17787 http://blog.tinfoilsecurity.com/building-a-browser-extension...
- brokentone 14y ago
  I'm familiar with the meme. The humor isn't in the meme, but in the usage by a professional journalism establishment that is unrelated to the title of the piece.
  - alxndr 14y ago
    Right.

sunyc 14y ago

one of my server got exposed too, it was being queried for ripe.net

ajross 14y ago

This really is a real issue. My home machine was an open recursor for a while too. I set up a dnsmasq installation and forgot to set an "except-interface" to restrict it to the internal network. I even like to think I know this stuff well, but still got burned. I'm sure at the time my security analysis (if I even thought of the externally-facing issue) was "who cares if I expose a caching nameserver with no sensitive content to the rest of the internet?".

unethical_ban 14y ago

Some have suggested that DNS move to TCP, but I don't think that's proper. The nature of DNS lends itself to connectionless, lightweight communication. That said, could the next iteration of DNS implement application-level handshaking? The reason not to do this at layer 4 is because I, in the several minutes of pondering it, think it could break lots of security devices that track connection state across lots of computers in a network. Make some kind of C -> S request C <- S ack C -> S yes C <- S lots of data done C -> S request C <- S ack C -> S no done

drdaeman 14y ago
Unfortunately, round-trip time is still important, too. I suspect, almost doubling the DNS request time may cause problems in some cases.

metalruler 14y ago

I don't understand why it's necessary for the server to be open, and have recursion enabled. I run a couple of authoritative name servers and have seen them used for amplification attacks. Sure, it's not as easy as querying every open recursive DNS server you can find for <single_domain_with_huge_sized_reply>.com, but there's still (literally) billions of unique hostnames on the internet which can be resolved "legitimately" via their authoritative name servers. There is no magical config option to prevent this; the only way to block this type of activity is to analyze traffic to find IPs that are repeatedly sending the same [spoofed] request.