5 ms·

"Trickle" attack on Minecraft servers, or Don't Use Thread-Per-Connection

44 points by AndyKelley 16y ago

rdtsc 16y ago
One thread per connection might not scale but in this case other things could be done to mitigate the potential problem before re-writing the whole server. Perhaps use a special encoding for the authentication that doesn't use unbounded strings for username but a fixed size array of chars? Then maybe start a timer and if authentication isn't performed during a given time period, disconnect that client.
- blake8086 16y ago
  Say you kill connections after 1 second (pretty aggressive) and you can hold 1,000 connections open (that's a fair number of threads) All I have to do is create 1,000 connections per second and I'll hog all your thread resources. That's not too hard to do.
  - wtallis 16y ago
    Killing connections after a full second has elapsed probably isn't at all aggressive in the context of a game server. If the connection is legit but takes a full second to send authentication info, then there's no hope for the game actually being playable over that connection.
    - kpreid 16y ago
      Depends on the game. Minecraft is hardly twitchy, especially if you're building, not fighting; the client proceeds as if each action has already happened, so you don't have to wait for the roundtrip between actions.
  - rdtsc 16y ago
    1 second might not be too agressive for initial log-in -- that is where the real problem he is trying to solve is. If the username + password are not unbounded strings, but say, fit in an array of a 100 chars, then you should expect to get that data a lot sooner than 1 second (maybe 200ms?). So then maybe you can handle 5K connections... Yeah you are right, this is still not as good as a good async framework but the point was that some smaller things can be tried before a complete re-write. It might just be good enough.
- MostAwesomeDude 16y ago
  Nobody has really publicly talked about it, but there is now code in the Notchian and CraftBukkit servers to limit connections from single IPs to once every 5 seconds, and to disconnect slow logins by killing the thread after 30 seconds, effectively limiting the attack to 6 threads used per attacker. To actually bring down a server this way would then take 100 attackers or so. This really is just a proof of concept, after all, and not a tool which can bring down entire servers single-handedly. (At least, not anymore.)
bartwe 16y ago
Exhausting ports or generating a lot of TIME_WAIT states would also cause similar problems. How to protect against those ?
- bnoordhuis 16y ago
  You'll be hard-pressed to exhaust all ports: modern operating systems track connections by source address + source port + target address + target port. I wouldn't be surprised if the TCP sequence number is also part of the mix. TIME_WAIT times can be tweaked with the net.ipv4.tcp_tw_recycle and net.ipv4.tcp_tw_reuse sysctls, on Linux systems anyway.
  - cnlwsu 16y ago
    Exactly, you can only really exhaust the ports from the server address/port to your single host, which is effectively only DoSing your own access to the server. Ive seen over a million sockets in time_wait and it did not effect the servers. However if doing something like behind a single software load balancer then the number of ports available will be limited to the 65k connections since the source address/port and target address(the lb) are fixed.
  - MostAwesomeDude 16y ago
    I seem to recall that there are five numbers involved: Source address, source port, target address, target port, and a fifth number. I want to say it's the file descriptor, but that just sounds wrong when I type it out.
  - sophacles 16y ago
    Running out of ports is a poor way to put it, but it can be read as "Run the target machine out of fd (file descriptors)." In unixy systems there is a system parameter limiting the maximum number of file descriptors that can be open at one time (actually 2, per-proces and system-wide limits). These limits need to be turned up for big servers. The why is a pretty boring story: the number exists as a check against runaway processes or big jobs (e.g. busy servers) causing an I/O overload, so the numbers are set to a "reasonable value" by default. Of course the reasonable value is usually much lower than the system can handle, and at tuned for older common hardware.
gojomo 16y ago
Paul Tyma's counterpoint from 2008, that thread-per-connection is scalable to tens (and maybe hundreds) of thousands of threads, using recent Java & Linux NPTL: http://paultyma.blogspot.com/2008/03/writing-java-multithreaded-servers.html http://paultyma.blogspot.com/2008/03/writing-java-multithrea...
- KirinDave 16y ago
  Indeed, I'm reading this wondering how 400 connections could even crash a JVM on an OSX machine. It's surprising. I suspect it's not a threading issue in the sense he's describing. 400 threads probably shouldn't give you an OOM unless your heap size is set perilously low (in which case just about everything will give you that error). It's also painful to then see him argue that epoll/select/NIO is the magical path to freedom. Erlang and Akka developers are trying to to cry and laugh at the same time when they see this.
  - caf 16y ago
    400 threads with a thread stack size of 8M means that 3.2GB of virtual address space needs to be committed, which suspiciously close to the usual limit on a 32 bit system. If you want to create that many threads, you need to start by severely tuning down the stack size.
    - riobard 16y ago
      And use 64-bit systems.
      - ajross 16y ago
        Or use 64 bit systems. That 8M isn't physical memory, it's just allocated address space. Only the tiny handful of pages at the end of the region actually get faulted in.
    - KirinDave 16y ago
      Good point. It does require tuning.
trotsky 16y ago
Lots of purpose built servers are vulnerable to slowloris type attacks. Which doesn't mean that they shouldn't be fixed, just that it's hardly remarkable to find one. While certainly everyone is entitled to do disclosure however they see fit, it comes off kinda jerky to drop a working POC on the community just to advertise your unfinished competition.
- MostAwesomeDude 16y ago
  You're entirely correct. I should explain that this PoC was written not to advertise Bravo, but to explain the nature of the attack to a handful of members of the community. I don't advertise Bravo -- the people that it was written for were using it back when it was called Beta, and the wider community really doesn't care about custom servers very much. Additionally, mineserver is a much better server than Bravo; if I'm plugging custom servers, I will plug them instead.
- jschrf 16y ago
  Agreed. Yes, it is easy to attack video game servers. Indie game servers even more so. If you want to DoS a server so that players can't die wool or smelt iron or hit pigs for porkchops, you should probably reevaluate what you're spending your time on.
vegai 16y ago
Except if your language/platform hides the actual thread implementation in an epoll loop. coughhaskellcough

44 points by AndyKelley 16y ago

rdtsc 16y ago

One thread per connection might not scale but in this case other things could be done to mitigate the potential problem before re-writing the whole server. Perhaps use a special encoding for the authentication that doesn't use unbounded strings for username but a fixed size array of chars? Then maybe start a timer and if authentication isn't performed during a given time period, disconnect that client.

blake8086 16y ago
Say you kill connections after 1 second (pretty aggressive) and you can hold 1,000 connections open (that's a fair number of threads) All I have to do is create 1,000 connections per second and I'll hog all your thread resources. That's not too hard to do.
- wtallis 16y ago
  Killing connections after a full second has elapsed probably isn't at all aggressive in the context of a game server. If the connection is legit but takes a full second to send authentication info, then there's no hope for the game actually being playable over that connection.
  - kpreid 16y ago
    Depends on the game. Minecraft is hardly twitchy, especially if you're building, not fighting; the client proceeds as if each action has already happened, so you don't have to wait for the roundtrip between actions.
- rdtsc 16y ago
  1 second might not be too agressive for initial log-in -- that is where the real problem he is trying to solve is. If the username + password are not unbounded strings, but say, fit in an array of a 100 chars, then you should expect to get that data a lot sooner than 1 second (maybe 200ms?). So then maybe you can handle 5K connections... Yeah you are right, this is still not as good as a good async framework but the point was that some smaller things can be tried before a complete re-write. It might just be good enough.
MostAwesomeDude 16y ago
Nobody has really publicly talked about it, but there is now code in the Notchian and CraftBukkit servers to limit connections from single IPs to once every 5 seconds, and to disconnect slow logins by killing the thread after 30 seconds, effectively limiting the attack to 6 threads used per attacker. To actually bring down a server this way would then take 100 attackers or so. This really is just a proof of concept, after all, and not a tool which can bring down entire servers single-handedly. (At least, not anymore.)

bartwe 16y ago

Exhausting ports or generating a lot of TIME_WAIT states would also cause similar problems. How to protect against those ?

bnoordhuis 16y ago
You'll be hard-pressed to exhaust all ports: modern operating systems track connections by source address + source port + target address + target port. I wouldn't be surprised if the TCP sequence number is also part of the mix. TIME_WAIT times can be tweaked with the net.ipv4.tcp_tw_recycle and net.ipv4.tcp_tw_reuse sysctls, on Linux systems anyway.
- cnlwsu 16y ago
  Exactly, you can only really exhaust the ports from the server address/port to your single host, which is effectively only DoSing your own access to the server. Ive seen over a million sockets in time_wait and it did not effect the servers. However if doing something like behind a single software load balancer then the number of ports available will be limited to the 65k connections since the source address/port and target address(the lb) are fixed.
- MostAwesomeDude 16y ago
  I seem to recall that there are five numbers involved: Source address, source port, target address, target port, and a fifth number. I want to say it's the file descriptor, but that just sounds wrong when I type it out.
- sophacles 16y ago
  Running out of ports is a poor way to put it, but it can be read as "Run the target machine out of fd (file descriptors)." In unixy systems there is a system parameter limiting the maximum number of file descriptors that can be open at one time (actually 2, per-proces and system-wide limits). These limits need to be turned up for big servers. The why is a pretty boring story: the number exists as a check against runaway processes or big jobs (e.g. busy servers) causing an I/O overload, so the numbers are set to a "reasonable value" by default. Of course the reasonable value is usually much lower than the system can handle, and at tuned for older common hardware.

gojomo 16y ago

Paul Tyma's counterpoint from 2008, that thread-per-connection is scalable to tens (and maybe hundreds) of thousands of threads, using recent Java & Linux NPTL: http://paultyma.blogspot.com/2008/03/writing-java-multithreaded-servers.html http://paultyma.blogspot.com/2008/03/writing-java-multithrea...

KirinDave 16y ago
Indeed, I'm reading this wondering how 400 connections could even crash a JVM on an OSX machine. It's surprising. I suspect it's not a threading issue in the sense he's describing. 400 threads probably shouldn't give you an OOM unless your heap size is set perilously low (in which case just about everything will give you that error). It's also painful to then see him argue that epoll/select/NIO is the magical path to freedom. Erlang and Akka developers are trying to to cry and laugh at the same time when they see this.
- caf 16y ago
  400 threads with a thread stack size of 8M means that 3.2GB of virtual address space needs to be committed, which suspiciously close to the usual limit on a 32 bit system. If you want to create that many threads, you need to start by severely tuning down the stack size.
  - riobard 16y ago
    And use 64-bit systems.
    - ajross 16y ago
      Or use 64 bit systems. That 8M isn't physical memory, it's just allocated address space. Only the tiny handful of pages at the end of the region actually get faulted in.
  - KirinDave 16y ago
    Good point. It does require tuning.

trotsky 16y ago

Lots of purpose built servers are vulnerable to slowloris type attacks. Which doesn't mean that they shouldn't be fixed, just that it's hardly remarkable to find one. While certainly everyone is entitled to do disclosure however they see fit, it comes off kinda jerky to drop a working POC on the community just to advertise your unfinished competition.

MostAwesomeDude 16y ago
You're entirely correct. I should explain that this PoC was written not to advertise Bravo, but to explain the nature of the attack to a handful of members of the community. I don't advertise Bravo -- the people that it was written for were using it back when it was called Beta, and the wider community really doesn't care about custom servers very much. Additionally, mineserver is a much better server than Bravo; if I'm plugging custom servers, I will plug them instead.
jschrf 16y ago
Agreed. Yes, it is easy to attack video game servers. Indie game servers even more so. If you want to DoS a server so that players can't die wool or smelt iron or hit pigs for porkchops, you should probably reevaluate what you're spending your time on.

vegai 16y ago

Except if your language/platform hides the actual thread implementation in an epoll loop. coughhaskellcough