10 ms·

DoS attacks against my online game

171 points by def- 5y ago

Shadonototra 5y ago
i thought valve offered protection against DDoS attacks, or it's not available for free titles?
- def- 5y ago
  I actually received a nice email from someone at Valve about this following this post and we are currently evaluating if we can use it: https://partner.steamgames.com/doc/features/multiplayer/steamdatagramrelay https://partner.steamgames.com/doc/features/multiplayer/stea...
  - sapphyrus 5y ago
    SDR works well and the latency is decent, but this would probably force you to drop the non-steam release or make some of the servers steam-only?
    - leath 5y ago
      Documentation seems to say that they let you use the network from any platform as long as you have a Steam release. The bigger concern is that this requires linking with a closed source library, which means the open source version of the client could no longer connect to our servers without linking to a closed source blob. That is sort of a non starter for us without some workaround like maybe hosting our own relays for the open source clients.
  - znxster 5y ago
    Ah! I was going to this actually. I discovered it when Bungie switched Destiny2 over to use it.
  - Matheus28 5y ago
    Their approach is quite simple and effective, it is what I suggest game developers do. For reference, it goes somewhat like this: The clients connect to a relay server that just forwards the packets back and forth between the client and the real server. The client never gets to know the real server IP, preventing attackers from DDoSing the servers. If the connection to the relay server drops (which can easily happen if the attacker DDoSes the relay server instead), it can easily resume the connection with any other relay server, and the real server never notices it dropped. This relies on the fact that there are too many relay servers to DDoS at once, and attackers never get to know the real server running the game code, so they can't make it unreachable.
Scaevolus 5y ago
DoSing small games like this sucks. Have you looked at ddos-guard's pricing? They seem to be a common budget option.
- def- 5y ago
  Thanks for the tip, I didn't know about them. From an initial look $240 for a 1 core vps is a bit too expensive for us.
  - gibs0ns 5y ago
    Also checkout secured.gg [0], I recently discovered them while researching DDoS solutions used by other popular game servers (for Minecraft, and GTA5). I haven't used them but their pricing seemed reasonable to me. [0] https://secured.gg/dedicated https://secured.gg/dedicated
    - heinrich5991 5y ago
      The cheapest server seems to be about as much as we pay for 24 locations today.
debian3 5y ago
You should look into OVH. They have those gaming server and they handle the DDOS protection. I was getting hit by DDOS before, but since I moved there, nothing (except an email from OVH to let me know that my server is being attacked and that they are filtering my trafic). On the server itself you just don't feel anything. Edit: I should add that the DDOS protection is included with the server rental and there is no limit on the size or duration of the attack.
- pronoiac 5y ago
  > Instead of cheap VPS servers we have tried getting dedicated servers at larger European hosters like OVH, Hetzner, ihor and NFOrce. The idea is that we have exclusive resources, so the chances of us impacting other customers is lower, and thus we won’t get nullrouted so easily. Largely this works, but the available network bandwidth (usually 1-10 Gbit/s) as well as CPU usage become the limit.
- dan_wood 5y ago
  I don't think OVH is viable in this case, they do mitigate the attack but in my personal experience they also mitigate legit traffic during the attack. Mind you, this is a process using a single port, with only around 100 active connections. You'll easily see half if not more lose connection during a DDoS attack.
  - debian3 5y ago
    I did notice that a few years back, but now when the filter activate I no longer see any drop in bandwidth usage or any customer complaints. How long ago did you experimented with their filter?
    - dan_wood 5y ago
      Hmm good point, I haven't had any complaints for awhile. Maybe I'll look into it again see what happens. Thanks for the heads up :)
  - chmod775 5y ago
plasma 5y ago
A few suggestions: 1. Cloudflare offer TCP based DDoS protection too, see their Magic Transit or Spectrum product 2. This sucks, but put your servers behind WireGuard or Tailscale VPN so that in order to connect you need to have authenticated
- leath 5y ago
  As the blogpost mentions TCP is not exactly desirable for our project. Moreover, if you contact Cloudflare about those products you'll get a monthly quote that is far beyond what an open source project run by donations can sustain :)
  - deleted 5y ago
    [deleted]
  - eastdakota 5y ago
    That shouldn’t be the case. Email me if it is. I’ll make sure we help: matthewatcloudflaredotcom.
    - def- 5y ago
      Thanks, reached out to you.
ChrisMarshallNY 5y ago
Sounds like modern-day “protection rackets.” I know that, if you own a gambling site, you can look forward to meeting exciting slavs. I didn’t realize they were taking it to other types of games, but I guess that makes sense. Wiseguys coerce Grandma’s Bake Shop, just as they do Moneybags National Bank.
dividuum 5y ago
Not sure if this is helpful, but I remember reading about the tribes network protocol years ago. IIRC it also UDP based and essentially had a mechanism to request a proof of work depending on server load before accepting any complex packets from a client. You could probably require some proof for their source IP utilizing the HTTPS request already used during server browsing. Edit: Found it. http://opentnl.sourceforge.net/doxydocs/history.html http://opentnl.sourceforge.net/doxydocs/history.html (the „puzzles“)
- leath 5y ago
  The proof of work idea is cute, but at this stage it's not necessary as profiling seems to show attacks don't get too deep into the netcode before getting dropped. It's hard to know without testing but I'm fairly certain any kind of PoW would cost more to check. This reminds me of a blogpost that is at least fun to read: http://ithare.com/udp-for-games-security-encryption-and-ddos-protection/ http://ithare.com/udp-for-games-security-encryption-and-ddos... Regarding the HTTPS request thing, we already did this for an event and I'm currently getting it set up on all servers. The blog post forgot to mention this one :)
  - Cpoll 5y ago
    > any kind of PoW would cost more to check Cost more to check than it would for the attacker to generate an _incorrect_ proof. I'm sure that's what you mean, but my brain first went to NP problems. And actually, now that I think of it, maybe, maybe not. A PoW can be pre-calculated (possibly using off-peak cloud resources for very cheap) and then stored in a lookup table. It can be reused until the attacker actually solves the problem. Then _maybe_ you could offload that verification to a cheaper, harder to DDoS service, like a cloud function that won't charge you for SYN flooding. But then the hard part is letting legitimate users bypass the check after doing their PoW, but not letting an attacker through.
    - da_big_ghey 5y ago
      [removed]
cft 5y ago
In the US, Cogent offered $900 per month promotion for 10G commit on a 100G burstable circuit. If you rent several rack units and put a used Juniper hardware firewall from eBay in front, that may be one of the cheapest solutions to absorb 100Gbps attacks
- john37386 5y ago
  Make sure you won't use the 100Gbps in the full month because it might DDoS your wallet ;)
markus_zhang 5y ago
If you received anyone asking whether you need services that defend against DDOS, and if you refused, those are probably the guys who initiated the DDOS attack.
- schemescape 5y ago
  From the article, it sounds like they know at least some of the attackers already.
tyingq 5y ago
"For the individual server infos the client currently has to communicate with each game server by UDP, thus revealing its own IP address without having connected to a server. Since one of the known attackers is running their own DDNet server, they can use this method to collect legitimate player IP addresses and spoof them in their attacks." Interesting. I wonder if running an overlay network would help there. More choices today for userspace overlay networks. Rogue server owners would still see an IP, but they could only attack it from their connected server, not the internet at large. And you could do some kind of ingress/egress filtering. Some sort of periodic coordinated switching from UDP port A to port B might help too, like a control message that tells the game client to switch ports. Or randomized initial port assignments combined with filters/firewalling or just in-band 'you're not supposed to send here, bye'.
123pie123 5y ago
I've no idea how complex this is to code or if it will work for you but I'll throw the idea out to see if anyone else could improve on it etc.. initial strawman draft idea: have a front door service that just verifies your gamers (eg log on server) This will need to be protected by a Ddos but the throughput shouldn't be large. once authenticated your clients IP address is then passed to some sort of software based firewall protecting each of the main game servers
- Deathmax 5y ago
  The firewall would need to be able to handle all the DDoS traffic as well, since your current idea would still pass the game server's IP back to a client. This is doable if you're hosting on a cloud provider and let their firewalls filter the traffic before hitting the game server. Embark Studios recently open sourced (in alpha) a UDP proxy[1] designed for games that lets you implement a load balancing layer. This allows you to remove servers in the load balancing layer in the event that it comes under attack, allowing the game server to stay up and only having to disconnect a portion of players connected to the attacked loadbalancer. Having a proxy layer is also how Steam protects game servers using the Steam Datagram Relay[2]. [1]: https://github.com/googleforgames/quilkin https://github.com/googleforgames/quilkin [2]: https://partner.steamgames.com/doc/features/multiplayer/steamdatagramrelay https://partner.steamgames.com/doc/features/multiplayer/stea...
noobgrammer 5y ago
EDIT: Cloudflare already made a game demo on Workers... multiplayer Doom: https://blog.cloudflare.com/doom-multiplayer-workers/ https://blog.cloudflare.com/doom-multiplayer-workers/ that was written using WASM + WebSockets, porting over an open-source Doom and shimming in a UDP-over-Websockets networking layer. Despite all that, it's still fast enough for a first person shooter. Original post: Have you looked into using a serverless pub/sub model, like Cloudflare's Workers KV? The example they give is a simple IRC-like distributed chatroom (https://github.com/cloudflare/workers-chat-demo https://github.com/cloudflare/workers-chat-demo), but theoretically it may work for games too. Player state can be stored in a decentralized key-value store that Cloudflare manages (Cloudflare Durable Objects). They absorb all the DDoS and handle replication between edge nodes. You don't see any of that. https://developers.cloudflare.com/workers/learning/using-durable-objects https://developers.cloudflare.com/workers/learning/using-dur... Then each game client uses a worker to access that KV on a subscription basis, and Cloudflare will route that worker to its nearest edge node and retrieve the state from there (which was previously replicated a moment ago, internal to Cloudflare's infrastructure). Changes to state are replicated across the edge network and pushed to client workers. https://workers.cloudflare.com/ https://workers.cloudflare.com/ I don't know if this would result in acceptable latency, but it could help with DDOS at least. The main benefit is that it's incredibly affordable, especially when you're only talking about thousands of players.
seiferteric 5y ago
Why is ddos still possible? It is possible for isp’s to stop this. There is a proposal for isp level blocking if spoofed source addresses. Also there should be something like an api where I can tell my isp that I don’t want to receive anymore packets from a given source and it should be propagated up the chain.
- zeta0134 5y ago
  The key is in the first D: "distributed." A DDoS is designed to look just like legitimate traffic, but coming from many sources all at once. The goal of a successful attack is to both overwhelm the target network by sheer volume, and to make it difficult to stop the attacker without also blocking legitimate traffic. They persist in large part because they exploit the interconnectivity that makes the internet useful in the first place, without which it would cease to be.
  - seiferteric 5y ago
    It depends, some attacks rely on spoofing source address.. This should not be possible and is easily detectable by ISPs as illegitimate traffic. As far as my other suggestion, it would be a user accessible API. Whomever controls an IP should be able to instruct their ISP what is/isn't legit traffic, so the ISP does not have to know anything.
  - remram 5y ago
    The firewall can't tell what is legitimate traffic and what looks like legitimate traffic, but the application often can. This is what DOTS us trying to cover: let the application server tell the firewall who to allow or block in real time. https://www.rfc-editor.org/rfc/rfc8782.html https://www.rfc-editor.org/rfc/rfc8782.html
- remram 5y ago
  That exists: DOTS: https://www.rfc-editor.org/rfc/rfc8783.html https://www.rfc-editor.org/rfc/rfc8783.html
- leath 5y ago
api 5y ago
Why would someone DoS a small free game?
- bsder 5y ago
  Testing. Proof that they can DDoS something bigger. For the Lulz. etc.
- deleted 5y ago
  [deleted]
- tetromino_ 5y ago
  Because they are a teenage sociopath. (Quoting from the article: "... since the attacker was a minor ...")
  - pitay 5y ago
    That was a previous attacker, not the one now. The article does not give any identifying information about the new attacker except they know who he is, but don't have the data to prove it. Look at the top of the paragraph quoted to see this.
- willvarfar 5y ago
  From the hint about knowing the likely identity of the attacker, its probably an old and very specific vendetta. Perhaps an early player who was spurned by the community or developer who fell out? As the attacker has run tweaked gaming servers and things, it’s puts it at the upper end of scriptkiddying.
- nkrisc 5y ago
  Some people just have anti-social mental illnesses and this is how they manifest.
- stormcode 5y ago
  Some people are just not great. It sucks. This kind of thing saps the energy from the devs and makes it a job instead of a hobby.
bashy 5y ago
DoS attacks are something I’ve had to put up with too while hosting game servers since 2008. I run fshost[1] and we see attacks almost weekly. Even though we host mainstream games, we still see legitimate traffic being filtered. Do you modify any kernel options? net.ipv4.conf.all.rp_filter=1 [1] https://fshost.me https://fshost.me
reggieband 5y ago
I worked on a mid-sized online game a few years ago and we experienced several DDoS attacks. I recall one employee tracking down the specific botnet that was rented for the attack against us and we calculated the attacker probably spent a couple of hundred dollars based on the rates. IIRC, we eventually used AWS Elastic Load Balancer to just soak up the attack, which was a pretty basic SYN flood. Then we waited the attacker out until he got sick of spending money. That temporary redirect definitely impacted performance and cost us some money but it pretty well mitigated the issue. We also spent a bit of time optimizing our servers to drop obvious nonsense requests as quickly as possible but in the end the ELB handled most of the issue for us.
- NetOpWibby 5y ago
  Fascinating that there are people out there with disposable income for the sole purpose of griefing someone’s project/product.
  - nightfly 5y ago
    I think it can be cheaper to initiate these types of attacks than to defend against them. Just pissing someone off with an extra $20 to spend can give you a bad day
    - NetOpWibby 5y ago
      Well damn. The more you know. ):
    - ezoe 5y ago
      Real life amplification attack, eh?
    - lotsofpulp 5y ago
      Offense in cheaper than defense even in non internet related situations, hence societies with high levels of trust between its members are able to advance more quickly due to not needing to waste so many resources on defense.
lwansbrough 5y ago
Cloudflare supports HTTP/3 apparently. Could you run the game on WebTransport over QUIC?
void_mint 5y ago
I worked in games for a few years. We constantly got DoS'd. Probably twice a week. Luckily our content was extremely cachable, so once it happened a few times we just black holed stuff through Fastly. The attacks weren't super sophisticated, mostly just hitting a public endpoint a ton of times or trying to crawl API endpoints.
ethotool 5y ago
Maybe try implementing a reverse proxy, it would mask your real IP address.
Thaxll 5y ago
Player should be authenticated first to be able to to talk to the gameserver ( I'm talking about the accept() syscall ). 1h timeout is way too long, you should not have more than a couple of minutes max. I worked on some popular online games and it was a combination of 1) + some IP tables rules ( to allow the traffic ). Too many login would block the IP etc ... With proper auth ( based on TCP ), IPtables, kernel tuning you can get a lot of good results without doing some complicated things like UDP proxy / relay / load balancing.
- paulgb 5y ago
  Have you found any good resources/write-ups on the best practices for this sort of thing?
  - Thaxll 5y ago
    Unfortunatly like a lot of things in video games it's a lot in-house r&d, public doc for those kind of problems is non existent. The idea from my original post is that your gameserver should allow traffic only if the player is authenticated.
stickfigure 5y ago
Have you tried reaching out to the kid's parents?
- rjmunro 5y ago
  Or their ISP?
- leath 5y ago
  The public prosecutor did. Not sure how much more intimidating we could be than a prosecutor showing up threatening legal action
python273 5y ago
I don't know much about DDoS protection, but wouldn't it be possible to add token to UDP packets? So the client first connects to a TCP server and gets 4 bytes token unique per IP. Every UDP packed should start with that token. Since an attacker with spoofed IP can't get a valid token, their packets can be ignored with a cheap check. Might be even possible to use iptables/XDP for it?
- leath 5y ago
  We already have a token in the protocol and this is indeed a viable way to help drop invalid traffic. However, none of us are really experienced enough with the networking stack of the linux kernel (nor are these things very well documented) to craft a conntrack module or an XDP program to parse the header and keep track of the assigned tokens. Dropping these any later than the nftables stage is just not enough as we already do that. We have a small team who work on this project during our free times and most of us just don’t currently have the time to dig into conntrack/nftables/xdp.
  - python273 5y ago
    I found this tutorial [0] and example [1] how to read/update BPF hashtable data from python bcc module. If UDP packet structure is not too complex, it might be not too hard to implement I guess. [0] https://duo.com/labs/tech-notes/writing-an-xdp-network-filter-with-ebpf#section8 https://duo.com/labs/tech-notes/writing-an-xdp-network-filte... [1] https://lwn.net/Articles/747640/ https://lwn.net/Articles/747640/
- Arwill 5y ago
  It is a standard practice to have a "login server" and a separate "game server". The login server generates the token, and notifies the game server to accept it. The login server handles a queue, and can throttle login attempts. The login server can handle load balancing for multiple game servers, redirecting logged in players to different game servers. Its possible to run multiple login servers too, and restrict the access to specific login servers geographically and to run them with different providers.
mariusor 5y ago
I wonder if using some sort of port-knocking scheme would raise the barrier for these attacks enough.
- def- 5y ago
  It would raise the barrier slightly, but in the end the attackers can try out all ports quickly and check which one causes lags.
bencollier49 5y ago
Is this game a really big thing in Germany? Seems like all the major teams are German.
- deleted 5y ago
  [deleted]
- leath 5y ago
  A mention in a German gaming magazine very early on in the games lifetime brought a fair bit of attention
  - def- 5y ago
    Historically Europe and South America were our main locations, but recently we have grown mostly in East Asia.
eb0la 5y ago
>> we are a free online game and thus have no measurable economic damage, the public prosecutor left it at a sternly worded warning. The prosecutor is a #@*&%! : your time costs money. Working outside normal office hours is costly. Maybe you need to setup a contract between the "organization" that runs the servers and yourself that states how much time (and money) does it cost to run the game.
- authorityofnil 5y ago
  Not to mention the damage to the service providers that have their other clients affected because of this. Besides, if someone robs you, would the prosecutor handwave it away saying it's fine because they only got $20
  - thaumasiotes 5y ago
    > if someone robs you, would the prosecutor handwave it away saying it's fine because they only got $20 Yes. In fact, when my mother's grandfather was in conservatorship, his conservator (my mother's cousin) embezzled over $600,000 from his estate, and the prosecutor refused to prosecute. We were able to get somewhere over half of the amount back through a civil suit. Don't ever expect that the prosecutor is there to help you.
  - fragmede 5y ago
    Monetary boundaries are quite common in law. The "grand" in the video game titled Grand Theft Auto refers to stealing of a car worth more than $5,000. Stealing of a car worth less than that is simply theft (of an) auto. There are a lot of factors that go into a decision not to prosecute, but there are many petty crimes that go unpunished and unprosecuted. (There are also some that are prosecuted, but that doesn't disprove that some aren't.)
longsword 5y ago
I'am a Developer of an larger custom modded GTA Server (Roleplay Server), with 500-700 online users in peak hours. The community is very toxic. In the beginning, we received several DDoS Attacks daily, so we decided to build our own protection. We are working with IPtables and IPSets, as you are. But we dont bother ispecting the contents of a packet: Our users are logging in to the Server over HTTPS to a dedicated Server, which is protected by Cloudflare. Once they verified their identify, the Server sends a Request to the Main Game-Server, which then adds the User's IP (from which the Login Request came) to an IPSet. We setup IPTables to only accept packets from IP Adresses in this IPSet. When the IPSet is updated, the Login requests finishes and the User is connecting to the Main Gameserver. Also we are unsing UDP only for Game related traffic, so we completly blocked SYN and TCP related packets in our Hardware Firewall on the Gameserver Port (its a Hetzner Feature). Since then, we had no more major outage, besides sometimes some clients are dropping, because an attacker clogs our 10GB line. Of corse, this can't hold against every attack, but so far this is enought to protect us.
- hutrdvnj 5y ago
  > because an attacker clogs our 10GB line Yeah that's the problem, attackers get 10+ gigabits UDP flood quiet easily these days and with that they simply hammer down your line, well before any of your software protection mechanism could react.

171 points by def- 5y ago

Shadonototra 5y ago

i thought valve offered protection against DDoS attacks, or it's not available for free titles?

def- 5y ago
I actually received a nice email from someone at Valve about this following this post and we are currently evaluating if we can use it: https://partner.steamgames.com/doc/features/multiplayer/steamdatagramrelay https://partner.steamgames.com/doc/features/multiplayer/stea...
- sapphyrus 5y ago
  SDR works well and the latency is decent, but this would probably force you to drop the non-steam release or make some of the servers steam-only?
  - leath 5y ago
    Documentation seems to say that they let you use the network from any platform as long as you have a Steam release. The bigger concern is that this requires linking with a closed source library, which means the open source version of the client could no longer connect to our servers without linking to a closed source blob. That is sort of a non starter for us without some workaround like maybe hosting our own relays for the open source clients.
- znxster 5y ago
  Ah! I was going to this actually. I discovered it when Bungie switched Destiny2 over to use it.
- Matheus28 5y ago
  Their approach is quite simple and effective, it is what I suggest game developers do. For reference, it goes somewhat like this: The clients connect to a relay server that just forwards the packets back and forth between the client and the real server. The client never gets to know the real server IP, preventing attackers from DDoSing the servers. If the connection to the relay server drops (which can easily happen if the attacker DDoSes the relay server instead), it can easily resume the connection with any other relay server, and the real server never notices it dropped. This relies on the fact that there are too many relay servers to DDoS at once, and attackers never get to know the real server running the game code, so they can't make it unreachable.

Scaevolus 5y ago

DoSing small games like this sucks. Have you looked at ddos-guard's pricing? They seem to be a common budget option.

def- 5y ago
Thanks for the tip, I didn't know about them. From an initial look $240 for a 1 core vps is a bit too expensive for us.
- gibs0ns 5y ago
  Also checkout secured.gg [0], I recently discovered them while researching DDoS solutions used by other popular game servers (for Minecraft, and GTA5). I haven't used them but their pricing seemed reasonable to me. [0] https://secured.gg/dedicated https://secured.gg/dedicated
  - heinrich5991 5y ago
    The cheapest server seems to be about as much as we pay for 24 locations today.

debian3 5y ago

You should look into OVH. They have those gaming server and they handle the DDOS protection. I was getting hit by DDOS before, but since I moved there, nothing (except an email from OVH to let me know that my server is being attacked and that they are filtering my trafic). On the server itself you just don't feel anything. Edit: I should add that the DDOS protection is included with the server rental and there is no limit on the size or duration of the attack.

pronoiac 5y ago
> Instead of cheap VPS servers we have tried getting dedicated servers at larger European hosters like OVH, Hetzner, ihor and NFOrce. The idea is that we have exclusive resources, so the chances of us impacting other customers is lower, and thus we won’t get nullrouted so easily. Largely this works, but the available network bandwidth (usually 1-10 Gbit/s) as well as CPU usage become the limit.
dan_wood 5y ago
I don't think OVH is viable in this case, they do mitigate the attack but in my personal experience they also mitigate legit traffic during the attack. Mind you, this is a process using a single port, with only around 100 active connections. You'll easily see half if not more lose connection during a DDoS attack.
- debian3 5y ago
  I did notice that a few years back, but now when the filter activate I no longer see any drop in bandwidth usage or any customer complaints. How long ago did you experimented with their filter?
  - dan_wood 5y ago
    Hmm good point, I haven't had any complaints for awhile. Maybe I'll look into it again see what happens. Thanks for the heads up :)
- chmod775 5y ago

plasma 5y ago

A few suggestions: 1. Cloudflare offer TCP based DDoS protection too, see their Magic Transit or Spectrum product 2. This sucks, but put your servers behind WireGuard or Tailscale VPN so that in order to connect you need to have authenticated

leath 5y ago
As the blogpost mentions TCP is not exactly desirable for our project. Moreover, if you contact Cloudflare about those products you'll get a monthly quote that is far beyond what an open source project run by donations can sustain :)
- deleted 5y ago
  [deleted]
- eastdakota 5y ago
  That shouldn’t be the case. Email me if it is. I’ll make sure we help: matthewatcloudflaredotcom.
  - def- 5y ago
    Thanks, reached out to you.

ChrisMarshallNY 5y ago

Sounds like modern-day “protection rackets.” I know that, if you own a gambling site, you can look forward to meeting exciting slavs. I didn’t realize they were taking it to other types of games, but I guess that makes sense. Wiseguys coerce Grandma’s Bake Shop, just as they do Moneybags National Bank.

dividuum 5y ago

Not sure if this is helpful, but I remember reading about the tribes network protocol years ago. IIRC it also UDP based and essentially had a mechanism to request a proof of work depending on server load before accepting any complex packets from a client. You could probably require some proof for their source IP utilizing the HTTPS request already used during server browsing. Edit: Found it. http://opentnl.sourceforge.net/doxydocs/history.html http://opentnl.sourceforge.net/doxydocs/history.html (the „puzzles“)

leath 5y ago
The proof of work idea is cute, but at this stage it's not necessary as profiling seems to show attacks don't get too deep into the netcode before getting dropped. It's hard to know without testing but I'm fairly certain any kind of PoW would cost more to check. This reminds me of a blogpost that is at least fun to read: http://ithare.com/udp-for-games-security-encryption-and-ddos-protection/ http://ithare.com/udp-for-games-security-encryption-and-ddos... Regarding the HTTPS request thing, we already did this for an event and I'm currently getting it set up on all servers. The blog post forgot to mention this one :)
- Cpoll 5y ago
  > any kind of PoW would cost more to check Cost more to check than it would for the attacker to generate an _incorrect_ proof. I'm sure that's what you mean, but my brain first went to NP problems. And actually, now that I think of it, maybe, maybe not. A PoW can be pre-calculated (possibly using off-peak cloud resources for very cheap) and then stored in a lookup table. It can be reused until the attacker actually solves the problem. Then _maybe_ you could offload that verification to a cheaper, harder to DDoS service, like a cloud function that won't charge you for SYN flooding. But then the hard part is letting legitimate users bypass the check after doing their PoW, but not letting an attacker through.
  - da_big_ghey 5y ago
    [removed]

cft 5y ago

In the US, Cogent offered $900 per month promotion for 10G commit on a 100G burstable circuit. If you rent several rack units and put a used Juniper hardware firewall from eBay in front, that may be one of the cheapest solutions to absorb 100Gbps attacks

john37386 5y ago
Make sure you won't use the 100Gbps in the full month because it might DDoS your wallet ;)

markus_zhang 5y ago

If you received anyone asking whether you need services that defend against DDOS, and if you refused, those are probably the guys who initiated the DDOS attack.

schemescape 5y ago
From the article, it sounds like they know at least some of the attackers already.

tyingq 5y ago

"For the individual server infos the client currently has to communicate with each game server by UDP, thus revealing its own IP address without having connected to a server. Since one of the known attackers is running their own DDNet server, they can use this method to collect legitimate player IP addresses and spoof them in their attacks." Interesting. I wonder if running an overlay network would help there. More choices today for userspace overlay networks. Rogue server owners would still see an IP, but they could only attack it from their connected server, not the internet at large. And you could do some kind of ingress/egress filtering. Some sort of periodic coordinated switching from UDP port A to port B might help too, like a control message that tells the game client to switch ports. Or randomized initial port assignments combined with filters/firewalling or just in-band 'you're not supposed to send here, bye'.

123pie123 5y ago

I've no idea how complex this is to code or if it will work for you but I'll throw the idea out to see if anyone else could improve on it etc.. initial strawman draft idea: have a front door service that just verifies your gamers (eg log on server) This will need to be protected by a Ddos but the throughput shouldn't be large. once authenticated your clients IP address is then passed to some sort of software based firewall protecting each of the main game servers

Deathmax 5y ago
The firewall would need to be able to handle all the DDoS traffic as well, since your current idea would still pass the game server's IP back to a client. This is doable if you're hosting on a cloud provider and let their firewalls filter the traffic before hitting the game server. Embark Studios recently open sourced (in alpha) a UDP proxy[1] designed for games that lets you implement a load balancing layer. This allows you to remove servers in the load balancing layer in the event that it comes under attack, allowing the game server to stay up and only having to disconnect a portion of players connected to the attacked loadbalancer. Having a proxy layer is also how Steam protects game servers using the Steam Datagram Relay[2]. [1]: https://github.com/googleforgames/quilkin https://github.com/googleforgames/quilkin [2]: https://partner.steamgames.com/doc/features/multiplayer/steamdatagramrelay https://partner.steamgames.com/doc/features/multiplayer/stea...

noobgrammer 5y ago

EDIT: Cloudflare already made a game demo on Workers... multiplayer Doom: https://blog.cloudflare.com/doom-multiplayer-workers/ https://blog.cloudflare.com/doom-multiplayer-workers/ that was written using WASM + WebSockets, porting over an open-source Doom and shimming in a UDP-over-Websockets networking layer. Despite all that, it's still fast enough for a first person shooter. Original post: Have you looked into using a serverless pub/sub model, like Cloudflare's Workers KV? The example they give is a simple IRC-like distributed chatroom (https://github.com/cloudflare/workers-chat-demo https://github.com/cloudflare/workers-chat-demo), but theoretically it may work for games too. Player state can be stored in a decentralized key-value store that Cloudflare manages (Cloudflare Durable Objects). They absorb all the DDoS and handle replication between edge nodes. You don't see any of that. https://developers.cloudflare.com/workers/learning/using-durable-objects https://developers.cloudflare.com/workers/learning/using-dur... Then each game client uses a worker to access that KV on a subscription basis, and Cloudflare will route that worker to its nearest edge node and retrieve the state from there (which was previously replicated a moment ago, internal to Cloudflare's infrastructure). Changes to state are replicated across the edge network and pushed to client workers. https://workers.cloudflare.com/ https://workers.cloudflare.com/ I don't know if this would result in acceptable latency, but it could help with DDOS at least. The main benefit is that it's incredibly affordable, especially when you're only talking about thousands of players.

seiferteric 5y ago

Why is ddos still possible? It is possible for isp’s to stop this. There is a proposal for isp level blocking if spoofed source addresses. Also there should be something like an api where I can tell my isp that I don’t want to receive anymore packets from a given source and it should be propagated up the chain.

zeta0134 5y ago
The key is in the first D: "distributed." A DDoS is designed to look just like legitimate traffic, but coming from many sources all at once. The goal of a successful attack is to both overwhelm the target network by sheer volume, and to make it difficult to stop the attacker without also blocking legitimate traffic. They persist in large part because they exploit the interconnectivity that makes the internet useful in the first place, without which it would cease to be.
- seiferteric 5y ago
  It depends, some attacks rely on spoofing source address.. This should not be possible and is easily detectable by ISPs as illegitimate traffic. As far as my other suggestion, it would be a user accessible API. Whomever controls an IP should be able to instruct their ISP what is/isn't legit traffic, so the ISP does not have to know anything.
- remram 5y ago
  The firewall can't tell what is legitimate traffic and what looks like legitimate traffic, but the application often can. This is what DOTS us trying to cover: let the application server tell the firewall who to allow or block in real time. https://www.rfc-editor.org/rfc/rfc8782.html https://www.rfc-editor.org/rfc/rfc8782.html
remram 5y ago
That exists: DOTS: https://www.rfc-editor.org/rfc/rfc8783.html https://www.rfc-editor.org/rfc/rfc8783.html
leath 5y ago

api 5y ago

Why would someone DoS a small free game?

bsder 5y ago
Testing. Proof that they can DDoS something bigger. For the Lulz. etc.
deleted 5y ago
[deleted]
tetromino_ 5y ago
Because they are a teenage sociopath. (Quoting from the article: "... since the attacker was a minor ...")
- pitay 5y ago
  That was a previous attacker, not the one now. The article does not give any identifying information about the new attacker except they know who he is, but don't have the data to prove it. Look at the top of the paragraph quoted to see this.
willvarfar 5y ago
From the hint about knowing the likely identity of the attacker, its probably an old and very specific vendetta. Perhaps an early player who was spurned by the community or developer who fell out? As the attacker has run tweaked gaming servers and things, it’s puts it at the upper end of scriptkiddying.
nkrisc 5y ago
Some people just have anti-social mental illnesses and this is how they manifest.
stormcode 5y ago
Some people are just not great. It sucks. This kind of thing saps the energy from the devs and makes it a job instead of a hobby.

bashy 5y ago

DoS attacks are something I’ve had to put up with too while hosting game servers since 2008. I run fshost[1] and we see attacks almost weekly. Even though we host mainstream games, we still see legitimate traffic being filtered. Do you modify any kernel options? net.ipv4.conf.all.rp_filter=1 [1] https://fshost.me https://fshost.me

reggieband 5y ago

I worked on a mid-sized online game a few years ago and we experienced several DDoS attacks. I recall one employee tracking down the specific botnet that was rented for the attack against us and we calculated the attacker probably spent a couple of hundred dollars based on the rates. IIRC, we eventually used AWS Elastic Load Balancer to just soak up the attack, which was a pretty basic SYN flood. Then we waited the attacker out until he got sick of spending money. That temporary redirect definitely impacted performance and cost us some money but it pretty well mitigated the issue. We also spent a bit of time optimizing our servers to drop obvious nonsense requests as quickly as possible but in the end the ELB handled most of the issue for us.

NetOpWibby 5y ago
Fascinating that there are people out there with disposable income for the sole purpose of griefing someone’s project/product.
- nightfly 5y ago
  I think it can be cheaper to initiate these types of attacks than to defend against them. Just pissing someone off with an extra $20 to spend can give you a bad day
  - NetOpWibby 5y ago
    Well damn. The more you know. ):
  - ezoe 5y ago
    Real life amplification attack, eh?
  - lotsofpulp 5y ago
    Offense in cheaper than defense even in non internet related situations, hence societies with high levels of trust between its members are able to advance more quickly due to not needing to waste so many resources on defense.

lwansbrough 5y ago

Cloudflare supports HTTP/3 apparently. Could you run the game on WebTransport over QUIC?

void_mint 5y ago

I worked in games for a few years. We constantly got DoS'd. Probably twice a week. Luckily our content was extremely cachable, so once it happened a few times we just black holed stuff through Fastly. The attacks weren't super sophisticated, mostly just hitting a public endpoint a ton of times or trying to crawl API endpoints.

ethotool 5y ago

Maybe try implementing a reverse proxy, it would mask your real IP address.

Thaxll 5y ago

Player should be authenticated first to be able to to talk to the gameserver ( I'm talking about the accept() syscall ). 1h timeout is way too long, you should not have more than a couple of minutes max. I worked on some popular online games and it was a combination of 1) + some IP tables rules ( to allow the traffic ). Too many login would block the IP etc ... With proper auth ( based on TCP ), IPtables, kernel tuning you can get a lot of good results without doing some complicated things like UDP proxy / relay / load balancing.

paulgb 5y ago
Have you found any good resources/write-ups on the best practices for this sort of thing?
- Thaxll 5y ago
  Unfortunatly like a lot of things in video games it's a lot in-house r&d, public doc for those kind of problems is non existent. The idea from my original post is that your gameserver should allow traffic only if the player is authenticated.

stickfigure 5y ago

Have you tried reaching out to the kid's parents?

rjmunro 5y ago
Or their ISP?
leath 5y ago
The public prosecutor did. Not sure how much more intimidating we could be than a prosecutor showing up threatening legal action

python273 5y ago

I don't know much about DDoS protection, but wouldn't it be possible to add token to UDP packets? So the client first connects to a TCP server and gets 4 bytes token unique per IP. Every UDP packed should start with that token. Since an attacker with spoofed IP can't get a valid token, their packets can be ignored with a cheap check. Might be even possible to use iptables/XDP for it?

leath 5y ago
We already have a token in the protocol and this is indeed a viable way to help drop invalid traffic. However, none of us are really experienced enough with the networking stack of the linux kernel (nor are these things very well documented) to craft a conntrack module or an XDP program to parse the header and keep track of the assigned tokens. Dropping these any later than the nftables stage is just not enough as we already do that. We have a small team who work on this project during our free times and most of us just don’t currently have the time to dig into conntrack/nftables/xdp.
- python273 5y ago
  I found this tutorial [0] and example [1] how to read/update BPF hashtable data from python bcc module. If UDP packet structure is not too complex, it might be not too hard to implement I guess. [0] https://duo.com/labs/tech-notes/writing-an-xdp-network-filter-with-ebpf#section8 https://duo.com/labs/tech-notes/writing-an-xdp-network-filte... [1] https://lwn.net/Articles/747640/ https://lwn.net/Articles/747640/
Arwill 5y ago
It is a standard practice to have a "login server" and a separate "game server". The login server generates the token, and notifies the game server to accept it. The login server handles a queue, and can throttle login attempts. The login server can handle load balancing for multiple game servers, redirecting logged in players to different game servers. Its possible to run multiple login servers too, and restrict the access to specific login servers geographically and to run them with different providers.

mariusor 5y ago

I wonder if using some sort of port-knocking scheme would raise the barrier for these attacks enough.

def- 5y ago
It would raise the barrier slightly, but in the end the attackers can try out all ports quickly and check which one causes lags.

bencollier49 5y ago

Is this game a really big thing in Germany? Seems like all the major teams are German.

deleted 5y ago
[deleted]
leath 5y ago
A mention in a German gaming magazine very early on in the games lifetime brought a fair bit of attention
- def- 5y ago
  Historically Europe and South America were our main locations, but recently we have grown mostly in East Asia.

eb0la 5y ago

>> we are a free online game and thus have no measurable economic damage, the public prosecutor left it at a sternly worded warning. The prosecutor is a #@*&%! : your time costs money. Working outside normal office hours is costly. Maybe you need to setup a contract between the "organization" that runs the servers and yourself that states how much time (and money) does it cost to run the game.

authorityofnil 5y ago
Not to mention the damage to the service providers that have their other clients affected because of this. Besides, if someone robs you, would the prosecutor handwave it away saying it's fine because they only got $20
- thaumasiotes 5y ago
  > if someone robs you, would the prosecutor handwave it away saying it's fine because they only got $20 Yes. In fact, when my mother's grandfather was in conservatorship, his conservator (my mother's cousin) embezzled over $600,000 from his estate, and the prosecutor refused to prosecute. We were able to get somewhere over half of the amount back through a civil suit. Don't ever expect that the prosecutor is there to help you.
- fragmede 5y ago
  Monetary boundaries are quite common in law. The "grand" in the video game titled Grand Theft Auto refers to stealing of a car worth more than $5,000. Stealing of a car worth less than that is simply theft (of an) auto. There are a lot of factors that go into a decision not to prosecute, but there are many petty crimes that go unpunished and unprosecuted. (There are also some that are prosecuted, but that doesn't disprove that some aren't.)

longsword 5y ago

I'am a Developer of an larger custom modded GTA Server (Roleplay Server), with 500-700 online users in peak hours. The community is very toxic. In the beginning, we received several DDoS Attacks daily, so we decided to build our own protection. We are working with IPtables and IPSets, as you are. But we dont bother ispecting the contents of a packet: Our users are logging in to the Server over HTTPS to a dedicated Server, which is protected by Cloudflare. Once they verified their identify, the Server sends a Request to the Main Game-Server, which then adds the User's IP (from which the Login Request came) to an IPSet. We setup IPTables to only accept packets from IP Adresses in this IPSet. When the IPSet is updated, the Login requests finishes and the User is connecting to the Main Gameserver. Also we are unsing UDP only for Game related traffic, so we completly blocked SYN and TCP related packets in our Hardware Firewall on the Gameserver Port (its a Hetzner Feature). Since then, we had no more major outage, besides sometimes some clients are dropping, because an attacker clogs our 10GB line. Of corse, this can't hold against every attack, but so far this is enought to protect us.

hutrdvnj 5y ago
> because an attacker clogs our 10GB line Yeah that's the problem, attackers get 10+ gigabits UDP flood quiet easily these days and with that they simply hammer down your line, well before any of your software protection mechanism could react.