6 ms·

Overview of Running an Online Game for 3 Years

197 points by def- 10y ago

themartorana 10y ago
€10/m/location is damn impressive. I've run servers for a turn-based asynchronous and real-time casual game for the past three years. With ~700k unique monthly players (about 200k/day) we do about 1500 request/s at peak and pay thousands a month for our AWS stack. I'm not mad at it, I think we get great utility for what we pay, but this is lean and mean for realz. Kudos!
- def- 10y ago
  Thanks! Interesting to hear about your experience as well. By requests per seconds you mean packets? I was curious and with 330 players playing on DDNet right now[1] we get ~7400 packets/s incoming, ~8300 packets/s outgoing. [1] https://ddnet.tw/status/ https://ddnet.tw/status/
  - Cyph0n 10y ago
    My guess is that he's talking about HTTP requests.
    - themartorana 10y ago
      Yes, I am talking about HTTP requests. We're just exploring moving over to web sockets, the original implementations were done before that was practical on mobile networks.
- hueving 10y ago
  AWS is not anywhere near cheap when it comes to raw performance per dollar. You are paying a very large premium for that flexibility of an arbitrary number of instances at any time.
- meritt 10y ago
  Care to share some details about your setup? 1500 reqs/sec should not cost very much at all, definitely not "thousands a month".
  - themartorana 10y ago
    Well, a larger chunk of that is the database. A db.r3.4xlarge runs about $1600 by itself. We have several hundred gig of data, and indexes alone keep us in that instance size for RAM allotment. Aside from that, we run 2-3 c3.xlarge API servers, two c3.medium webserver (hosts Facebook versions of our games) and two more c3.medium servers are joining in for a new launch next month. We have one large (size escapes me) ElastiCache instance. 3 ELBs sit in front of those servers, all in a VPC. Finally, we have 11 or so S3 buckets that back Cloudfront distributions for static content for websites and whatnot. We also pay for several TB of traffic per month, and enterprise support. Finally, we're paying for not having to fix any hardware ever, 5 minute incremental DB backups, a VPC I can define in a JSON doc, automatic failovers, and so much more, managed by yours truly, a software engineer, not a network/server/devops/etc. engineer. Yeah, there's a premium, but so far it's paid off.
    - grogenaut 10y ago
      What drove the choice to the larger instance types instead of more of the small ones? Not questioning your judgment I'm just curious.
      - themartorana 10y ago
        We moved from Python to Go. With Python behind uwsgi->nginx, we'd try to tune the number of uwsgi processes to cores, and it was easy to scale up based on process=number of concurrent requests handled. With Go, every request is handled concurrently, being much more efficient with cores and RAM. I found that my requests/core were being artificially limited by RAM in an autoscaling group (we had scaling triggered at about avg 80% CPU) so I let some more RAM play. Now, instead of 15-20 c3.medium servers at peak, we run three c3.xlarge instances at peak. I can be sure that the larger server will handle more concurrent requests than process-restricted uwsgi, and thus be able to better allocate the resources given to us by larger instances.
- ryanlol 10y ago
  Unless you need AWS for the scaling it offers, it tends to be the worst possible host you could choose. And even if you need the scaling, a mixed stack with dedis and EC2 on demand shouldn't be hard to run.
  - eropple 10y ago
    > Unless you need AWS for the scaling it offers, it tends to be the worst possible host you could choose. Unless, in turn, you use reserved instances, which bring the price down to equal or cheaper than most other options. "dedis and EC2 on demand" is an order of magnitude more complex, probably actually more expensive, and increases your failure cases significantly.
    - ryanlol 10y ago
      I think you're a little out of touch with hosting prices these days. https://www.ovh.com/us/dedicated-servers/details-servers.xml?range=HOST&id=2016-HOST-32L https://www.ovh.com/us/dedicated-servers/details-servers.xml... For 12 months will cost you about $800 A somewhat comparable EC2 instance (that still has way lower specs) m3.2xlarge will have an up-front cost of $1772, and an hourly cost of $0.146. So you'd be looking at a total yearly bill of $3051, which would pay for 3 of those dedis. I honestly don't know how anyone could describe that as competitive. > "dedis and EC2 on demand" is an order of magnitude more complex, probably actually more expensive. If you're one of the few people that actually need the rapid scaling that EC2 can provide, sure. But that's an extremely specific and rare use case.
      - blahi 10y ago
        Cloud platforms like AWS offer other things besides just scaling. Like Hadoop processing, hosted databases, email infrastructure, DNS services and the list goes on and on.
      - eropple 10y ago
        I'm alarmingly familiar with OVH. And their hardware failures. And their slow support exactly when you need them most. And their poor inventory control, leaving popular system configurations unavailable when I (or, rather, my clients) need them. I'm also familiar with exactly how misleading a specs comparison ends up when you can architect an application to leverage burstable instances and rapid auto-scaling, letting you require little expenditure during slow periods while still responding gracefully to load. This isn't some deep-magic thing for sites with tens of thousands of instances, either--I've had clients with ten total instances benefit both financially and operationally from building out fairly straightforward elastic capacity (not least because the same code paths necessary for elasticity also makes things like setting up development environments single-click affairs, which enables deeper and more complete testing of the application to ensure correctness). Even were a $2200 premium a credible number, that's eleven hours of my billing rate. Reimplementing what AWS provides even a fairly straightforward system--easy, API-driven system deployment and configuration (and while OVH's public cloud APIs can enable some of this, your "dedis" cannot), straightforward and API-controlled support services of more or less any stripe, integrated system monitoring and alerting--would be hundreds of hours of work, be a worse solution, and require ongoing maintenance. (A decent hosting provider will offer some of that, of course. But not all. And you'll pay for it.)
- teknologist 10y ago
  Would you get anywhere with using those Scaleway ARM BareMetal servers? They're insanely cheap if you can run on ARM.
lccarrasco 10y ago
This was really great to read, in-depth and interesting, thanks a lot for taking the time to write it. :)
mentos 10y ago
Hey great work! Curious to hear what the client stack was? Did you use LibGDX by chance?
- def- 10y ago
  Thanks! You can see the client here: https://github.com/ddnet/ddnet https://github.com/ddnet/ddnet SDL2, OpenGL, FreeType, pnglite, zlib, curl, md5, wavpack, opus, json-parser So pretty much low level, keeps the performance and flexibility high.
eggy 10y ago
Awesome and inspiring to me, great work! I am now looking at LFE (Lisp Flavored Erlang) and ELM to create a very small online game. It makes me want to maintain C/C++ chops. It's sad Apple is so walled in that you need a VM to build for OS X, and iOS doesn't even make the list. I have an iPad, but I use an Android phone for that reason, and I only program mobile for Android. Apple is getting better at supporting iOS devs of late though...
ashitlerferad 10y ago
What was the reason for the fork with Teeworlds?
- def- 10y ago
  Teeworlds is a shooter. People wanted to race in Teeworlds, so a modification was required. From there it developed into multiplayer racing and the last few years of DDNet adding features.
ObeyTheGuts 10y ago
Maidsafe will eliminate all this server problems!
adynatos 10y ago
The author writes: "Reduce the number of syscalls by caching the value of gettimeofday() until a new tick happens or network packet comes in" But I'm pretty sure glibc on recent Linux handles gettimeofday in user-space, without context switch (kernel maps the data to userspace). I guess caching the value locally and updating it 1/sec or something would still help if there are thousands of calls/sec, but not as much as if it was really a syscall.
- def- 10y ago
  You're right. Unfortunately with virtual machines that doesn't always work. You can enable it manually but that's more inefficient because the Kernel can't use paravirtualization for clock then. I stumbled upon this a month ago: https://ddnet.tw/irclogs/2016-05-17.log https://ddnet.tw/irclogs/2016-05-17.log 21:37 <@deen> A third of the sys calls of a DDNet server are recfrom and sendto each 21:37 <@deen> they always occur in large chunks, so ideal for sendmmsg/recvmmsg 21:38 <@deen> the last third are mostly strange time and gettimeofday calls, I thought I got rid of most of them 21:40 <@deen> server with 30 players causes 3000 syscalls a second 22:00 <@deen> PACKET_MMAP is very cool, reading packets with 0 syscalls, too bad it's not for regular applications (requires root, doesn't work with normal udp socket): https://www.kernel.org/doc/Documentation/networking/packet_mmap.txt 22:04 <@deen> and then the glibc version matters a lot for syscalls. new versions of glibc don't syscall at all for gettimeofday 22:05 <@deen> (or something else is causing that, not sure yet) 22:13 <@deen> Reading the glibc implementation, that's done by vDSO, interesting: http://man7.org/linux/man-pages/man7/vdso.7.html 22:32 <@deen> totally confused why some of our servers use vdso gettimeofday, others not even though they have more recent kernel and glibc 22:41 <@deen> ok, probably depends on the underlying clock that the vps uses. pvclock is used with kvm and doesn't support vdso. but looks like there's some progress being made: https://lkml.org/lkml/2015/12/9/914
urza 10y ago
I would like to see this kind of post for slither.io
qwertyuiop924 10y ago
I just re-downloaded (vanilla) teeworlds the other day. After playing a lot of QW and Xonotic, it's nice to play something like HLDM or TeeWorlds that's a bit more wacky and less competitive. There is no bunnyhopping in teeworlds. Just cute fluffballs, hookshots, and heavy weaponry. Although, for whatever reason, all of the players are in the EU, or SA, and I'm USEast, so the ping's really high. One of many reasons I want to get a GPU that can handle UT4. Yes, mine is really that bad.
qwertyuiop924 10y ago
I just re-downloaded (vanilla) teeworlds the other day. After playing a lot of QW and Xonotic, it's nice to play something like HLDM or TeeWorlds that's a bit more wacky and less competitive. There is no bunnyhopping in teeworlds. Just cute balls, hookshots, and heavy weaponry. Although, for whatever reason, all of the players are in the EU, or SA, and I'm USEast, so the ping's really high. One of many reasons I want to get a GPU that can handle UT4. Yes, mine is really that bad.

197 points by def- 10y ago

€10/m/location is damn impressive. I've run servers for a turn-based asynchronous and real-time casual game for the past three years. With ~700k unique monthly players (about 200k/day) we do about 1500 request/s at peak and pay thousands a month for our AWS stack. I'm not mad at it, I think we get great utility for what we pay, but this is lean and mean for realz. Kudos!

def- 10y ago
Thanks! Interesting to hear about your experience as well. By requests per seconds you mean packets? I was curious and with 330 players playing on DDNet right now[1] we get ~7400 packets/s incoming, ~8300 packets/s outgoing. [1] https://ddnet.tw/status/ https://ddnet.tw/status/
- Cyph0n 10y ago
  My guess is that he's talking about HTTP requests.
  - themartorana 10y ago
    Yes, I am talking about HTTP requests. We're just exploring moving over to web sockets, the original implementations were done before that was practical on mobile networks.
hueving 10y ago
AWS is not anywhere near cheap when it comes to raw performance per dollar. You are paying a very large premium for that flexibility of an arbitrary number of instances at any time.
meritt 10y ago
Care to share some details about your setup? 1500 reqs/sec should not cost very much at all, definitely not "thousands a month".
- themartorana 10y ago
  Well, a larger chunk of that is the database. A db.r3.4xlarge runs about $1600 by itself. We have several hundred gig of data, and indexes alone keep us in that instance size for RAM allotment. Aside from that, we run 2-3 c3.xlarge API servers, two c3.medium webserver (hosts Facebook versions of our games) and two more c3.medium servers are joining in for a new launch next month. We have one large (size escapes me) ElastiCache instance. 3 ELBs sit in front of those servers, all in a VPC. Finally, we have 11 or so S3 buckets that back Cloudfront distributions for static content for websites and whatnot. We also pay for several TB of traffic per month, and enterprise support. Finally, we're paying for not having to fix any hardware ever, 5 minute incremental DB backups, a VPC I can define in a JSON doc, automatic failovers, and so much more, managed by yours truly, a software engineer, not a network/server/devops/etc. engineer. Yeah, there's a premium, but so far it's paid off.
  - grogenaut 10y ago
    What drove the choice to the larger instance types instead of more of the small ones? Not questioning your judgment I'm just curious.
    - themartorana 10y ago
      We moved from Python to Go. With Python behind uwsgi->nginx, we'd try to tune the number of uwsgi processes to cores, and it was easy to scale up based on process=number of concurrent requests handled. With Go, every request is handled concurrently, being much more efficient with cores and RAM. I found that my requests/core were being artificially limited by RAM in an autoscaling group (we had scaling triggered at about avg 80% CPU) so I let some more RAM play. Now, instead of 15-20 c3.medium servers at peak, we run three c3.xlarge instances at peak. I can be sure that the larger server will handle more concurrent requests than process-restricted uwsgi, and thus be able to better allocate the resources given to us by larger instances.
ryanlol 10y ago
Unless you need AWS for the scaling it offers, it tends to be the worst possible host you could choose. And even if you need the scaling, a mixed stack with dedis and EC2 on demand shouldn't be hard to run.
- eropple 10y ago
  > Unless you need AWS for the scaling it offers, it tends to be the worst possible host you could choose. Unless, in turn, you use reserved instances, which bring the price down to equal or cheaper than most other options. "dedis and EC2 on demand" is an order of magnitude more complex, probably actually more expensive, and increases your failure cases significantly.
  - ryanlol 10y ago
    I think you're a little out of touch with hosting prices these days. https://www.ovh.com/us/dedicated-servers/details-servers.xml?range=HOST&id=2016-HOST-32L https://www.ovh.com/us/dedicated-servers/details-servers.xml... For 12 months will cost you about $800 A somewhat comparable EC2 instance (that still has way lower specs) m3.2xlarge will have an up-front cost of $1772, and an hourly cost of $0.146. So you'd be looking at a total yearly bill of $3051, which would pay for 3 of those dedis. I honestly don't know how anyone could describe that as competitive. > "dedis and EC2 on demand" is an order of magnitude more complex, probably actually more expensive. If you're one of the few people that actually need the rapid scaling that EC2 can provide, sure. But that's an extremely specific and rare use case.
    - blahi 10y ago
      Cloud platforms like AWS offer other things besides just scaling. Like Hadoop processing, hosted databases, email infrastructure, DNS services and the list goes on and on.
    - eropple 10y ago
      I'm alarmingly familiar with OVH. And their hardware failures. And their slow support exactly when you need them most. And their poor inventory control, leaving popular system configurations unavailable when I (or, rather, my clients) need them. I'm also familiar with exactly how misleading a specs comparison ends up when you can architect an application to leverage burstable instances and rapid auto-scaling, letting you require little expenditure during slow periods while still responding gracefully to load. This isn't some deep-magic thing for sites with tens of thousands of instances, either--I've had clients with ten total instances benefit both financially and operationally from building out fairly straightforward elastic capacity (not least because the same code paths necessary for elasticity also makes things like setting up development environments single-click affairs, which enables deeper and more complete testing of the application to ensure correctness). Even were a $2200 premium a credible number, that's eleven hours of my billing rate. Reimplementing what AWS provides even a fairly straightforward system--easy, API-driven system deployment and configuration (and while OVH's public cloud APIs can enable some of this, your "dedis" cannot), straightforward and API-controlled support services of more or less any stripe, integrated system monitoring and alerting--would be hundreds of hours of work, be a worse solution, and require ongoing maintenance. (A decent hosting provider will offer some of that, of course. But not all. And you'll pay for it.)
teknologist 10y ago
Would you get anywhere with using those Scaleway ARM BareMetal servers? They're insanely cheap if you can run on ARM.

lccarrasco 10y ago

This was really great to read, in-depth and interesting, thanks a lot for taking the time to write it. :)

mentos 10y ago

Hey great work! Curious to hear what the client stack was? Did you use LibGDX by chance?

def- 10y ago
Thanks! You can see the client here: https://github.com/ddnet/ddnet https://github.com/ddnet/ddnet SDL2, OpenGL, FreeType, pnglite, zlib, curl, md5, wavpack, opus, json-parser So pretty much low level, keeps the performance and flexibility high.

eggy 10y ago

Awesome and inspiring to me, great work! I am now looking at LFE (Lisp Flavored Erlang) and ELM to create a very small online game. It makes me want to maintain C/C++ chops. It's sad Apple is so walled in that you need a VM to build for OS X, and iOS doesn't even make the list. I have an iPad, but I use an Android phone for that reason, and I only program mobile for Android. Apple is getting better at supporting iOS devs of late though...

ashitlerferad 10y ago

What was the reason for the fork with Teeworlds?

def- 10y ago
Teeworlds is a shooter. People wanted to race in Teeworlds, so a modification was required. From there it developed into multiplayer racing and the last few years of DDNet adding features.

ObeyTheGuts 10y ago

Maidsafe will eliminate all this server problems!

adynatos 10y ago

The author writes: "Reduce the number of syscalls by caching the value of gettimeofday() until a new tick happens or network packet comes in" But I'm pretty sure glibc on recent Linux handles gettimeofday in user-space, without context switch (kernel maps the data to userspace). I guess caching the value locally and updating it 1/sec or something would still help if there are thousands of calls/sec, but not as much as if it was really a syscall.

def- 10y ago
You're right. Unfortunately with virtual machines that doesn't always work. You can enable it manually but that's more inefficient because the Kernel can't use paravirtualization for clock then. I stumbled upon this a month ago: https://ddnet.tw/irclogs/2016-05-17.log https://ddnet.tw/irclogs/2016-05-17.log 21:37 <@deen> A third of the sys calls of a DDNet server are recfrom and sendto each 21:37 <@deen> they always occur in large chunks, so ideal for sendmmsg/recvmmsg 21:38 <@deen> the last third are mostly strange time and gettimeofday calls, I thought I got rid of most of them 21:40 <@deen> server with 30 players causes 3000 syscalls a second 22:00 <@deen> PACKET_MMAP is very cool, reading packets with 0 syscalls, too bad it's not for regular applications (requires root, doesn't work with normal udp socket): https://www.kernel.org/doc/Documentation/networking/packet_mmap.txt 22:04 <@deen> and then the glibc version matters a lot for syscalls. new versions of glibc don't syscall at all for gettimeofday 22:05 <@deen> (or something else is causing that, not sure yet) 22:13 <@deen> Reading the glibc implementation, that's done by vDSO, interesting: http://man7.org/linux/man-pages/man7/vdso.7.html 22:32 <@deen> totally confused why some of our servers use vdso gettimeofday, others not even though they have more recent kernel and glibc 22:41 <@deen> ok, probably depends on the underlying clock that the vps uses. pvclock is used with kvm and doesn't support vdso. but looks like there's some progress being made: https://lkml.org/lkml/2015/12/9/914

urza 10y ago

I would like to see this kind of post for slither.io

qwertyuiop924 10y ago

I just re-downloaded (vanilla) teeworlds the other day. After playing a lot of QW and Xonotic, it's nice to play something like HLDM or TeeWorlds that's a bit more wacky and less competitive. There is no bunnyhopping in teeworlds. Just cute fluffballs, hookshots, and heavy weaponry. Although, for whatever reason, all of the players are in the EU, or SA, and I'm USEast, so the ping's really high. One of many reasons I want to get a GPU that can handle UT4. Yes, mine is really that bad.

qwertyuiop924 10y ago

I just re-downloaded (vanilla) teeworlds the other day. After playing a lot of QW and Xonotic, it's nice to play something like HLDM or TeeWorlds that's a bit more wacky and less competitive. There is no bunnyhopping in teeworlds. Just cute balls, hookshots, and heavy weaponry. Although, for whatever reason, all of the players are in the EU, or SA, and I'm USEast, so the ping's really high. One of many reasons I want to get a GPU that can handle UT4. Yes, mine is really that bad.