9 ms·

€10/m/location is damn impressive. I've run servers for a turn-based asynchronous and real-time casual game for the past three years. With ~700k unique monthly

€10/m/location is damn impressive. I've run servers for a turn-based asynchronous and real-time casual game for the past three years. With ~700k unique monthly players (about 200k/day) we do about 1500 request/s at peak and pay thousands a month for our AWS stack. I'm not mad at it, I think we get great utility for what we pay, but this is lean and mean for realz. Kudos!

def- 10y ago
Thanks! Interesting to hear about your experience as well. By requests per seconds you mean packets? I was curious and with 330 players playing on DDNet right now[1] we get ~7400 packets/s incoming, ~8300 packets/s outgoing. [1] https://ddnet.tw/status/ https://ddnet.tw/status/
- Cyph0n 10y ago
  My guess is that he's talking about HTTP requests.
  - themartorana 10y ago
    Yes, I am talking about HTTP requests. We're just exploring moving over to web sockets, the original implementations were done before that was practical on mobile networks.
hueving 10y ago
AWS is not anywhere near cheap when it comes to raw performance per dollar. You are paying a very large premium for that flexibility of an arbitrary number of instances at any time.
meritt 10y ago
Care to share some details about your setup? 1500 reqs/sec should not cost very much at all, definitely not "thousands a month".
- themartorana 10y ago
  Well, a larger chunk of that is the database. A db.r3.4xlarge runs about $1600 by itself. We have several hundred gig of data, and indexes alone keep us in that instance size for RAM allotment. Aside from that, we run 2-3 c3.xlarge API servers, two c3.medium webserver (hosts Facebook versions of our games) and two more c3.medium servers are joining in for a new launch next month. We have one large (size escapes me) ElastiCache instance. 3 ELBs sit in front of those servers, all in a VPC. Finally, we have 11 or so S3 buckets that back Cloudfront distributions for static content for websites and whatnot. We also pay for several TB of traffic per month, and enterprise support. Finally, we're paying for not having to fix any hardware ever, 5 minute incremental DB backups, a VPC I can define in a JSON doc, automatic failovers, and so much more, managed by yours truly, a software engineer, not a network/server/devops/etc. engineer. Yeah, there's a premium, but so far it's paid off.
  - grogenaut 10y ago
    What drove the choice to the larger instance types instead of more of the small ones? Not questioning your judgment I'm just curious.
    - themartorana 10y ago
      We moved from Python to Go. With Python behind uwsgi->nginx, we'd try to tune the number of uwsgi processes to cores, and it was easy to scale up based on process=number of concurrent requests handled. With Go, every request is handled concurrently, being much more efficient with cores and RAM. I found that my requests/core were being artificially limited by RAM in an autoscaling group (we had scaling triggered at about avg 80% CPU) so I let some more RAM play. Now, instead of 15-20 c3.medium servers at peak, we run three c3.xlarge instances at peak. I can be sure that the larger server will handle more concurrent requests than process-restricted uwsgi, and thus be able to better allocate the resources given to us by larger instances.
      - grogenaut 10y ago
ryanlol 10y ago
Unless you need AWS for the scaling it offers, it tends to be the worst possible host you could choose. And even if you need the scaling, a mixed stack with dedis and EC2 on demand shouldn't be hard to run.
- eropple 10y ago
  > Unless you need AWS for the scaling it offers, it tends to be the worst possible host you could choose. Unless, in turn, you use reserved instances, which bring the price down to equal or cheaper than most other options. "dedis and EC2 on demand" is an order of magnitude more complex, probably actually more expensive, and increases your failure cases significantly.
  - ryanlol 10y ago
    I think you're a little out of touch with hosting prices these days. https://www.ovh.com/us/dedicated-servers/details-servers.xml?range=HOST&id=2016-HOST-32L https://www.ovh.com/us/dedicated-servers/details-servers.xml... For 12 months will cost you about $800 A somewhat comparable EC2 instance (that still has way lower specs) m3.2xlarge will have an up-front cost of $1772, and an hourly cost of $0.146. So you'd be looking at a total yearly bill of $3051, which would pay for 3 of those dedis. I honestly don't know how anyone could describe that as competitive. > "dedis and EC2 on demand" is an order of magnitude more complex, probably actually more expensive. If you're one of the few people that actually need the rapid scaling that EC2 can provide, sure. But that's an extremely specific and rare use case.
    - blahi 10y ago
      Cloud platforms like AWS offer other things besides just scaling. Like Hadoop processing, hosted databases, email infrastructure, DNS services and the list goes on and on.
    - eropple 10y ago
      I'm alarmingly familiar with OVH. And their hardware failures. And their slow support exactly when you need them most. And their poor inventory control, leaving popular system configurations unavailable when I (or, rather, my clients) need them. I'm also familiar with exactly how misleading a specs comparison ends up when you can architect an application to leverage burstable instances and rapid auto-scaling, letting you require little expenditure during slow periods while still responding gracefully to load. This isn't some deep-magic thing for sites with tens of thousands of instances, either--I've had clients with ten total instances benefit both financially and operationally from building out fairly straightforward elastic capacity (not least because the same code paths necessary for elasticity also makes things like setting up development environments single-click affairs, which enables deeper and more complete testing of the application to ensure correctness). Even were a $2200 premium a credible number, that's eleven hours of my billing rate. Reimplementing what AWS provides even a fairly straightforward system--easy, API-driven system deployment and configuration (and while OVH's public cloud APIs can enable some of this, your "dedis" cannot), straightforward and API-controlled support services of more or less any stripe, integrated system monitoring and alerting--would be hundreds of hours of work, be a worse solution, and require ongoing maintenance. (A decent hosting provider will offer some of that, of course. But not all. And you'll pay for it.)
teknologist 10y ago
Would you get anywhere with using those Scaleway ARM BareMetal servers? They're insanely cheap if you can run on ARM.

def- 10y ago

Thanks! Interesting to hear about your experience as well. By requests per seconds you mean packets? I was curious and with 330 players playing on DDNet right now[1] we get ~7400 packets/s incoming, ~8300 packets/s outgoing. [1] https://ddnet.tw/status/ https://ddnet.tw/status/

Cyph0n 10y ago
My guess is that he's talking about HTTP requests.
- themartorana 10y ago
  Yes, I am talking about HTTP requests. We're just exploring moving over to web sockets, the original implementations were done before that was practical on mobile networks.

hueving 10y ago

AWS is not anywhere near cheap when it comes to raw performance per dollar. You are paying a very large premium for that flexibility of an arbitrary number of instances at any time.

meritt 10y ago

Care to share some details about your setup? 1500 reqs/sec should not cost very much at all, definitely not "thousands a month".

themartorana 10y ago
Well, a larger chunk of that is the database. A db.r3.4xlarge runs about $1600 by itself. We have several hundred gig of data, and indexes alone keep us in that instance size for RAM allotment. Aside from that, we run 2-3 c3.xlarge API servers, two c3.medium webserver (hosts Facebook versions of our games) and two more c3.medium servers are joining in for a new launch next month. We have one large (size escapes me) ElastiCache instance. 3 ELBs sit in front of those servers, all in a VPC. Finally, we have 11 or so S3 buckets that back Cloudfront distributions for static content for websites and whatnot. We also pay for several TB of traffic per month, and enterprise support. Finally, we're paying for not having to fix any hardware ever, 5 minute incremental DB backups, a VPC I can define in a JSON doc, automatic failovers, and so much more, managed by yours truly, a software engineer, not a network/server/devops/etc. engineer. Yeah, there's a premium, but so far it's paid off.
- grogenaut 10y ago
  What drove the choice to the larger instance types instead of more of the small ones? Not questioning your judgment I'm just curious.
  - themartorana 10y ago
    We moved from Python to Go. With Python behind uwsgi->nginx, we'd try to tune the number of uwsgi processes to cores, and it was easy to scale up based on process=number of concurrent requests handled. With Go, every request is handled concurrently, being much more efficient with cores and RAM. I found that my requests/core were being artificially limited by RAM in an autoscaling group (we had scaling triggered at about avg 80% CPU) so I let some more RAM play. Now, instead of 15-20 c3.medium servers at peak, we run three c3.xlarge instances at peak. I can be sure that the larger server will handle more concurrent requests than process-restricted uwsgi, and thus be able to better allocate the resources given to us by larger instances.
    - grogenaut 10y ago

ryanlol 10y ago

Unless you need AWS for the scaling it offers, it tends to be the worst possible host you could choose. And even if you need the scaling, a mixed stack with dedis and EC2 on demand shouldn't be hard to run.

eropple 10y ago
> Unless you need AWS for the scaling it offers, it tends to be the worst possible host you could choose. Unless, in turn, you use reserved instances, which bring the price down to equal or cheaper than most other options. "dedis and EC2 on demand" is an order of magnitude more complex, probably actually more expensive, and increases your failure cases significantly.
- ryanlol 10y ago
  I think you're a little out of touch with hosting prices these days. https://www.ovh.com/us/dedicated-servers/details-servers.xml?range=HOST&id=2016-HOST-32L https://www.ovh.com/us/dedicated-servers/details-servers.xml... For 12 months will cost you about $800 A somewhat comparable EC2 instance (that still has way lower specs) m3.2xlarge will have an up-front cost of $1772, and an hourly cost of $0.146. So you'd be looking at a total yearly bill of $3051, which would pay for 3 of those dedis. I honestly don't know how anyone could describe that as competitive. > "dedis and EC2 on demand" is an order of magnitude more complex, probably actually more expensive. If you're one of the few people that actually need the rapid scaling that EC2 can provide, sure. But that's an extremely specific and rare use case.
  - blahi 10y ago
    Cloud platforms like AWS offer other things besides just scaling. Like Hadoop processing, hosted databases, email infrastructure, DNS services and the list goes on and on.
  - eropple 10y ago
    I'm alarmingly familiar with OVH. And their hardware failures. And their slow support exactly when you need them most. And their poor inventory control, leaving popular system configurations unavailable when I (or, rather, my clients) need them. I'm also familiar with exactly how misleading a specs comparison ends up when you can architect an application to leverage burstable instances and rapid auto-scaling, letting you require little expenditure during slow periods while still responding gracefully to load. This isn't some deep-magic thing for sites with tens of thousands of instances, either--I've had clients with ten total instances benefit both financially and operationally from building out fairly straightforward elastic capacity (not least because the same code paths necessary for elasticity also makes things like setting up development environments single-click affairs, which enables deeper and more complete testing of the application to ensure correctness). Even were a $2200 premium a credible number, that's eleven hours of my billing rate. Reimplementing what AWS provides even a fairly straightforward system--easy, API-driven system deployment and configuration (and while OVH's public cloud APIs can enable some of this, your "dedis" cannot), straightforward and API-controlled support services of more or less any stripe, integrated system monitoring and alerting--would be hundreds of hours of work, be a worse solution, and require ongoing maintenance. (A decent hosting provider will offer some of that, of course. But not all. And you'll pay for it.)

teknologist 10y ago

Would you get anywhere with using those Scaleway ARM BareMetal servers? They're insanely cheap if you can run on ARM.