4 ms·

I run game servers. Our intra-day swing gives us about 10x traffic during primetime hours compared to off hours. Autoscaling sheds a lot of overhead.

by zemo 9y ago

pathseeker 9y ago
Have you run the calculations to determine where that cutoff is? If you have a daily primetime of 10x traffic that's pretty consistently the same amount, then it's still cheaper (if we're talking the scale of hundreds of instances) to rack up the servers in a datacenter (COLO or your own) with the capacity for the standard 10x load. Auto-scaling instances on EC2 only makes sense if you don't have the in-house skills to troubleshoot/manage hardware servers or if the traffic is unpredictable and the bursts are infrequent (e.g. once or twice a week or just busy for a short time period of the year).
- Thaxll 9y ago
  You don't use physical servers in a DC to pay the 10x peak time, it's a big waste of money, if you want to go this way you pay the minimum numbers of servers at any time and you use cloud for the rest. But again I think noways it doesn't make any sense to buy servers in DC. You pay for hardware that you need to mange and have to be sure that you will be using it after the game launch. ( most games player base decrease by 2x just after release. )
  - twic 9y ago
    > You don't use physical servers in a DC to pay the 10x peak time, it's a big waste of money No, it's cheaper once you factor in the real performance of physical vs virtual machines. Someone really needs to bring some actual numbers to this conversation.
    - Thaxll 9y ago
      Numbers were done it's cheaper to run in the cloud nowdays, also VM ( EC2 on AWS ) have the same perf that baremetal. Have a read: https://aws.amazon.com/ec2/instance-types/c5/ https://aws.amazon.com/ec2/instance-types/c5/
      - twic 9y ago
        > VM ( EC2 on AWS ) have the same perf that baremetal Extraordinary claims require extraordinary evidence.
        ApolloFortyNine 9y ago
        AWS makes $20 billion a year in revenue. Maybe it's possible that you're wrong?
        twic 9y ago
        That is a complete non sequitur. AWS has traditionally never been the cheapest way to rent machine power, but people don't use it because it's cheap, they use it because it's easy to get started with a wide range of facilities without dedicated sysadmins.
        cthalupa 9y ago
        http://www.brendangregg.com/blog/2017-11-29/aws-ec2-virtualization-2017.html http://www.brendangregg.com/blog/2017-11-29/aws-ec2-virtuali... Brendan Gregg has written here and elsewhere about the performance of VMs on EC2. Worth reading if you're interested in figuring out how close it is (or isn't).
- spookthesunset 9y ago
  > in-house skills to troubleshoot/manage hardware servers Which costs time, money and opportunity cost. Something that all the "rack and stack" people seem to forget when they compare owning bare-metal servers against managed cloud providers. The amount of time these "bare-metal" shops waste building and maintaining infrastructure management tools, scaling tools, reporting and monitoring tools is mind boggling. None of this stuff has anything to do with the core business the company actually is in. Opportunity costs here are immense....
evanosaurus 9y ago
Hi. Guy who does the autoscaling code here at Epic. It's really fun to watch the graphs of how many server sessions we have in a region at any given time over the course of a day.
- matt_s 9y ago
  I like that you're in here commenting. After reading through the postmortem, it reminds me of scaling issues we had at previous job. We had hundreds of thousands of clients that would get "hyper active" if they had issues connecting, retry loops FTW. System goes down and it was hard resurrecting it since the traffic just kept pounding away. No autoscaling, no cloud. It woulda been handy to just fire up some more servers, let alone have things auto-scale via CPU %.

I run game servers. Our intra-day swing gives us about 10x traffic during primetime hours compared to off hours. Autoscaling sheds a lot of overhead.

by zemo 9y ago

I run game servers. Our intra-day swing gives us about 10x traffic during primetime hours compared to off hours. Autoscaling sheds a lot of overhead.

pathseeker 9y ago

Have you run the calculations to determine where that cutoff is? If you have a daily primetime of 10x traffic that's pretty consistently the same amount, then it's still cheaper (if we're talking the scale of hundreds of instances) to rack up the servers in a datacenter (COLO or your own) with the capacity for the standard 10x load. Auto-scaling instances on EC2 only makes sense if you don't have the in-house skills to troubleshoot/manage hardware servers or if the traffic is unpredictable and the bursts are infrequent (e.g. once or twice a week or just busy for a short time period of the year).

Thaxll 9y ago
You don't use physical servers in a DC to pay the 10x peak time, it's a big waste of money, if you want to go this way you pay the minimum numbers of servers at any time and you use cloud for the rest. But again I think noways it doesn't make any sense to buy servers in DC. You pay for hardware that you need to mange and have to be sure that you will be using it after the game launch. ( most games player base decrease by 2x just after release. )
- twic 9y ago
  > You don't use physical servers in a DC to pay the 10x peak time, it's a big waste of money No, it's cheaper once you factor in the real performance of physical vs virtual machines. Someone really needs to bring some actual numbers to this conversation.
  - Thaxll 9y ago
    Numbers were done it's cheaper to run in the cloud nowdays, also VM ( EC2 on AWS ) have the same perf that baremetal. Have a read: https://aws.amazon.com/ec2/instance-types/c5/ https://aws.amazon.com/ec2/instance-types/c5/
    - twic 9y ago
      > VM ( EC2 on AWS ) have the same perf that baremetal Extraordinary claims require extraordinary evidence.
      - ApolloFortyNine 9y ago
        AWS makes $20 billion a year in revenue. Maybe it's possible that you're wrong?
        twic 9y ago
        That is a complete non sequitur. AWS has traditionally never been the cheapest way to rent machine power, but people don't use it because it's cheap, they use it because it's easy to get started with a wide range of facilities without dedicated sysadmins.
      - cthalupa 9y ago
        http://www.brendangregg.com/blog/2017-11-29/aws-ec2-virtualization-2017.html http://www.brendangregg.com/blog/2017-11-29/aws-ec2-virtuali... Brendan Gregg has written here and elsewhere about the performance of VMs on EC2. Worth reading if you're interested in figuring out how close it is (or isn't).
spookthesunset 9y ago
> in-house skills to troubleshoot/manage hardware servers Which costs time, money and opportunity cost. Something that all the "rack and stack" people seem to forget when they compare owning bare-metal servers against managed cloud providers. The amount of time these "bare-metal" shops waste building and maintaining infrastructure management tools, scaling tools, reporting and monitoring tools is mind boggling. None of this stuff has anything to do with the core business the company actually is in. Opportunity costs here are immense....

evanosaurus 9y ago

Hi. Guy who does the autoscaling code here at Epic. It's really fun to watch the graphs of how many server sessions we have in a region at any given time over the course of a day.

matt_s 9y ago
I like that you're in here commenting. After reading through the postmortem, it reminds me of scaling issues we had at previous job. We had hundreds of thousands of clients that would get "hyper active" if they had issues connecting, retry loops FTW. System goes down and it was hard resurrecting it since the traffic just kept pounding away. No autoscaling, no cloud. It woulda been handy to just fire up some more servers, let alone have things auto-scale via CPU %.