3 ms·

Very cool project, but I wonder about the economics about using public cloud providers as a gaming host where compute and network are cornerstones to their prof

by kodah 5y ago

Very cool project, but I wonder about the economics about using public cloud providers as a gaming host where compute and network are cornerstones to their profitability.

alias_neo 5y ago
I guess it makes sense for Microsoft to be able to put Xbox game servers in Azure but also game devs can benefit from the scalability of Kubernetes. If have is low tire not paying for hardware you aren't using and vice versa.
imbriaco 5y ago
The vast majority of game servers are hosted on public cloud providers. They turn out to be a pretty perfect example of an ideal cloud workload: - Very elastic demand curve - Generally ephemeral with any necessary persistent state stored off-host - Benefit from geographic distribution of cloud regions because they're generally quite latency sensitive
- deleted 5y ago
  [deleted]
- dijit 5y ago
  That’s not true in my experience. The cost of running on public cloud is extremely prohibitive. There are some things that can run on cloud, but the heavy hitting game servers should really be bare metal for the baseline. I gave a talk about this in Stockholm once, unfortunately it wasn’t recorded but I can share the slides if you want.
  - imbriaco 5y ago
    It's very true in my experience, and we've looked pretty hard at all the options. Thanks for the offer of the slides, I have first hand experience with this at scale. It certainly depends a lot on how sophisticated your autoscaling is and how closely you're able to follow demand to limit waste, how well you can manage per-host utilization, whether you are CPU or memory bound, and lots of other factors. But even at truly massive scale the cloud hosting option can be very competitive without nearly as much management overhead.
    - dijit 5y ago
      Interesting! I have the opposite experience and I also operated at quite a decent scale. I wonder why our experiences are so different. For context I was running 2,500+ instances at peak with 40vCPU and 256GiB of memory: the most expensive of those of course being in the regions with low density of players like South America and Australia. We also had a predictive auto-scaler for our cloud components, I would estimate that our waste was 15% at any time, but if we ran bare metal only it would have been cheaper (except for the operational cost and the fact we needed lengthy commitments for hardware)
      - imbriaco 5y ago
        We run rather a lot more than 2500 instances at peak and a variety of instance types dependent on game mode. Our waste is on the order of 5-7% depending on the time of day -- we can run less waste on the downward slope of the demand curve for instance. We're also seeing some pretty compelling results from ARM based instances: https://www.wired.com/sponsored/story/changing-the-game/ https://www.wired.com/sponsored/story/changing-the-game/
        dijit 5y ago
        Aha, Fortnite. You guys will get discounts that we will never get, please keep that in mind. Our waste was the inverse of yours, less waste on the upswing and more waste on the downswing, due to the long lived nature of our instances. One instance takes roughly 1,000 players, but if you as a player decide to just keep playing, then we won't kick you out of the game for some hours. Some percentage of players will just continue playing as long as possible without matchmaking or map transfers.
- jayd16 5y ago
  Not nearly as ephemeral as web traffic though. Games usually last longer than a spot instance could normally support.
BaconPackets 5y ago
It might be more expensive to run but you have savings in terms of time to deployment, orchestration and the overall maturity of the cloud itself. Managing bare metal at scale is challenging.
- dijit 5y ago
  This is very true. But never underestimate the cost of running gameservers in cloud. For us (with really good discounts and a fantastic custom made predictive auto-scaler) it was still more than double the cost than using our previously used bare-metal outsourcing provider. The cost difference would have been enough to hire 2.5x my team as additional resources, but that differs with scale (and so do the discounts)… So YMMV.
remram 5y ago
You can always run Kubernetes on bare metal. This seems easy in this case because you don't need persistent storage or load balancing to use Thundernetes, and you gain the ability to use the generic Kubernetes tools for self-healing, scaling, rolling deployments, etc.

by kodah 5y ago

Very cool project, but I wonder about the economics about using public cloud providers as a gaming host where compute and network are cornerstones to their profitability.

alias_neo 5y ago

I guess it makes sense for Microsoft to be able to put Xbox game servers in Azure but also game devs can benefit from the scalability of Kubernetes. If have is low tire not paying for hardware you aren't using and vice versa.

imbriaco 5y ago

The vast majority of game servers are hosted on public cloud providers. They turn out to be a pretty perfect example of an ideal cloud workload: - Very elastic demand curve - Generally ephemeral with any necessary persistent state stored off-host - Benefit from geographic distribution of cloud regions because they're generally quite latency sensitive

deleted 5y ago
[deleted]
dijit 5y ago
That’s not true in my experience. The cost of running on public cloud is extremely prohibitive. There are some things that can run on cloud, but the heavy hitting game servers should really be bare metal for the baseline. I gave a talk about this in Stockholm once, unfortunately it wasn’t recorded but I can share the slides if you want.
- imbriaco 5y ago
  It's very true in my experience, and we've looked pretty hard at all the options. Thanks for the offer of the slides, I have first hand experience with this at scale. It certainly depends a lot on how sophisticated your autoscaling is and how closely you're able to follow demand to limit waste, how well you can manage per-host utilization, whether you are CPU or memory bound, and lots of other factors. But even at truly massive scale the cloud hosting option can be very competitive without nearly as much management overhead.
  - dijit 5y ago
    Interesting! I have the opposite experience and I also operated at quite a decent scale. I wonder why our experiences are so different. For context I was running 2,500+ instances at peak with 40vCPU and 256GiB of memory: the most expensive of those of course being in the regions with low density of players like South America and Australia. We also had a predictive auto-scaler for our cloud components, I would estimate that our waste was 15% at any time, but if we ran bare metal only it would have been cheaper (except for the operational cost and the fact we needed lengthy commitments for hardware)
    - imbriaco 5y ago
      We run rather a lot more than 2500 instances at peak and a variety of instance types dependent on game mode. Our waste is on the order of 5-7% depending on the time of day -- we can run less waste on the downward slope of the demand curve for instance. We're also seeing some pretty compelling results from ARM based instances: https://www.wired.com/sponsored/story/changing-the-game/ https://www.wired.com/sponsored/story/changing-the-game/
      - dijit 5y ago
        Aha, Fortnite. You guys will get discounts that we will never get, please keep that in mind. Our waste was the inverse of yours, less waste on the upswing and more waste on the downswing, due to the long lived nature of our instances. One instance takes roughly 1,000 players, but if you as a player decide to just keep playing, then we won't kick you out of the game for some hours. Some percentage of players will just continue playing as long as possible without matchmaking or map transfers.
jayd16 5y ago
Not nearly as ephemeral as web traffic though. Games usually last longer than a spot instance could normally support.

BaconPackets 5y ago

It might be more expensive to run but you have savings in terms of time to deployment, orchestration and the overall maturity of the cloud itself. Managing bare metal at scale is challenging.

dijit 5y ago
This is very true. But never underestimate the cost of running gameservers in cloud. For us (with really good discounts and a fantastic custom made predictive auto-scaler) it was still more than double the cost than using our previously used bare-metal outsourcing provider. The cost difference would have been enough to hire 2.5x my team as additional resources, but that differs with scale (and so do the discounts)… So YMMV.

remram 5y ago

You can always run Kubernetes on bare metal. This seems easy in this case because you don't need persistent storage or load balancing to use Thundernetes, and you gain the ability to use the generic Kubernetes tools for self-healing, scaling, rolling deployments, etc.