7 ms·
> We're running on average around 25k servers a day 25k number of servers or its the cost of the servers ?
by joe563323 10y ago
> We're running on average around 25k servers a day
25k number of servers or its the cost of the servers ?
- doh 10y agoNumber of servers
- nodesocket 10y agoWow, your bill must be gigantic.
- doh 10y agoWell, it's not small, but definitely not that crazy. We're running an immutable infrastructure and the majority of the servers are a "cattle". To be more specific, we redeploy servers from images instead of replacing just the code. The majority of servers (99.9%) is preemptible (meaning that they will be deleted every 24 hours) and thus we get a huge discount. Also our whole communication is through pubsub so the servers don't communicate directly between each other. Thanks to all these optimizations, we run only what we need at any given time, which keeps the cost low. It's still much larger that anything I've ever expected, but I'm still trying to convince myself that this is the sign of success.
- nodesocket 10y agoAwesome, would you mind sending me an email (in profile), I'd like to pick your brain a bit further.
- doh 10y agoWe were sharing the same office for over a year [geekdom]. You can find my email in of the discussion chains.
- nodesocket 10y agoLol, sorry I failed to put 2+2 together. I can't imagine why you'd guys need 25,000 servers. Your G bill at a minimum must be over 5 million a month. I could be wrong, but those numbers don't add up too me. What instance type are you running?
- doh 10y agoWe don't host on AWS, we host on GCE. Also you're way off with your estimation. We pay significantly less. As the instance go, we have almost everything ranging from many thousands on the micro side to hundreds on the largest side.
- nodesocket 10y ago$200 a server was my estimate which is pretty conservative. I assumed that price included disk and bandwidth per server. 25,000 * $200 = 5 million a month.
- doh 10y agoWell, no. You're calculating with 100% utilization + with normal servers. Preemptible (or spot) instances are way cheaper. For instance 8 core 7GB ram preemptible server on GCE costs around $45/month under full utilization. We may use it for couple of hours a day and then terminate and open a different one. It truly differs minute by minute.
- deleted 10y ago[deleted]
- deleted 10y ago[deleted]
- banachtarski 10y agoYou should have moved to bare metal months/years ago. Things like making your servers preemptible is like putting a bandaid on a gaping bullet hole.
- lma21 10y agoI may be missing something, why is bare metal better in this case (immutable/premptible servers)?
- banachtarski 10y agoJust the sheer number of instances the OP is using has put the usage FAR into the territory where the premiums spent on virtualized instance costs far outweigh any clever strategy one might use to make things cheaper in the cloud. It has nothing to do with what the infrastructure is, and everything to do with raw volume.
- phonon 10y agoSpot instances are super cheap...about 80% less than regular instances.
- banachtarski 10y agoYea but it's still virtualized slow molasses at the end. And still more expensive. My point is that at 25k servers, even a 5% efficiency gain is significant, and I'm willing to bet the margins are still higher.
- doh 10y agoThere are many reasons why bare metal doesn't make much sense for us, at least at the moment. Here are couple: - we're very small team (13 people). We would never be able to manage this kind of infrastructure - we're growing incredibly fast. Just 6 months ago we were running on ~300 servers. We grew so quickly that we are now working with GCE's infrastructure planers to plan our resource needs - even we're not suffering from massive bursts, we don't grow gradually but rather in massive jumps. It would take us months instead of days to grow this quickly on bare metal. This way we can grow in days. - it is way more complicated to host in many geographic locations, especially in asia. We need to be present in many parts of the world and with GCE/AWS/Azure it's very easy and convenient. - we did try to work with some bare metal hosters (like OVH), but they are not able to deliver the servers on the schedule we need. They require way longer times and obviously much longer commitments - the pricing is actually not that different once you run on preemptible (spot) instances. The bare bone would give us some performance boost, but the freedom is worth every penny.
- zeta0134 10y agoI have to wonder if their Lambda service might be worth looking into. Any reason you're using full on servers rather than some kind of a compute engine? I don't have the great fortune of getting to work with those kinds of numbers, so forgive my ignorance here, I'm really curious about your setup. :)
- xtracto 10y agoI'm not the original commented but I have analyzed Lambda for some tasks and found its costs prohibitive. Moreover, aws Lambda has a 300 second timeout limit so it is not feasable for heavy processing.
- doh 10y agoI played with lambda recently. It's further than Functions [0] but it has a lot of limitations. We like to have control over our infrastructure, at least some kind of control. [0] https://cloud.google.com/functions/ https://cloud.google.com/functions/