6 ms·
We moved back and forth between AWS and GCE (based on who gave us free credits). Once we ran out, we chose GCE and never regretted it. GCE has many quirks, for
by doh 10y ago
We moved back and forth between AWS and GCE (based on who gave us free credits). Once we ran out, we chose GCE and never regretted it.
GCE has many quirks, for instance the inconsistency between API and the UI, it misses the richness of the services offered by AWS but everything GCE does offer is just faster, more stable and much more consistent.
One of the biggest problems with AWS is that once you outgrow the assigned limits, it becomes a hell to get more resources from them. We're running on average around 25k servers a day, majority of them preemptible (spot). AWS requires that you request exact type and location for your instances. GCE only asks for a region and then overall resources (e.g. number of CPUs).
Also the pricing is much less complicated. 1 core costs y, 32 cores cost 32*y.
- paulddraper 10y ago> AWS requires that you request exact type and location for your instances Are you running 25k without using Spot Fleets? Spot fleets let you specify value per instance type, and then total value you need. ("Value" could be CPU, memory, network or whatever) AWS will maintain the lowest cost spot instances that fulfill your requirements.
- doh 10y agoWe couldn't even get them to approve 5k servers on AWS. We run everything in GCE.
- paulddraper 10y agoThat is hard to believe. Their case studies have organizations using thousands or tens of thousands of servers for scientific computing, etc.
- doh 10y agoI didn't blur anything so you can see for yourself http://imgur.com/14upQ5j http://imgur.com/14upQ5j
- mdekkers 10y agoCase studies tend to be written by the marketing team, and will tell you what the marketing team wants you to believe. I'll take "hey, this is my hands-on experience" over "this is what we want you to think" everytime.
- paulddraper 10y agoI understand spin, but the claim is that AWS markets a service/product that they will not provide.
- dekhn 10y agoThose studies always involve close coordination between the PMs in the Cloud org and the client.
- joe563323 10y ago> We're running on average around 25k servers a day 25k number of servers or its the cost of the servers ?
- doh 10y agoNumber of servers
- nodesocket 10y agoWow, your bill must be gigantic.
- doh 10y agoWell, it's not small, but definitely not that crazy. We're running an immutable infrastructure and the majority of the servers are a "cattle". To be more specific, we redeploy servers from images instead of replacing just the code. The majority of servers (99.9%) is preemptible (meaning that they will be deleted every 24 hours) and thus we get a huge discount. Also our whole communication is through pubsub so the servers don't communicate directly between each other. Thanks to all these optimizations, we run only what we need at any given time, which keeps the cost low. It's still much larger that anything I've ever expected, but I'm still trying to convince myself that this is the sign of success.
- nodesocket 10y agoAwesome, would you mind sending me an email (in profile), I'd like to pick your brain a bit further.
- doh 10y agoWe were sharing the same office for over a year [geekdom]. You can find my email in of the discussion chains.
- 10y ago
- dmcatrollin 10y agoSounds like there's a lot of money in DMCA take downs.
- doh 10y agoWe don't do takedown. We do this https://blog.pex.com/the-daily-wars-of-online-video-platforms-18c60ad6bb5e#.rx6v0f8t8 https://blog.pex.com/the-daily-wars-of-online-video-platform...
- dmcatrollin 10y agoActually that's super neat. I grew to envy you as I read. It looks like interesting work.
- deleted 10y ago[deleted]
- geggam 10y agoAt 25k servers / day one would think pulling this into real hardware and doing capacity planning would be cost effective.
- Turing_Machine 10y agoThat's what I was thinking. Unless they're really bursty (so they only need the 25k servers for an hour at a time or whatever), I would think that 25k servers would be well beyond the number where it makes more economic sense to build out your own data center.
- deleted 10y ago[deleted]
- user5994461 10y ago> Unless they're really bursty Last I checked. AWS has hourly billing and Google Cloud has sub-hour billing. If they are really bursty, they are in troubles with AWS pricing model. > the number where it makes more economic sense to build out your own data center. I would think that this number doesn't exist... unless you have already built datacenters for yourselves and you have a major internal expertise in that.
- geggam 10y agoI can tell you less than 1000 servers in AWS runs around 200-300k US / month I am not sure what he is doing at 25k servers but you can see real world math puts this in a serious money range. Data centers can be built for less. I wonder if the down voters on the hardware comments actually run services at scale.
- doh 10y agoWhat kind of servers run at that price range? I've listed [0] some of the reasons why are we hosting in cloud instead of our own datacenter. [0] https://news.ycombinator.com/item?id=13260805 https://news.ycombinator.com/item?id=13260805
- lma21 10y agoBased on which criteria do you preempt your servers? 25K sounds like a lot !
- doh 10y agoWhat do you mean? We use them for many sorts of tasks. We have only couple of fixed servers that are keeping the state of the whole infrastructure and databases that keep the data.
- vgt 10y agoYour post has motivated me to blog some details about the difference between AWS EC2 Spot Instances and Google Cloud's Preemptible VMs: https://medium.com/@thetinot/google-clouds-spot-instances-win-big-and-you-should-too-5b244ca3facf#.nl66oyyol https://medium.com/@thetinot/google-clouds-spot-instances-wi... (work at Google Cloud)