9 ms·

I love GCP but honestly have no idea how these big internet properties can make the $ case for “the cloud”, unless they’re also planning to fire 90% of the Ops/

I love GCP but honestly have no idea how these big internet properties can make the $ case for “the cloud”, unless they’re also planning to fire 90% of the Ops/SysAdmin team (ala Evernote). We run on bare metal and to move to GCP would triple our hosting cost, not including bandwidth (which is free* for us). Do people get discounts for doing PR? I’m really curious.

merb 9y ago
most cost cuts come from automation (as you already noticed). but not just more automation in ops/sysadmin space, but mostly because you can just free resources you don't need. especially one-off jobs (analytics, etc.) take a huge amount of resources for a very short amount of time and after that time the resources aren't needed. on clouds you can hope to schedule them preemtivly. also most cost savings come at night, if you are not active on the whole globe you probably only need half of your resources at night. even if your global the chances are high that at some time you still need way less resources. etsy is a marketplace on most marketplaces there are times where demand is really high and demand is certainly not so high. (christmas, black firday, regressions, etc..) also if you self host, high availability can and is a pain. monitoring your database is a pain, it takes a huge amount of time, which most companies do not have. p.s. working at a small company and services like RDS/gcloud postgres! (thanks google) google buckets (not sure how they are called their name is not as cool as s3)/S3 are a huge win for us.
- khc 9y ago
  "google buckets"/Google S3 is called GCS: Google Cloud Storage https://cloud.google.com/storage/ https://cloud.google.com/storage/ Aside: strangely enough googling GCS gives me Glasgow Coma Scale, you'd think by now google would have personalized the result for me
- ThrustVectoring 9y ago
  Honest question: why not use both? If spinning up cloud computing resources is the most cost-efficient way to do those computing tasks, why not have that be the route for doing one-off computing tasks while your own machines run the workload for recurring stuff? Is there overhead for not having 100% of your computing done on a cloud platform that I'm not aware of or something?
jbob2000 9y ago
GCP is an operational expense, whereas bare metal is operational + capital expense. It might be more expensive for cloud, but the accounting looks a little cleaner. I worked for a company that had bare metal. They were always shipping parts around, lots of movement back and forth between the data center, lots of night deploys for ops team which meant a burden on the HR department. There was a general anxiety about security. It might be "cheaper" on paper to run bare metal, but it means your managers and executives have to make a lot of dinky little decisions all the time. I'd happily pay more to "push and forget".
- foobarbazetc 9y ago
  You don’t need to worry about that stuff with someone like IBM SoftLayer and a bunch of other professional unmanaged/semi-managed bare metal. E.g. WhatsApp have been on SoftLayer for a long time and are still there many years post-acquisition.
geofft 9y ago
I don't have very detailed knowledge, but my impression is that this move doesn't make long-term financial sense for the company, but it's being pushed by activist investors / board members who want to appear like they're good stewards of money because investors think cloud is cheaper / etc. (Note that this is on investors.etsy.com, not their tech blog; you can also find a call for public cloud in the Black & White Capital letter, in the NYTimes story from a couple weeks ago, etc.) I heard some rumors about this when I was applying for a job right before the layoffs happened, and I didn't hear anyone expressing interest in moving out of their datacenters before the layoffs.
- foobarbazetc 9y ago
  Yeah. The exact same thing happened at Evernote. They hit a sales slump, new “corporate” management, cuts etc -> GCP. The Ops people there were pretty proud of their infrastructure, but they took it to a pretty extreme level. I think they were racking their own stuff and even running their own routers etc.
  - lykr0n 9y ago
    Racking your own servers and running your own routers is extreme? I don't know about you, but I haven't set foot in the DC in two years after initial setup.
    - foobarbazetc 9y ago
      Yeah I meant to put it in quotes. :) Hard to find B2C type SaaS companies doing that though (with the exception of Pinboard, who’s crazy :) We actually haven’t had a hardware failure (apart from HDD that have all been replaced with SSD) in something like 7 years. We don’t even pay big bucks for Dell, HP etc. Just cheap SuperMicro boxes.
      - amigoingtodie 9y ago
        What does your hardware and software stack look like?
olingern 9y ago
There is so much more overhead with bare metal. There's an entire other side of building a web server that devs, thankfully, never have to see. OpenCompute [1] aims to standardize what that process looks like. To think that you can run bare metal, and do better than industry standards is naive IMO. In the end, if you're worrying more about anything than business logic, you're not really delivering value -- you're solving engineering problems. You can sacrifice that control and solving problems for greater abstractions, i.e. serverless, fargate, etc. Unless you have some extreme security requirements (which most don't) you probably don't have a good reason to run bare metal. [1] - http://www.opencompute.org/ http://www.opencompute.org/
- geofft 9y ago
  Cloud providers have to make money, though. You're doing more with bare metal, yes, but you're not doing much more (especially compared to learning how to use a cloud infrastructure effectively), and the cost may not be worthwhile. Generally, the cost of buying and running one server in a datacenter and staffing the ops team needed to keep the hardware running is cheaper than paying for a 24/7 server of comparable specs in the cloud. And if you're already running in a datacenter, it's extremely not obvious that the cost of moving your workload ever will be net profitable. Of course, if you don't need comparable specs to an entire physical server (I do a lot of personal stuff on a t2.micro!) or don't need the same capacity 24/7, the math is very different. The cloud is definitely cost-effective for elastic workloads.
  - olingern 9y ago
    > Generally, the cost of buying and running one server in a datacenter and staffing the ops team needed to keep the hardware running is cheaper than paying for a 24/7 server of comparable specs in the cloud I would disagree here. You can run the metrics for your stack, but it's likely cheaper on AWS. https://awstcocalculator.com/ https://awstcocalculator.com/ If you don't need to compute something but twice a day, or just certain times -- you can pay pennies on the dollar for a serverless function. There's no need for an entire server for things like this.
azurezyq 9y ago
Operation cost is only one side. There are more things: * Well designed security / encryption may cost a lot. * Compliance may be a pain. * multi-DC and HA * Surge resource usage over holidays. * "Sleeping" resources at night or wasted.
- tedmiston 9y ago
  Support for auto-scaling services in a self-managed environment too
- ehsankia 9y ago
  There's also just all sorts of other extra features you may get for free. These are things that may not be worth it for you to spend precious dev time developing, but since cloud providers can make it once and offer it to thousands of customers, it makes much more sense. You can look at the full list of GCP features [0], and it's growing every month. [0] https://cloud.google.com/products/ https://cloud.google.com/products/
- lethargic_meat 9y ago
  A critical security issue in public cloud is that users often misunderstand what infrastructure security entails. I've been in public cloud pitches that sold their security measures as a golden standard that made my managers believe there's no need for a security team anymore. No need for protocols or proper architecture as x cloud has enough flashy certifications.
spencerhakim 9y ago
I would be shocked if Etsy/Spotify/whoever else wasn't cutting a deal for a significant discount, getting these name brands on GCP is great PR for the platform.
- jasonlotito 9y ago
  You don't even need to be a name brand to get discounts.
  - spencerhakim 9y ago
    Of course, but I'm sure being a name brand helps them negotiate a larger discount.
Thaxll 9y ago
With bare metal you pay for servers you don't use. Cloud is more expensive on the compute side but you can really scale on your usage pattern, also security on the cloud is way better that on bare metal.
pm90 9y ago
As with everything else, the prices will reduce over time. But to be honest: you are not the market. If you have a functioning, reliable infrastructure that isn't too expensive or PITA to maintain, you have a good enough infra and should continue using that. If you're a startup who wants to minimize capex and quickly spin up a reliable, secure infra to host your app on to get it out of the door ASAP, you would go with cloud. If you're a non-technical company with a small IT department that wants to have an online presence and/or build internal online tools without having to build out DC's everywhere, you would go with cloud. I could come up with dozens and dozens more. Cloud isn't optimal for everyone, but it is VERY helpful for certain organizations.
- alexasmyths 9y ago
  "If you're a startup who wants to minimize capex and quickly spin up a reliable, secure infra to host your app on to get it out of the door ASAP, you would go with cloud." I agree, but I don't think that's the market either. I think the market is in massive big-corps, moving their IT infrastructure to the cloud. Like regular, non-tech companies. Small startup bills might be hundreds, thousands, possibly 100's of K but probably not millions. Big corp contracts may be millions - and stable.
  - toomuchtodo 9y ago
    At megacorp scales, you're wasting money going to the cloud unless your load profile is highly variable. GCP/AWS profit margins are very high versus bare metal, and if you're megacorp scale you can afford ops and bare metal you can depreciate over several years. (and before you trot out Netflix, the vast majority of their bits are served out of OpenConnect appliances colocated at eyeball ISPs and peering points)
    - jermaustin1 9y ago
      I recently watched a Mega Corp fire the entire IT Ops department and hire a handful cloud consultants to move all of their current vSphere (thousands of VMs) infrastructure to Azure. The rationalization was that in the long run it was cheaper to maintain the status quo, and not have to worry about networking, firewall, and replication issues by hand. I'm pretty sure MS also gave them a large discount at that scale.
manigandham 9y ago
These calculations don't make sense unless you really consider TCO over any appreciable amount of time. It's easy to say you can buy a huge server for a few thousand but the "cloud" provides incredible agility, security, and basically removes operational overhead. When compared to buying, maintaining, and decommissioning servers and software along with the time and personnel costs, it becomes a much closer race. Include global scale, elastic capacity, managed services, and support policies, and it's closer still. Rarely are companies paying 3x IT spend just to run in the cloud, especially for large companies like Etsy.
noahl 9y ago
To answer your question, people get discounts just for negotiating, no PR required. I would guess the conversation went something like this: "___ is how much we pay to host Etsy on bare metal right now. Can you match that?" "Yes." I'm oversimplifying - there are a bunch of other considerations, like guarantees about future prices and capacity availability and whether having more flexibility makes it worthwhile for Etsy to pay more, but fundamentally Etsy looked at their current costs and asked Google to make an offer. (I would assume they actually had all three major players bid, and Google won.) Also to your point, the announcement specifically called out Google's AI and big data capabilities. Those are workloads that can involve a lot of bursting, so the cloud might be especially attractive. Imagine trying to implement BigQuery in bare metal - you can do it, but you have to have a lot of extra servers lying around just in case someone tries to run an analytics query. The same goes even more strongly for something like a model training workload. For bursty enough workloads, running in the cloud can be cheaper than bare metal even given the cloud provider's profit margin. Depending on how much model training Etsy wants to do, something like TPUs might have also given Google a fundamental edge in costs over bare metal or other providers. I don't know enough to know exactly what factors went into Etsy's decision, but I think these are reasonable ideas.
- gesman 9y ago
  I not sure about them asking AWS to bid as they'd see AMZN in a "conflict of interest" online ecommerce space.
  - alpha_squared 9y ago
    As far as I know, which is very little generally, AWS doesn't bid. Prices are as advertised. Not sure if GCP is similar. Azure certainly does, though.
    - jimmywanger 9y ago
      I know for a fact that AWS that AWS negotiates. My buddy used to work at Netflix and he told me that they got a really good rate for AWS that was not available to the general public. They might not bid in the strictest sense of the word, but when the vendors are mostly 3 players (Google, Amazon, and Microsoft) you just ask each one for a quote. I don't see how functionally that is different from bidding. Btw, I know there are other hosting providers out there (Rackspace comes to mind). I can't think of any one of them that would be able to deal with Netflix scale without shitting the bed, missing features, or an almost unacceptable lead time as they build out infrastructure.
blaisio 9y ago
The hardware and bandwidth costs are only small parts of the full cost of running your own infrastructure. There are so many other benefits: - you don't have to support your own low level services like Pub/Sub, Cloud Storage, Spanner, etc. - you don't have to worry as much about things like HIPAA compliance - developers don't have to wait for you to allocate more hardware, greatly slowing down the software development process - it's much easier to test your applications, since you can spin up a full copy using rented infrastructure, and then throw it all away once your tests are complete - many other benefits that I'm forgetting And of course if you buy enough from them, you do get additional discounts. For many (most? I don't have data) companies, the actual cost of the infrastructure and bandwidth itself is tiny compared to all the other "hidden costs".
- deleted 9y ago
  [deleted]
- ShakataGaNai 9y ago
  So having experienced Ops in both cloud and non cloud companies. A few rebuttals: - Bandwidth is VERY expensive in AWS compared to what you can buy in Datacenters. A 1 GBPS connection is so "cheap" you can get it and a 42U rack at HE.net for $400/mo ( http://he.net/colocation.html http://he.net/colocation.html ). That same 1GBPS is roughly 300TB/month. Assuming it's all outbound - that's roughly $21k in AWS. So bandwidth is a small cost IF you're not moving large amounts of data out of AWS. If you stream media - it can be a HUGE portion of your bill. - HIPAA Compliance - That and other compliance aspects assumes that you need those compliance frameworks. In enterprise world that might be the case (SOC2 and ISO27k1 are very common requirements for Enterprise SaaS clients). However in the B2C? Users don't care. - Waiting for hardware allocation. Once you get to a certain size you have hardware spare, typically. Developers can use that for their tests. You can also have your datacenter/operations group work with your developers during project planning so the code and HW can be ready at the same time. That being said, I don't disagree there are advantages to the cloud. I'm AWS certified, I drink the coolaid. But saying that there are "so many other benefits" is ignoring the fact that there are "so many benefits" to on prem too. There are pro's and cons to both sides.
NelsonMinar 9y ago
Related: "Etsy slashes almost a quarter of its staff in attempt to refocus" https://www.engadget.com/2017/06/21/etsy-layoffs-focus-on-core-marketplace/ https://www.engadget.com/2017/06/21/etsy-layoffs-focus-on-co...
dboreham 9y ago
>I’m really curious. I think you can imagine a scenario under which the end goal to have this headline was someone’s primary goal for the past couple of years.
7ewis 9y ago
I work for a company similar to GrubHub outside the US. 'The Cloud' works great for us. Think, how many people order food at 9AM on a Monday? Therefore, we only have a very small amount of servers running. However, 7PM on a Friday... thousands. Yes, it's still mega expensive. But allows us to scale when we need it and not have to worry about the hardware, cost, or management of it.
apendleton 9y ago
Two big savings opportunities: * elastic demand, either in the sense that your global traffic has peaks and troughs, or your global traffic is steady but its global distribution changes such that the balance of where you're deployed would ideally change over time; in either case you can benefit enormously from automatic scaling * workloads that can take advantage of the spot market (or whatever GCP's equivalent is)
mathattack 9y ago
Lifetime costs for much of this technology is more people costs. Also, much of the benefit comes from what's on top of the hosting. (Not having to worry about old versions of software, containers, better security, no issues if the one person who knows what's going on quits, etc)
seanwilson 9y ago
> I love GCP but honestly have no idea how these big internet properties can make the $ case for “the cloud”, unless they’re also planning to fire 90% of the Ops/SysAdmin team (ala Evernote). What kind of price differences are you thinking about? How much hardware and how many people are employed to take care of this? Does the tripling of cost take salaries into account? Having to hire a team that will maintain hardware, deal with scalability and give guaranteed uptime isn't cheap and requires a lot of management. I'm not saying it never makes sense, but offloading all that work on to someone like AWS and Google who have teams doing this for you reliably can be well worth the cost.
cobookman 9y ago
We should talk. Email me at bookman@google.com, and I'll be sure to get you a competitive pricing estimate. (Disclamer: I'm a Cloud Solutions Engineer at Google Cloud)

merb 9y ago

most cost cuts come from automation (as you already noticed). but not just more automation in ops/sysadmin space, but mostly because you can just free resources you don't need. especially one-off jobs (analytics, etc.) take a huge amount of resources for a very short amount of time and after that time the resources aren't needed. on clouds you can hope to schedule them preemtivly. also most cost savings come at night, if you are not active on the whole globe you probably only need half of your resources at night. even if your global the chances are high that at some time you still need way less resources. etsy is a marketplace on most marketplaces there are times where demand is really high and demand is certainly not so high. (christmas, black firday, regressions, etc..) also if you self host, high availability can and is a pain. monitoring your database is a pain, it takes a huge amount of time, which most companies do not have. p.s. working at a small company and services like RDS/gcloud postgres! (thanks google) google buckets (not sure how they are called their name is not as cool as s3)/S3 are a huge win for us.

khc 9y ago
"google buckets"/Google S3 is called GCS: Google Cloud Storage https://cloud.google.com/storage/ https://cloud.google.com/storage/ Aside: strangely enough googling GCS gives me Glasgow Coma Scale, you'd think by now google would have personalized the result for me
ThrustVectoring 9y ago
Honest question: why not use both? If spinning up cloud computing resources is the most cost-efficient way to do those computing tasks, why not have that be the route for doing one-off computing tasks while your own machines run the workload for recurring stuff? Is there overhead for not having 100% of your computing done on a cloud platform that I'm not aware of or something?

jbob2000 9y ago

GCP is an operational expense, whereas bare metal is operational + capital expense. It might be more expensive for cloud, but the accounting looks a little cleaner. I worked for a company that had bare metal. They were always shipping parts around, lots of movement back and forth between the data center, lots of night deploys for ops team which meant a burden on the HR department. There was a general anxiety about security. It might be "cheaper" on paper to run bare metal, but it means your managers and executives have to make a lot of dinky little decisions all the time. I'd happily pay more to "push and forget".

foobarbazetc 9y ago
You don’t need to worry about that stuff with someone like IBM SoftLayer and a bunch of other professional unmanaged/semi-managed bare metal. E.g. WhatsApp have been on SoftLayer for a long time and are still there many years post-acquisition.

geofft 9y ago

I don't have very detailed knowledge, but my impression is that this move doesn't make long-term financial sense for the company, but it's being pushed by activist investors / board members who want to appear like they're good stewards of money because investors think cloud is cheaper / etc. (Note that this is on investors.etsy.com, not their tech blog; you can also find a call for public cloud in the Black & White Capital letter, in the NYTimes story from a couple weeks ago, etc.) I heard some rumors about this when I was applying for a job right before the layoffs happened, and I didn't hear anyone expressing interest in moving out of their datacenters before the layoffs.

foobarbazetc 9y ago
Yeah. The exact same thing happened at Evernote. They hit a sales slump, new “corporate” management, cuts etc -> GCP. The Ops people there were pretty proud of their infrastructure, but they took it to a pretty extreme level. I think they were racking their own stuff and even running their own routers etc.
- lykr0n 9y ago
  Racking your own servers and running your own routers is extreme? I don't know about you, but I haven't set foot in the DC in two years after initial setup.
  - foobarbazetc 9y ago
    Yeah I meant to put it in quotes. :) Hard to find B2C type SaaS companies doing that though (with the exception of Pinboard, who’s crazy :) We actually haven’t had a hardware failure (apart from HDD that have all been replaced with SSD) in something like 7 years. We don’t even pay big bucks for Dell, HP etc. Just cheap SuperMicro boxes.
    - amigoingtodie 9y ago
      What does your hardware and software stack look like?

olingern 9y ago

There is so much more overhead with bare metal. There's an entire other side of building a web server that devs, thankfully, never have to see. OpenCompute [1] aims to standardize what that process looks like. To think that you can run bare metal, and do better than industry standards is naive IMO. In the end, if you're worrying more about anything than business logic, you're not really delivering value -- you're solving engineering problems. You can sacrifice that control and solving problems for greater abstractions, i.e. serverless, fargate, etc. Unless you have some extreme security requirements (which most don't) you probably don't have a good reason to run bare metal. [1] - http://www.opencompute.org/ http://www.opencompute.org/

geofft 9y ago
Cloud providers have to make money, though. You're doing more with bare metal, yes, but you're not doing much more (especially compared to learning how to use a cloud infrastructure effectively), and the cost may not be worthwhile. Generally, the cost of buying and running one server in a datacenter and staffing the ops team needed to keep the hardware running is cheaper than paying for a 24/7 server of comparable specs in the cloud. And if you're already running in a datacenter, it's extremely not obvious that the cost of moving your workload ever will be net profitable. Of course, if you don't need comparable specs to an entire physical server (I do a lot of personal stuff on a t2.micro!) or don't need the same capacity 24/7, the math is very different. The cloud is definitely cost-effective for elastic workloads.
- olingern 9y ago
  > Generally, the cost of buying and running one server in a datacenter and staffing the ops team needed to keep the hardware running is cheaper than paying for a 24/7 server of comparable specs in the cloud I would disagree here. You can run the metrics for your stack, but it's likely cheaper on AWS. https://awstcocalculator.com/ https://awstcocalculator.com/ If you don't need to compute something but twice a day, or just certain times -- you can pay pennies on the dollar for a serverless function. There's no need for an entire server for things like this.

azurezyq 9y ago

Operation cost is only one side. There are more things: * Well designed security / encryption may cost a lot. * Compliance may be a pain. * multi-DC and HA * Surge resource usage over holidays. * "Sleeping" resources at night or wasted.

tedmiston 9y ago
Support for auto-scaling services in a self-managed environment too
ehsankia 9y ago
There's also just all sorts of other extra features you may get for free. These are things that may not be worth it for you to spend precious dev time developing, but since cloud providers can make it once and offer it to thousands of customers, it makes much more sense. You can look at the full list of GCP features [0], and it's growing every month. [0] https://cloud.google.com/products/ https://cloud.google.com/products/
lethargic_meat 9y ago
A critical security issue in public cloud is that users often misunderstand what infrastructure security entails. I've been in public cloud pitches that sold their security measures as a golden standard that made my managers believe there's no need for a security team anymore. No need for protocols or proper architecture as x cloud has enough flashy certifications.

spencerhakim 9y ago

I would be shocked if Etsy/Spotify/whoever else wasn't cutting a deal for a significant discount, getting these name brands on GCP is great PR for the platform.

jasonlotito 9y ago
You don't even need to be a name brand to get discounts.
- spencerhakim 9y ago
  Of course, but I'm sure being a name brand helps them negotiate a larger discount.

Thaxll 9y ago

With bare metal you pay for servers you don't use. Cloud is more expensive on the compute side but you can really scale on your usage pattern, also security on the cloud is way better that on bare metal.

pm90 9y ago

As with everything else, the prices will reduce over time. But to be honest: you are not the market. If you have a functioning, reliable infrastructure that isn't too expensive or PITA to maintain, you have a good enough infra and should continue using that. If you're a startup who wants to minimize capex and quickly spin up a reliable, secure infra to host your app on to get it out of the door ASAP, you would go with cloud. If you're a non-technical company with a small IT department that wants to have an online presence and/or build internal online tools without having to build out DC's everywhere, you would go with cloud. I could come up with dozens and dozens more. Cloud isn't optimal for everyone, but it is VERY helpful for certain organizations.

alexasmyths 9y ago
"If you're a startup who wants to minimize capex and quickly spin up a reliable, secure infra to host your app on to get it out of the door ASAP, you would go with cloud." I agree, but I don't think that's the market either. I think the market is in massive big-corps, moving their IT infrastructure to the cloud. Like regular, non-tech companies. Small startup bills might be hundreds, thousands, possibly 100's of K but probably not millions. Big corp contracts may be millions - and stable.
- toomuchtodo 9y ago
  At megacorp scales, you're wasting money going to the cloud unless your load profile is highly variable. GCP/AWS profit margins are very high versus bare metal, and if you're megacorp scale you can afford ops and bare metal you can depreciate over several years. (and before you trot out Netflix, the vast majority of their bits are served out of OpenConnect appliances colocated at eyeball ISPs and peering points)
  - jermaustin1 9y ago
    I recently watched a Mega Corp fire the entire IT Ops department and hire a handful cloud consultants to move all of their current vSphere (thousands of VMs) infrastructure to Azure. The rationalization was that in the long run it was cheaper to maintain the status quo, and not have to worry about networking, firewall, and replication issues by hand. I'm pretty sure MS also gave them a large discount at that scale.

manigandham 9y ago

These calculations don't make sense unless you really consider TCO over any appreciable amount of time. It's easy to say you can buy a huge server for a few thousand but the "cloud" provides incredible agility, security, and basically removes operational overhead. When compared to buying, maintaining, and decommissioning servers and software along with the time and personnel costs, it becomes a much closer race. Include global scale, elastic capacity, managed services, and support policies, and it's closer still. Rarely are companies paying 3x IT spend just to run in the cloud, especially for large companies like Etsy.

noahl 9y ago

To answer your question, people get discounts just for negotiating, no PR required. I would guess the conversation went something like this: "___ is how much we pay to host Etsy on bare metal right now. Can you match that?" "Yes." I'm oversimplifying - there are a bunch of other considerations, like guarantees about future prices and capacity availability and whether having more flexibility makes it worthwhile for Etsy to pay more, but fundamentally Etsy looked at their current costs and asked Google to make an offer. (I would assume they actually had all three major players bid, and Google won.) Also to your point, the announcement specifically called out Google's AI and big data capabilities. Those are workloads that can involve a lot of bursting, so the cloud might be especially attractive. Imagine trying to implement BigQuery in bare metal - you can do it, but you have to have a lot of extra servers lying around just in case someone tries to run an analytics query. The same goes even more strongly for something like a model training workload. For bursty enough workloads, running in the cloud can be cheaper than bare metal even given the cloud provider's profit margin. Depending on how much model training Etsy wants to do, something like TPUs might have also given Google a fundamental edge in costs over bare metal or other providers. I don't know enough to know exactly what factors went into Etsy's decision, but I think these are reasonable ideas.

gesman 9y ago
I not sure about them asking AWS to bid as they'd see AMZN in a "conflict of interest" online ecommerce space.
- alpha_squared 9y ago
  As far as I know, which is very little generally, AWS doesn't bid. Prices are as advertised. Not sure if GCP is similar. Azure certainly does, though.
  - jimmywanger 9y ago
    I know for a fact that AWS that AWS negotiates. My buddy used to work at Netflix and he told me that they got a really good rate for AWS that was not available to the general public. They might not bid in the strictest sense of the word, but when the vendors are mostly 3 players (Google, Amazon, and Microsoft) you just ask each one for a quote. I don't see how functionally that is different from bidding. Btw, I know there are other hosting providers out there (Rackspace comes to mind). I can't think of any one of them that would be able to deal with Netflix scale without shitting the bed, missing features, or an almost unacceptable lead time as they build out infrastructure.

blaisio 9y ago

The hardware and bandwidth costs are only small parts of the full cost of running your own infrastructure. There are so many other benefits: - you don't have to support your own low level services like Pub/Sub, Cloud Storage, Spanner, etc. - you don't have to worry as much about things like HIPAA compliance - developers don't have to wait for you to allocate more hardware, greatly slowing down the software development process - it's much easier to test your applications, since you can spin up a full copy using rented infrastructure, and then throw it all away once your tests are complete - many other benefits that I'm forgetting And of course if you buy enough from them, you do get additional discounts. For many (most? I don't have data) companies, the actual cost of the infrastructure and bandwidth itself is tiny compared to all the other "hidden costs".

deleted 9y ago
[deleted]
ShakataGaNai 9y ago
So having experienced Ops in both cloud and non cloud companies. A few rebuttals: - Bandwidth is VERY expensive in AWS compared to what you can buy in Datacenters. A 1 GBPS connection is so "cheap" you can get it and a 42U rack at HE.net for $400/mo ( http://he.net/colocation.html http://he.net/colocation.html ). That same 1GBPS is roughly 300TB/month. Assuming it's all outbound - that's roughly $21k in AWS. So bandwidth is a small cost IF you're not moving large amounts of data out of AWS. If you stream media - it can be a HUGE portion of your bill. - HIPAA Compliance - That and other compliance aspects assumes that you need those compliance frameworks. In enterprise world that might be the case (SOC2 and ISO27k1 are very common requirements for Enterprise SaaS clients). However in the B2C? Users don't care. - Waiting for hardware allocation. Once you get to a certain size you have hardware spare, typically. Developers can use that for their tests. You can also have your datacenter/operations group work with your developers during project planning so the code and HW can be ready at the same time. That being said, I don't disagree there are advantages to the cloud. I'm AWS certified, I drink the coolaid. But saying that there are "so many other benefits" is ignoring the fact that there are "so many benefits" to on prem too. There are pro's and cons to both sides.

NelsonMinar 9y ago

Related: "Etsy slashes almost a quarter of its staff in attempt to refocus" https://www.engadget.com/2017/06/21/etsy-layoffs-focus-on-core-marketplace/ https://www.engadget.com/2017/06/21/etsy-layoffs-focus-on-co...

dboreham 9y ago

>I’m really curious. I think you can imagine a scenario under which the end goal to have this headline was someone’s primary goal for the past couple of years.

7ewis 9y ago

I work for a company similar to GrubHub outside the US. 'The Cloud' works great for us. Think, how many people order food at 9AM on a Monday? Therefore, we only have a very small amount of servers running. However, 7PM on a Friday... thousands. Yes, it's still mega expensive. But allows us to scale when we need it and not have to worry about the hardware, cost, or management of it.

apendleton 9y ago

Two big savings opportunities: * elastic demand, either in the sense that your global traffic has peaks and troughs, or your global traffic is steady but its global distribution changes such that the balance of where you're deployed would ideally change over time; in either case you can benefit enormously from automatic scaling * workloads that can take advantage of the spot market (or whatever GCP's equivalent is)

mathattack 9y ago

Lifetime costs for much of this technology is more people costs. Also, much of the benefit comes from what's on top of the hosting. (Not having to worry about old versions of software, containers, better security, no issues if the one person who knows what's going on quits, etc)

seanwilson 9y ago

> I love GCP but honestly have no idea how these big internet properties can make the $ case for “the cloud”, unless they’re also planning to fire 90% of the Ops/SysAdmin team (ala Evernote). What kind of price differences are you thinking about? How much hardware and how many people are employed to take care of this? Does the tripling of cost take salaries into account? Having to hire a team that will maintain hardware, deal with scalability and give guaranteed uptime isn't cheap and requires a lot of management. I'm not saying it never makes sense, but offloading all that work on to someone like AWS and Google who have teams doing this for you reliably can be well worth the cost.

cobookman 9y ago

We should talk. Email me at bookman@google.com, and I'll be sure to get you a competitive pricing estimate. (Disclamer: I'm a Cloud Solutions Engineer at Google Cloud)