9 ms·

How much is Spotify paying Google Cloud?

254 points by dmytton 11y ago

stephenitis 11y ago
As far as I've heard from Netflix's end the bottom line cost was not as big a factor in their infrastructure decisions as their content contracts were a much larger % of their budget. I suspect Google wants to highlight more the fact that it works and works well rather than highlight cost competitiveness at the moment. thanks for the post @dmytton. not sure if it's accurate It'd be lovely for Spotify or Google to chime in.
- simonebrunozzi 11y ago
  My guess is they won't. I suspect that Google heavily subsidized the deal. One way to get market traction is to get market awareness, and Spotify on GCP achieves that. Spotify would have been dumb not to get a huge discount in exchange for its logo.
- dropdatabase 11y ago
  However, Spotify still stores its music files on AWS and uses CloudFront for content distribution. They're only gonna use Google Cloud for analyzing customer data. They won't ditch AWS... http://arstechnica.co.uk/information-technology/2016/02/spotify-spurns-aws-in-cloudy-tango-with-google/ http://arstechnica.co.uk/information-technology/2016/02/spot...
neom 11y ago
I recall David Byttow mentioning that secret cost about this much to run on GCP so I'm not surprised at all.
- joeblau 11y ago
  Yeah, Secret was spending a ton of money on GAE. > For example, in one month, our server costs were well over $4M on Google App Engine after spiking 5M users in the course of two weeks. We scrambled and optimized, but the cost was real[1]. [1] - https://www.producthunt.com/live/david-byttow https://www.producthunt.com/live/david-byttow
yeukhon 11y ago
> Of course, for such a high profile customer and for the priviledge of a case study, we can assume Spotify has some heavy discounts. I am pretty sure it is also not unusual for average customers on AWS to get some discount, with some reserved instance committment). One such example is probably Redshift which is extremely expensive. Would be nice to get an estimate quote for AWS but just for SQS I don't expect to be too expensive.
i0exception 11y ago
You're making a big assumption here - that they don't batch events before publishing to Google's Pub/Sub system. The blog post says "We also added queuing and batching in the client. (This wasn’t first time that we needed to roll our sleeves up and reimplement a Google Cloud API client: in another project we implemented a high performance client for the Datastore API.)"
- dmytton 11y ago
  Batching doesn't make any difference to the pricing: > Batched calls are accounted for as multiple operations. https://cloud.google.com/pubsub/pricing https://cloud.google.com/pubsub/pricing
  - CaveTech 11y ago
    Batching could mean they're aggregating calls prior to handing then off to google as a single request. If you have 64k minimum block size (guessing) you can stuff in many small requests before sending them off.
    - dmytton 11y ago
      Right but that only affects networking costs, because the batch is split out into individual publish API calls on Google's side, and that's what you're billed for.
      - deleted 11y ago
        [deleted]
  - i0exception 11y ago
    Yes, but if I'm reading their blog post correctly, their individual messages are much smaller than 64K "Pub/Sub passed the test with flying colours. We published 2M messages without any service degradation and received almost no server errors from the Pub/Sub backend. Enabling batching and compression on the Event Service machines resulted in ~1Gbps of network traffic towards Pub/Sub."
gberger 11y ago
$290k per month is $3.5M per year. A software engineer at Google earns, on average, $127k per year [1]. We multiply that by 1.4 [2] to obtain a cost per employee of $177k. This means that according to this estimation, Spotify pays for 20 Google software engineers. [1] https://www.glassdoor.com/Salary/Google-Salaries-E9079.htm https://www.glassdoor.com/Salary/Google-Salaries-E9079.htm [2] http://web.mit.edu/e-club/hadzima/how-much-does-an-employee-cost.html http://web.mit.edu/e-club/hadzima/how-much-does-an-employee-...
- npalli 11y ago
  Note, google cloud requires hardware.
- jpatokal 11y ago
  Remember that at Google, like in most other big companies in the Valley, baseline salary is only one part of total compensation: stock grants and bonuses are also quite significant. On the other hand, this blog only estimates the cost for Pub/Sub, when Spotify is using a whole slew of other Google Cloud services as well.
  - dmytton 11y ago
    It'll be interesting to see what other numbers Spotify release in subsequent blog posts for the other services they're using. Hopefully some cost calculations can be run for those, too.
- accountatwork 11y ago
  > A software engineer at Google earns, on average, $127k per year [1]. We multiply that by 1.4 [2] to obtain a cost per employee of $177k. That ignores RSUs, signing bonus, and other bonuses, which make up the majority of the compensation package for senior engineers. Even at entry level, total comp (ignoring insurance, cost of office space, free food, and other things that aren't directly paid to the employee) is over $127k/yr. A baseline new grad offer last fall included $180k of stock, or $45k/yr, on top of a six figure salary plus a $50k signing bonus. This probably goes without saying, but new grad offers can go significantly higher if the new grad has strong competing offers, and senior offers start out even higher and have a much larger range.
stephenitis 11y ago
Is there a post talking about how much Netflix pays Amazon?
bkovacev 11y ago
The author also failed to notice "peaks" from the original spotify blog post about events per second. So 700k requests is not average, it's peak. They don't have that every second.
- stephenitis 11y ago
  nice point. If you want to further refine you could probably estimate a large chunk of early morning US/EU time where traffic will be very light. maybe reduce the estimate by 1/3 or 1/4th to account for this.
- dmytton 11y ago
  This is a good point, but it could be somewhat mitigated by their expected growth of up to 2M per second. To quote the full line from their blog post: > Currently our production load peaks at around 700K events per second. To account for the future growth and possible disaster recovery scenarios, we settled on a test load of 2M events per second. In the absence of any other numbers, using the 700k number seems reasonable otherwise you would start to get into modelling without any real data to back up your assumptions. This at least gives you a maximum they would be paying under current production conditions.
  - bkovacev 11y ago
    That's actually a reasonable response. From my first read the numbers seemed a bit inflated, so I dug into their blog post out of curiosity (I develop on GAE), so the comment was not ill intended. Now that you have pointed out they do seem a good estimate! Thanks for the great articles David, keep them coming! :)
  - mkagenius 11y ago
    How about a gaussian distribution which peaks at 700k? Assuming max as the norm is not too right. (just to make the number look high)
    - EricBurnett 11y ago
h1srf 11y ago
The post is a cool example but I'd be shocked if Spotify doesn't have a contract for up to X events/mo and overages at some reduced rate. No way they're paying per request. The only place I've never seen negotiate rates is Bloomberg.
mangeletti 11y ago
As a side note, $300k/mo is a pretty small amount of money, compared to the +$100MM/mo revenue Spotify brings in.
capkutay 11y ago
There must have been a steep discount given that Spotify has been an enormous reference customer and blogging about how much they love it.
prdonahue 11y ago
I'd be willing to bet they're paying almost $0 in the first year. Google REALLY wants a big name customer, so you have to think they gave this deal away for negative margin.
kordless 11y ago
A good guess using non-technical means would be $15-18M a year. I based this on the assumption that a customer doing less than $1M a month really isn't worth mentioning in the news or noteworthy. Also, Spotify did $1.2B last year. P.S. Howdy David!
Hengjie 11y ago
The bigger question is if Google Cloud Platform is cheaper than AWS?
Pyxl101 11y ago
> Thankfully, Google’s pricing is very transparent and simple to work with, so we can use Spotify’s blog posts to calculate some list pricing for what Spotify might be paying Google. The biggest way in which all such assumptions are likely to be wrong is that they overlook the likelihood that Spotify has been offered a potentially substantial discount in exchange for attempting to win their business, and to be an early adopter. Not that there's anything wrong with that, but it's something to take into account while evaluating what you think Spotify might be paying. Many cloud service providers offer a discount to, for example, qualified startups participating in certain accelerators. In the same fashion, it would not be surprising for a provider to entice a major potential user and hopeful net promoter with a significant price reduction as well as other perks, such as more access to senior people for operations support, as well as influence on the roadmap. So while I think this blog post could plausibly estimate what a typical customer with Spotify's usage pattern might pay, assuming the estimates are correct (I haven't checked the details), one cannot draw conclusions about what they're likely to actually be paying. [Edit: I see now that the article recognizes this at the bottom. At the time I made my comment, it seemed to have been overlooked in the discussion here, where only one comment acknowledged it and the rest seemed to discuss the numbers as factual and face-value: https://news.ycombinator.com/item?id=11242166 https://news.ycombinator.com/item?id=11242166 ]
- joshAg 11y ago
  The author covered this: >>> Is Spotify really paying $290,518/month? >>> Of course, for such a high profile customer and for the priviledge of a case study, we can assume Spotify has some heavy discounts. Cost is important but using public cloud and especially managed services like pub/sub, the biggest benefit is in ease of deployment and not having to manage/scale everything.
  - deleted 11y ago
    [deleted]
- jsmeaton 11y ago
  This was actually addressed in the article towards the end, and the part you quoted does say list price. > Is Spotify really paying $290,518/month? Of course, for such a high profile customer and for the priviledge of a case study, we can assume Spotify has some heavy discounts.
jamesblonde 11y ago
This article misses the biggest cost Spotify have at GCE - analytics. Spotify has migrated a 90 PB Hadoop cluster running on 2000 servers to GCE. That cluster was running Hadoop jobs (mostly Hive) 24x7. The YARN job queue was always full. They now have to pay for the storage and processing costs for that. How much will they pay? I don't know, but I know from their history they are very good at getting discounts on infrastructural software (e.g., their Hadoop software was licensed from Hortonworks and they got an incredible price, because Hortonworks needed a reference customer - sound familiar?)
- dmytton 11y ago
  I could only write about pub/sub in this article because that's the only service we have a full enough set of numbers for. I hope Spotify will be releasing more stats in their future blog posts about the other GCP services they'll be using.
  - jamesblonde 11y ago
    I have the stats from late January (at Stockholm Hadoop User Group). It's 90PB of data, running jobs with 95 TB of RAM, 40K jobs/day (approximately 12,000 CPUs with probably 60-90% utilization).
  - jamesblonde 11y ago
    I should add they also have ~100 servers streaming the music to clients for songs that aren't handled by the CDN.
- gedrap 11y ago
  I am wondering, how does one migrate huge amounts of data like this?
  - orthecreedence 11y ago
    Sometimes a truck full of hard drives is the fastest/cheapest way. Not saying that's what they did, but it has been done before.
lateguy 11y ago
Does Google cloud work in china?
sre_ops 11y ago
It is always amusing "smart people" in the room not to know about CAF.

254 points by dmytton 11y ago

As far as I've heard from Netflix's end the bottom line cost was not as big a factor in their infrastructure decisions as their content contracts were a much larger % of their budget. I suspect Google wants to highlight more the fact that it works and works well rather than highlight cost competitiveness at the moment. thanks for the post @dmytton. not sure if it's accurate It'd be lovely for Spotify or Google to chime in.

simonebrunozzi 11y ago
My guess is they won't. I suspect that Google heavily subsidized the deal. One way to get market traction is to get market awareness, and Spotify on GCP achieves that. Spotify would have been dumb not to get a huge discount in exchange for its logo.
dropdatabase 11y ago
However, Spotify still stores its music files on AWS and uses CloudFront for content distribution. They're only gonna use Google Cloud for analyzing customer data. They won't ditch AWS... http://arstechnica.co.uk/information-technology/2016/02/spotify-spurns-aws-in-cloudy-tango-with-google/ http://arstechnica.co.uk/information-technology/2016/02/spot...

neom 11y ago

I recall David Byttow mentioning that secret cost about this much to run on GCP so I'm not surprised at all.

joeblau 11y ago
Yeah, Secret was spending a ton of money on GAE. > For example, in one month, our server costs were well over $4M on Google App Engine after spiking 5M users in the course of two weeks. We scrambled and optimized, but the cost was real[1]. [1] - https://www.producthunt.com/live/david-byttow https://www.producthunt.com/live/david-byttow

yeukhon 11y ago

> Of course, for such a high profile customer and for the priviledge of a case study, we can assume Spotify has some heavy discounts. I am pretty sure it is also not unusual for average customers on AWS to get some discount, with some reserved instance committment). One such example is probably Redshift which is extremely expensive. Would be nice to get an estimate quote for AWS but just for SQS I don't expect to be too expensive.

i0exception 11y ago

You're making a big assumption here - that they don't batch events before publishing to Google's Pub/Sub system. The blog post says "We also added queuing and batching in the client. (This wasn’t first time that we needed to roll our sleeves up and reimplement a Google Cloud API client: in another project we implemented a high performance client for the Datastore API.)"

dmytton 11y ago
Batching doesn't make any difference to the pricing: > Batched calls are accounted for as multiple operations. https://cloud.google.com/pubsub/pricing https://cloud.google.com/pubsub/pricing
- CaveTech 11y ago
  Batching could mean they're aggregating calls prior to handing then off to google as a single request. If you have 64k minimum block size (guessing) you can stuff in many small requests before sending them off.
  - dmytton 11y ago
    Right but that only affects networking costs, because the batch is split out into individual publish API calls on Google's side, and that's what you're billed for.
    - deleted 11y ago
      [deleted]
- i0exception 11y ago
  Yes, but if I'm reading their blog post correctly, their individual messages are much smaller than 64K "Pub/Sub passed the test with flying colours. We published 2M messages without any service degradation and received almost no server errors from the Pub/Sub backend. Enabling batching and compression on the Event Service machines resulted in ~1Gbps of network traffic towards Pub/Sub."

gberger 11y ago

$290k per month is $3.5M per year. A software engineer at Google earns, on average, $127k per year [1]. We multiply that by 1.4 [2] to obtain a cost per employee of $177k. This means that according to this estimation, Spotify pays for 20 Google software engineers. [1] https://www.glassdoor.com/Salary/Google-Salaries-E9079.htm https://www.glassdoor.com/Salary/Google-Salaries-E9079.htm [2] http://web.mit.edu/e-club/hadzima/how-much-does-an-employee-cost.html http://web.mit.edu/e-club/hadzima/how-much-does-an-employee-...

npalli 11y ago
Note, google cloud requires hardware.
jpatokal 11y ago
Remember that at Google, like in most other big companies in the Valley, baseline salary is only one part of total compensation: stock grants and bonuses are also quite significant. On the other hand, this blog only estimates the cost for Pub/Sub, when Spotify is using a whole slew of other Google Cloud services as well.
- dmytton 11y ago
  It'll be interesting to see what other numbers Spotify release in subsequent blog posts for the other services they're using. Hopefully some cost calculations can be run for those, too.
accountatwork 11y ago
> A software engineer at Google earns, on average, $127k per year [1]. We multiply that by 1.4 [2] to obtain a cost per employee of $177k. That ignores RSUs, signing bonus, and other bonuses, which make up the majority of the compensation package for senior engineers. Even at entry level, total comp (ignoring insurance, cost of office space, free food, and other things that aren't directly paid to the employee) is over $127k/yr. A baseline new grad offer last fall included $180k of stock, or $45k/yr, on top of a six figure salary plus a $50k signing bonus. This probably goes without saying, but new grad offers can go significantly higher if the new grad has strong competing offers, and senior offers start out even higher and have a much larger range.

stephenitis 11y ago

Is there a post talking about how much Netflix pays Amazon?

bkovacev 11y ago

The author also failed to notice "peaks" from the original spotify blog post about events per second. So 700k requests is not average, it's peak. They don't have that every second.

stephenitis 11y ago
nice point. If you want to further refine you could probably estimate a large chunk of early morning US/EU time where traffic will be very light. maybe reduce the estimate by 1/3 or 1/4th to account for this.
dmytton 11y ago
This is a good point, but it could be somewhat mitigated by their expected growth of up to 2M per second. To quote the full line from their blog post: > Currently our production load peaks at around 700K events per second. To account for the future growth and possible disaster recovery scenarios, we settled on a test load of 2M events per second. In the absence of any other numbers, using the 700k number seems reasonable otherwise you would start to get into modelling without any real data to back up your assumptions. This at least gives you a maximum they would be paying under current production conditions.
- bkovacev 11y ago
  That's actually a reasonable response. From my first read the numbers seemed a bit inflated, so I dug into their blog post out of curiosity (I develop on GAE), so the comment was not ill intended. Now that you have pointed out they do seem a good estimate! Thanks for the great articles David, keep them coming! :)
- mkagenius 11y ago
  How about a gaussian distribution which peaks at 700k? Assuming max as the norm is not too right. (just to make the number look high)
  - EricBurnett 11y ago

h1srf 11y ago

The post is a cool example but I'd be shocked if Spotify doesn't have a contract for up to X events/mo and overages at some reduced rate. No way they're paying per request. The only place I've never seen negotiate rates is Bloomberg.

mangeletti 11y ago

As a side note, $300k/mo is a pretty small amount of money, compared to the +$100MM/mo revenue Spotify brings in.

capkutay 11y ago

There must have been a steep discount given that Spotify has been an enormous reference customer and blogging about how much they love it.

prdonahue 11y ago

I'd be willing to bet they're paying almost $0 in the first year. Google REALLY wants a big name customer, so you have to think they gave this deal away for negative margin.

kordless 11y ago

A good guess using non-technical means would be $15-18M a year. I based this on the assumption that a customer doing less than $1M a month really isn't worth mentioning in the news or noteworthy. Also, Spotify did $1.2B last year. P.S. Howdy David!

Hengjie 11y ago

The bigger question is if Google Cloud Platform is cheaper than AWS?

Pyxl101 11y ago

> Thankfully, Google’s pricing is very transparent and simple to work with, so we can use Spotify’s blog posts to calculate some list pricing for what Spotify might be paying Google. The biggest way in which all such assumptions are likely to be wrong is that they overlook the likelihood that Spotify has been offered a potentially substantial discount in exchange for attempting to win their business, and to be an early adopter. Not that there's anything wrong with that, but it's something to take into account while evaluating what you think Spotify might be paying. Many cloud service providers offer a discount to, for example, qualified startups participating in certain accelerators. In the same fashion, it would not be surprising for a provider to entice a major potential user and hopeful net promoter with a significant price reduction as well as other perks, such as more access to senior people for operations support, as well as influence on the roadmap. So while I think this blog post could plausibly estimate what a typical customer with Spotify's usage pattern might pay, assuming the estimates are correct (I haven't checked the details), one cannot draw conclusions about what they're likely to actually be paying. [Edit: I see now that the article recognizes this at the bottom. At the time I made my comment, it seemed to have been overlooked in the discussion here, where only one comment acknowledged it and the rest seemed to discuss the numbers as factual and face-value: https://news.ycombinator.com/item?id=11242166 https://news.ycombinator.com/item?id=11242166 ]

joshAg 11y ago
The author covered this: >>> Is Spotify really paying $290,518/month? >>> Of course, for such a high profile customer and for the priviledge of a case study, we can assume Spotify has some heavy discounts. Cost is important but using public cloud and especially managed services like pub/sub, the biggest benefit is in ease of deployment and not having to manage/scale everything.
- deleted 11y ago
  [deleted]
jsmeaton 11y ago
This was actually addressed in the article towards the end, and the part you quoted does say list price. > Is Spotify really paying $290,518/month? Of course, for such a high profile customer and for the priviledge of a case study, we can assume Spotify has some heavy discounts.

jamesblonde 11y ago

This article misses the biggest cost Spotify have at GCE - analytics. Spotify has migrated a 90 PB Hadoop cluster running on 2000 servers to GCE. That cluster was running Hadoop jobs (mostly Hive) 24x7. The YARN job queue was always full. They now have to pay for the storage and processing costs for that. How much will they pay? I don't know, but I know from their history they are very good at getting discounts on infrastructural software (e.g., their Hadoop software was licensed from Hortonworks and they got an incredible price, because Hortonworks needed a reference customer - sound familiar?)

dmytton 11y ago
I could only write about pub/sub in this article because that's the only service we have a full enough set of numbers for. I hope Spotify will be releasing more stats in their future blog posts about the other GCP services they'll be using.
- jamesblonde 11y ago
  I have the stats from late January (at Stockholm Hadoop User Group). It's 90PB of data, running jobs with 95 TB of RAM, 40K jobs/day (approximately 12,000 CPUs with probably 60-90% utilization).
- jamesblonde 11y ago
  I should add they also have ~100 servers streaming the music to clients for songs that aren't handled by the CDN.
gedrap 11y ago
I am wondering, how does one migrate huge amounts of data like this?
- orthecreedence 11y ago
  Sometimes a truck full of hard drives is the fastest/cheapest way. Not saying that's what they did, but it has been done before.

lateguy 11y ago

Does Google cloud work in china?

sre_ops 11y ago

It is always amusing "smart people" in the room not to know about CAF.