4 ms·
This article misses the biggest cost Spotify have at GCE - analytics. Spotify has migrated a 90 PB Hadoop cluster running on 2000 servers to GCE. That cluster w
by jamesblonde 11y ago
This article misses the biggest cost Spotify have at GCE - analytics. Spotify has migrated a 90 PB Hadoop cluster running on 2000 servers to GCE. That cluster was running Hadoop jobs (mostly Hive) 24x7. The YARN job queue was always full. They now have to pay for the storage and processing costs for that. How much will they pay? I don't know, but I know from their history they are very good at getting discounts on infrastructural software (e.g., their Hadoop software was licensed from Hortonworks and they got an incredible price, because Hortonworks needed a reference customer - sound familiar?)
- dmytton 11y agoI could only write about pub/sub in this article because that's the only service we have a full enough set of numbers for. I hope Spotify will be releasing more stats in their future blog posts about the other GCP services they'll be using.
- jamesblonde 11y agoI have the stats from late January (at Stockholm Hadoop User Group). It's 90PB of data, running jobs with 95 TB of RAM, 40K jobs/day (approximately 12,000 CPUs with probably 60-90% utilization).
- jamesblonde 11y agoI should add they also have ~100 servers streaming the music to clients for songs that aren't handled by the CDN.
- gedrap 11y agoI am wondering, how does one migrate huge amounts of data like this?
- orthecreedence 11y agoSometimes a truck full of hard drives is the fastest/cheapest way. Not saying that's what they did, but it has been done before.
- jamesblonde 11y agoYes, that's how the England 100,000 genomes project is transferring 10s of petabytes from San Diego to England. I assume it was the same here.
- JupiterMoon 11y agoIf any of the data to be migrated is music then much of that is probably somewhere inside google's datacenters already..