25 ms·

Spotify moves its back end to Google Cloud

715 points by dmichel 11y ago

dkubb 11y ago
(Serious question) Isn't it risky to rely on any Google product as a foundation for your business? They've shut down more popular services that people depend on than Google Cloud. They tend to enter a market, undercut all the competitors, and then once they are dominant in the niche they terminate or change the service in a way that makes it not fulfill the original promise.
- mastazi 11y ago
  Well Amazon killed the Fire Phone though ;-) https://fortune.com/2015/09/09/amazon-killing-fire-phone/ https://fortune.com/2015/09/09/amazon-killing-fire-phone/ Jokes aside, probably you are thinking about free products like Google Reader, because this: > once they are dominant in the niche they terminate or change the service would not make any sense if the product in question makes $$$
obulpathi 11y ago
> Google has long been a thought-leader in this space, and this shows in the sophistication and quality of its data offerings. From traditional batch processing with Dataproc, to rock-solid event delivery with Pub/Sub to the nearly magical abilities of BigQuery, building on Google’s data infrastructure provides us with a significant advantage where it matters the most. Big Data is the core strength of Google Cloud. Good to see this move by Spotify! > What really tipped the scales towards Google for us, however, has been our experience with Google’s data platform and tools. Good infrastructure isn’t just about keeping things up and running, it’s about making all of our teams more efficient and more effective, and Google’s data stack does that for us in spades. What I really really liked about Google Cloud is the ease of use. Spin up a VM, start Cloud shell, SSH into your instance, install a bunch of software and you will know what I mean. It's "Quality" Cloud.
sigmar 11y ago
>"...feature testing and more intelligent user-facing features," the Google staffer wrote. I have to wonder if Spotify is doing this so that they can (eventually) use Google's deep learning technology in improving their recommendation engine. They purchased Echo Nest in 2014 to achieve this.
atrudeau 11y ago
This reads like they've been acquired by Google and are integrating. Now Google just has to write the check and own music streaming.
- simonswords82 11y ago
  Hmmm...interesting point but Google already has an awful lot of music infrastructure in the form of Google Music and YouTube does it not? Would it really make sense for them to purchase Spotify too?
  - txru 11y ago
    Buying brand loyalty, perhaps?
  - toomuchtodo 11y ago
    They'd be buying the users.
    - simonswords82 11y ago
      That's about the only real value. But then what? Do they run the two entities separately? Or merge Google's offering with Spotify and risk pissing customers off who are loyal to the Spotify brand and app?
      - toomuchtodo 11y ago
        No idea whatsoever. Maybe merge Spotify into Google Fiber and include it as part of a content offering?
  - mcintyre1994 11y ago
    I use Google's offering and it's probably pretty solid with YouTube Red, but I'm guessing at this point they're third behind Spotify and Apple?
  - zymhan 11y ago
dang 11y ago
Also on this: http://www.wired.co.uk/news/archive/2016-02/23/spotify-move-to-google-cloud http://www.wired.co.uk/news/archive/2016-02/23/spotify-move-... and http://googlecloudplatform.blogspot.com/2016/02/Spotify-chooses-Google-Cloud-Platform-to-power-data-infrastructure.html http://googlecloudplatform.blogspot.com/2016/02/Spotify-choo...
shockzzz 11y ago
Smells like a puff-piece
- skj 11y ago
  A puff piece would be if some tech reporter wrote about the wonders of GCP. This is the Spotify team, and they've clearly put their money where their mouth is by actually making the move.
  - wcummings 11y ago
    Marketing blog spam is the new puff piece.
    - vgt 11y ago
      Would you prefer testaments like these from Spotify engineers? https://twitter.com/plamere/status/702168809445134336 https://twitter.com/plamere/status/702168809445134336 https://twitter.com/plamere/status/702169733454495745 https://twitter.com/plamere/status/702169733454495745 https://twitter.com/mpjme/status/702167231623516160 https://twitter.com/mpjme/status/702167231623516160
      - shockzzz 11y ago
        Kind of. We all know that companies cut deals with providers in exchange for blog posts, tweets, and other sorts of promotion. I'm always wary of posts like these, especially when they lack detail or awareness of competing products.
        vgt 11y ago
        I think if Spotify dressed down Google Cloud Platform competition, no matter the merit, there would be many more complaints (ex. see comments on this technical post https://news.ycombinator.com/item?id=11029032 https://news.ycombinator.com/item?id=11029032). Spotify did say that they really enjoy specific aspects of Google Cloud Platform, through their own engineering blog. Spotify engineers have also been independently vocal on Twitter. Netflix has been very similarly vocal of their support of AWS. I'd love to hear your thoughts on a better mindshare model, perhaps we can all learn to do things a bit better.
shockzzz 11y ago
Smells like a puff-piece
ed_blackburn 11y ago
I always used to think Facebook would make a move for Spotify..
- ed_blackburn 11y ago
  -1 :( Zuckerberg used to wax lyrically about Spotify and I always assumed Facebook would one day like to move to the subscription streaming market. A barrier to entry for Spotify to stream movies I always suspected would be cash. Moving to google cloud won't stop a Spotify sale but it may discourage some?
- ed_blackburn 11y ago
  -1 :( Zuckerberg used to wax lyrically about Spotify and I always assumed Facebook would one day like to move to the subscription streaming market. A barrier to entry for Spotify to stream movies I always suspected would be cash. Moving to google cloud won't stop a Spotify sale but it may discourage some?
waffle_ss 11y ago
I tried Google Container Engine (GKE) and really liked it - it's the best cloud solution for deploying Docker to production in my opinion, mainly due to its use of Kubernetes. Unfortunately in my Web apps I make heavy use of Postgres-specific features, and since Cloud SQL only supports MySQL, Google Cloud is a total non-starter for me. So for now I'm on AWS, using Postgres on RDS and deploying containers with ECS. ECS is a lot simpler than Kubernetes, but since my apps are pretty simple (a half dozen task definitions), it's not a big deal. I really hope Google adds Postgres to Cloud SQL at some point.
- stevenleeg 11y ago
  Is there any reason you can't mix and match Google Container Engine and Amazon RDS?
  - dabeeeenster 11y ago
    Latency
    - MBCook 11y ago
      And bandwidth charges
    - stevenleeg 11y ago
      Got it, thanks! What about running a postgres container on GKE?
      - andrewstuart2 11y ago
        There's nothing stopping anybody for sure, but it's definitely not going to be as hands-off for getting to high availability, backups, scaling well, etc would be with managed SQL as a service.
      - crb 11y ago
        Nice writeup on that here: https://blog.oestrich.org/2015/08/running-postgres-inside-kubernetes/ https://blog.oestrich.org/2015/08/running-postgres-inside-ku...
dublinben 11y ago
When did Spotify abandon their original peer-to-peer architecture?[0] [0]http://www.csc.kth.se/~gkreitz/spotify-p2p10/spotify-p2p10.pdf http://www.csc.kth.se/~gkreitz/spotify-p2p10/spotify-p2p10.p...
- philo23 11y ago
  Quite a while ago. Not sure on the exact time, but judging from this article, I'd guess sometime in 2014 https://torrentfreak.com/spotify-starts-shutting-down-its-massive-p2p-network-140416/ https://torrentfreak.com/spotify-starts-shutting-down-its-ma...
  - hengheng 11y ago
    Seems a great growth hack. Too bad it's not more.
  - api 11y ago
    I'd be curious if anyone knows the details about why: complexity, customer complaints, reliability?
    - natarius4k 11y ago
      Too much complexity for a world that's moving to mobile clients that can't be active peers of the network.
      - api 11y ago
        Seems to be a pattern: wimpy mobile endpoint devices drive everything to the cloud. If so this is perhaps a dated phenomenon. Moore's Law is still a thing and today's generation of mobile devices are getting fatter and fatter.
        LeoPanthera 11y ago
        Moore's Law has not been improving batteries, which is the real problem. Mobile P2P nodes need to be awake most/all the time, which is a real battery killer. Modern devices have improved battery life by being asleep almost all the time.
flurpitude 11y ago
Now how about Spotify fixing their front end so that Premium customers on Android can just hit "play" to play the album they want to play, in the right order? And how about an app in the Windows Store so that you can use it on a touchscreen? And how about some improved metadata that work for classical music? And a way to report corrupt or mistagged tracks and get them fixed? /rant
- voltagex_ 11y ago
  I'm on mobile so can't directly link you but there's a support form for getting songs fixed.
  - flurpitude 11y ago
    I have never found a form, but I have found forums where people complain about persistent issues and nothing gets done. They seem to go out of their way to make it difficult for users to report issues. In addition they seem to have contempt for user experience, or they're incompetent at it, or they are under weird nonsensical licensing pressures that require them to make things difficult for their users. Their front ends need an overhaul and they could easily introduce a "flag this track as having a problem" feature.
icaromedeiros 11y ago
I was wondering about Spotify's contributions to Hadoop stack such as Snakebite and Luigi. What will happen to these projects if they're moving to BigQuery?
- thesandlord 11y ago
  Google Cloud has a managed Hadoop service called Dataproc: https://cloud.google.com/dataproc/ https://cloud.google.com/dataproc/
- crb 11y ago
  Spotify are not moving to BigQuery alone. They are also using Dataproc, Google's managed Hadoop service (which co-incidentally went GA yesterday), which lets them get value out of all the orchestration tools they already use, like Luigi. (They are also adopting Cloud Dataflow, Cloud Pub/Sub and other parts of our Big Data stack.) Luigi has had BigQuery support for some time: https://github.com/spotify/luigi/pull/1002 https://github.com/spotify/luigi/pull/1002 Spotify will be talking more about the specifics of their data pipelines at GCP Next in SF in March, - I would expect to see a lot more on the new Google Cloud Big Data blog at https://cloud.google.com/blog/big-data/ https://cloud.google.com/blog/big-data/ too. [Disclosure, GCP guy, etc]
- dundun 11y ago
  Nothing in this move should affect contributions of Snakebite and Luigi. If anything, it will just make them easier to use with cloud environments in addition to bare metal. With over 20K jobs/day, Hadoop will be a part of Spotify's data processing stack for quite a while. BigQuery is just a (awesome) piece of the full puzzle. [spotifier]
lukasm 11y ago
Isn’t it risky to move to competition's infrastructure?
- beambot 11y ago
  IIRC, Netflix predominantly hosted on AWS. It's definitely a consideration, but focusing on core competency isn't bad either.
- Splines 11y ago
  Not really - many people rely on Google Cloud and if Google made it harder for Spotify to compete (via, say, raising prices for Google Cloud for them), they'd have to be pretty specific to Spotify to not cause other customers to move to AWS or Azure. Spotify has much bigger risks in their licensing deals.
  - lukasm 11y ago
    cost it not a problem and google wouldn't do it. I'm talking about stealing recommendation algorithms and models. Have a peek new features before release etc.
    - lukasm 11y ago
      Seems googlers are down voting massively.
    - zenlikethat 11y ago
      Recommendation algorithms and models don't do you any good if you don't have users.
      - lukasm 11y ago
        Google Music has a lot of users?
latchkey 11y ago
What Google Cloud needs is a better sales story. AWS consistently beats out Google in this respect. This story is welcomed and there should be more success stories published like this. My business is a heavy GC user (AppEngine, Datastore, CloudStorage, BQ, ComputeEngine, ManagedVM) and I couldn't imagine achieving what we have achieved in such a short amount of time on any other platform. We (myself and another guy) started at zero and shipped our product in 3 months on what is effectively an infinitely auto-scalable platform where we don't have to do any devops or carry a pager. I'd much rather work on features than chores. =) 4 months later and we've had profitable growth and zero downtime (knock on wood) and we're hiring. Thanks Google!
- tn13 11y ago
  One of the reasons I avoid Google is because they have taken lot of pain to make their customer care totally suck. I mean even IRS and Comcast support does better compared to Google. I published a very critical release to my Android app few weeks back and realized that it was not getting published at all. It took me a month to get the problem resolved. I don't want to buy a cloud service where the support might be nothing more than bunch of "Help yourself" pages.
  - latchkey 11y ago
    They have a paid support service that is actually pretty good. It sounds like you're avoiding the generic Google in the same way you'd not buy something from Amazon vs. using AWS.
    - tn13 11y ago
      As I said Google's reputation is what made me not even bother to try. We all know that AWS and AZURE both are state of art and very competitively priced. If Google wants to beat them they have to win the perception war too. (OR reduce prices substantially).
      - boulos 11y ago
misiti3780 11y ago
I had no clue spotify operated their own data centers. Does anyone have a link to that compares google cloud vs aws on a per price + per service basis ?
- crb 11y ago
  Last year, we commissioned an independent 3rd-party vendor to prepare a whitepaper comparing GCP pricing with AWS: https://cloud.google.com/files/esg-whitepaper.pdf https://cloud.google.com/files/esg-whitepaper.pdf We also have a handy guide for AWS professionals who are looking at Google's cloud: https://cloud.google.com/docs/google-cloud-platform-for-aws-professionals https://cloud.google.com/docs/google-cloud-platform-for-aws-... [Disclosure, I work for GCP, but I love all TLAs equally]
  - mikeklaas 11y ago
    There are some half-truths in that whitepaper, like "reserved instances cannot be resized" (They can, within the same "class" of instances)
- saturnism 11y ago
  there are a few resources out there, the most recent one: http://www.zdnet.com/article/what-google-says-to-aws-price-cuts-our-cloud-is-still-way-cheaper/ http://www.zdnet.com/article/what-google-says-to-aws-price-c... But GCP also has a pretty nice pricing calculator: https://cloud.google.com/pricing/tco/ https://cloud.google.com/pricing/tco/ Cheers,
an4rchy 11y ago
I'm curious regarding the economics of this but could anyone approximate the cost associated with this move or expenses going forward. It sounds like this would be a pretty big account for Google.
- jamesblonde 11y ago
  You can bet your bottom dollar Spotify are getting heavy, heavy discounts. Google are ramping up here in Stockholm, partly to get this account. This has been in the works since before last summer, so they've taken time before announcing it. The precedent for Spotify was switching from Cloudera to Hortonworks a couple of years ago. Rumours are that they got their licenses for close to nothing from Hortonworks, they were so desparate to close them. Same rumours going around vis-a-vi GCE...
  - jkestelyn 11y ago
    Just with respect to some details: Spotify changed from being a self-supporting (free) user of Cloudera's open source platform to being a paid support customer of Hortonworks. So, it was never actually a Cloudera customer (despite what was widely "reported"). Disclaimer: Cloudera emp here.
    - jamesblonde 11y ago
      Nice, reinforces the point that Spotify likes and Spotify gets discounts on infrastructure :)
simonebrunozzi 11y ago
I want to commend some of the Google Compute Platform employees that commented in this thread. I was at AWS for 6 years (left 2 years ago... today!), and I've always been a proponent of being more open and communicative with developers, but it rarely happened - I guess that AWS' PR policy and such are a big showstopper for these kind of discussions. Although, some individuals did their best (e.g. Jeff Barr) to try to share as much as possible. It seems that the Google guys know how to do it. Keep doing it. It will help your business a lot.
- milesward 11y ago
  Thx boss!
- abrookewood 11y ago
  AWS employees may not speak out very often, but they hardly need to. AWS is a marketing machine, from Jeff's regular blog posts, to the volume of free information they provide and to the conferences that they are running everywhere. I think Microsoft is starting to catch on with Azure promotions & conferences, but it's like Google didn't even know that the there was a race .... and they left their running gear at home.
  - rifung 11y ago
    This might be true but I think the issue is that many AWS employees actually want to speak out or respond to issues personally. I used to work at AWS and the things we could say were very controlled. It could just be my team/managers though.
    - simonebrunozzi 11y ago
      No, I agree with your last statement - at AWS you always feel the invisible shadow of PR :)
  - jsolson 11y ago
    For me personally[0], the difference isn't being able to speak in terms of marketing, but in terms of engaging with users who have problems. It may not translate to much in terms of run rate, but I like to think (or at least hope) the presence of GCP staff on HN, Stack Overflow, and in any other community we happen to be part of will in the long run pay off in terms of goodwill. I am free to comment on posts about GCP here on HN. If there are workarounds, I can post them. If it's a bug I can comment as such and file it internally. I can express opinions as long as they're clearly marked as my own. I didn't have to take any training or end up on a whitelist; Google expressed their faith in my good judgement when they hired me. If nothing else, it was a morale boost to feel trusted to act responsibly. At the other end of the spectrum, GCP has started building out in terms of large-scale engagement. Cloud has been a big part of Google I/O; now there's GCP NEXT as a Cloud-specific user conference: https://cloudplatformonline.com/NEXT2016.html https://cloudplatformonline.com/NEXT2016.html. Building more in this space seems easier to me than a cultural change which encourages (rather than prohibits) customer and community engagement on the individual level, but perhaps I'm being naïve. [0]: I currently work at Google on Compute Engine; I formerly worked at Amazon, although not on AWS.
snissn 11y ago
In the same way that facebook shut down parse, I am afraid that google is going to shut down google cloud. And reluctantly have become okay with relying on AWS.
- ariwilson 11y ago
  Google Cloud interacts with the core of Google's business (web services that needs scalable data infrastructure) in a way Parse never did for Facebook.
  - snissn 11y ago
    Google's core business is ads. Parse makes a lot of sense strategically for facebook, a big moat that they have built is around being your identity on the internet. Parse's platform encourages app developers to use facebook login as a core component of their apps. So even despite being ancillary to their business, they shut it down. Building a technology company around an ecosystem is potentially a lot riskier than people take it as. Especially one with a lot of proprietary bells and whistles..
    - ariwilson 11y ago
      Google's core business is providing web services and putting ads on them. That core business means Google focuses a lot on running those web services effectively, which is why they are now focusing hard on the business angle of running other people's web services. Google spends $2+B per quarter on capital expenditures; where do you think that money is going if not datacenters? Using Facebook as a default login was never core to Parse (e.g. PFUser: https://parse.com/docs/osx/api/Classes/PFUser.html https://parse.com/docs/osx/api/Classes/PFUser.html) nor is Parse nearly on the same scale as Google's investment in its datacenters / web software efficiency.
      - snissn 11y ago
        And at some point your company might decide that it's already its biggest customer for its cloud products and close it to the public.. or it might not.. but it's up for me to decide where I host :)
jamesblonde 11y ago
The big point of this move is the hit it will have on on-premises Hadoop offerings: Cloudera, Hortonworks, MapR. It's a massive vote of no-confidence in their offerings. Spotify are effectively ditching a 2000 server, 90 PB Hadoop cluster to go GCE. As Spotify say themselves, that's big news....
- nrh 11y ago
  Spotifier here. This is an important point. I have nothing bad to say about Cloudera or HWX (disclaimer: we're an HWX customer - we've had a pretty good experience), but I don't really see a compelling reason at this stage to manage your own cluster(s) (HIPAA/regulatory constraints, maybe?) Getting shared-storage and indepedently operated/scaled compute clusters on top of that storage isn't easily achievable with the standard Hadoop stack, and building that on top of HDFS is non-trivial.
  - jamesblonde 11y ago
    In fact, I don't think large orgs like you (Spotify) really want independently operated clusters. That prevents easy sharing of data, causing data silos to appear. You really want to have true multi-tenancy, which isn't in Hadoop yet. Hadoop has worked more on Kerberos support at the cost of features like easy-to-use access control - Apache Ranger or Sentry anybody!?!?
    - ec109685 11y ago
      Yahoo's Hadoop clusters operate in a multi tenant fashion for precisely this reason: to ease sharing of data between groups.
pdknsk 11y ago
I notice App Engine isn't mentioned in their stack. I'm a bit worried that Google abandons it once Snapchat moves off it. It has already been in maintenance mode for some years now (save for Managed VMs). The writing was on the wall when Guido left the team. Apparently Google still uses it internally (as the recent switch to Monorail made evident), so perhaps it's safe for a few more years.
- tschellenbach 11y ago
  With Lambda entering the space, I don't think they will remove app engine. Sounds like they should work hard on improving it.
  - jganetsk 11y ago
    Have you seen Google Cloud Functions? https://cloud.google.com/functions/docs https://cloud.google.com/functions/docs Disclaimer: Google Cloud Pub/Sub engineer here.
- nulltype 11y ago
  While Google could potentially abandon App Engine, it's much more likely it will simply remain unchanged for a number of years (as indeed it has roughly been for the past few years, save for Managed VMs). GCP is now offering an increasingly larger percentage of what App Engine used to offer, so really the future may look a lot more like just build your apps on GCE + PubSub + Cloud Datastore and don't even mess with App Engine at all.
- arouzrokh 11y ago
  Hi pdknsk, I'm a Product Manager on App Engine and wanted to reach out and say that 100% we are invested in app engine and are working to make it your favorite platform. For example, we heard that GAE documentation needs enhancements and we've kicked off a large project to do just this. It's an ongoing effort as you can imagine, but please take a look at the new landing page and managed VM content as well as the left-hand nav reorg we did. Here is the link. More to come of course. https://cloud.google.com/appengine/docs https://cloud.google.com/appengine/docs Please give the new set of docs a try and please give us feedback when you see errors or generally when the content is not up to par with what you expect. There is a feedback tool on the top right hand side of every page. Thank you! Amir
KaiserPro 11y ago
Sorry but spotify's infrastructure is not a shiny beacon of correctness. They've compounded mistake upon "Oooshiny lets use that" here is a post where someone dissects spotify's infrastructure bit by bit: http://www.secretbatcave.co.uk/software/docker-and-you/ http://www.secretbatcave.co.uk/software/docker-and-you/ This isn't an anti google compute, its got some really great features. But, spotify's backend infrastructure ain't one Yes, I appreciate that that post is from a while ago, and they might have completely paid back that massive technical debt/innovation token debt.
- rohansingh 11y ago
  To be fair, the presentation that that article is using as a reference for Spotify's infrastructure was from 3 years ago, and during most of the presentation I talked about what was wrong with how we were doing things. How we do things is quite different now. Check out my DockerCon 2014 presentation or our tech blog for details.
- nrh 11y ago
  Spotifier here. For the record, I hate puppet with the fiery intensity of a thousand suns. It's also pretty hard to magically make go away. I could probably do a whole talk on why puppet is difficult to kill, it's the kudzu of config management. I think it is the worst. We've got our warts and a pile of tech debt, and I wouldn't want anyone to think otherwise. Containers are a part of a long-term strategy to get away from puppet and onto more idempotent units of deployment, and move a lot of what is considered to be "configuration" back into the build process where it belongs.
  - laxk 11y ago
    Is there any public available information where we can read about deployment process in the company?
boulos 11y ago
This was mentioned somewhere deep in a thread, but Spotify (and others!) will be speaking at our upcoming customer conference (GCP Next) in a few weeks: https://cloudplatformonline.com/NEXT2016-schedule.html https://cloudplatformonline.com/NEXT2016-schedule.html You'll be able to livestream the sessions here: https://cloudplatformonline.com/NEXT2016-Live.html https://cloudplatformonline.com/NEXT2016-Live.html Disclosure: I work on Compute Engine (and I'll be at Next).
derEitel 11y ago
I would be interested to get some more technical info. OK so you use GC, but how? What products do you use, how do you build your scalable infrastructure so anywhere in the world I can have all the music in the world? I love reading those type of posts from Netflix and was hoping to find something similar here
- boulos 11y ago
  As I mentioned in a top-level comment (https://news.ycombinator.com/item?id=11159840 https://news.ycombinator.com/item?id=11159840), they will be speaking at GCP Next (which you can livestream) and should have some more detail to share.
verbatim 11y ago
Has anyone seen stats published on how big of an infrastructure Spotify runs? Or how it compares to Netflix, which runs on AWS?
jonesb6 11y ago
Great now they can free up some engineers to bring their Linux client out of beta, which has been in beta since dinosaurs.
- kingosticks 11y ago
  Or work on a successor to libspotify. Ha! Who am I kidding?
halayli 11y ago
Congrats to the sales team at Google Cloud. I wonder how much Google's new AI services played a role in convincing Spotify to use them.
- vgt 11y ago
  Hey guys it's Tino. When will you contribute to our Big Data blog with your excellent ideas? :) https://cloud.google.com/blog/big-data/ https://cloud.google.com/blog/big-data/
  - halayli 11y ago
    I am more than happy to contribute but I have a feeling you've mistaken me with someone else? :)
    - vgt 11y ago
      yup sorry posted in wrong thread.. but please do :)
matt_wulfeck 11y ago
The GCE model is the future. Google knows this because they've already discovered it internally by running their own services. They know containers are he future and they built a lot of the LXC kernel updates that makes things like docker work. Most people don't need the future yet. Really people are just trying to get out of the business of managing and scaling their hardware with as little work as possible. Containerization is much farther down the line. So I posit the future GCE customers are not people "moving to the cloud" like AWS, but rather people "moving to the small, contained, service" -- indeed, it's current AWS customers.
NearAP 11y ago
I've run a few small side projects on GAE (Google App Engine). I use it as a very simple and fast way to get an MVP off. My major issue is that their documentation tends to lag (sometimes way behind) when they change stuff, especially things that are integrated to other services they provide. For example after dropping support for OpenID 2.0, the documentation for 'Authentication Type' on Google App Engine was not updated to reflect this, neither was the documentation for securing your urls. This cost me considerable time and effort.
- arouzrokh 11y ago
  Hi NearAP, I'm a Product Manager on App Engine and I truly apologize for the trouble you went through when you tried the platform. We have kicked off a large initiative to update and enhance our docs, and we just recently launched our new App Engine docs, an ongoing project of course. You can check them out here: https://cloud.google.com/appengine/docs https://cloud.google.com/appengine/docs Please give the new set of docs a try and please give us feedback when you see errors or generally when the content is not up to par with what you expect. There is a tool on the top right hand side of every page that explains this. Thank you! Amir
frakkingcylons 11y ago
The short and sweet of my user story as a hobbyist fullstack developer and wannabe data analyst is AWS has standalone services for your apps that either don't exist on GCP or are better than what GCP offers. I'm talking about pay-as-you-go compute functions with Lambda (I'm aware that GC Functions are on the way, Lambda has been out for more than a year already), managed Postgres with RDS, and managed task queues with SQS. But GCP's console UX is killer compared to AWS, and using BigQuery feels too good to be true compared to how difficult it is to accomplish the same thing with Hadoop/Spark.
mratzloff 11y ago
Today: Highly publicized move onto Google Cloud Later: Silent, gradual move away from Google Cloud after experiencing a number of issues and bugs, constant connectivity issues with services like Cloud Storage, silent failures that do not return error messages on a random basis, detailed billing info only available through arcane APIs and not the administrative UI, broken functionality and rigid inflexibility of BigQuery, and other non-obvious negative behaviors
tschellenbach 11y ago
I wonder if they are still using Cassandra or moved that over as well.
ceocoder 11y ago
We (Sojern) moved to 100% GCP as well this month, admittedly not nearly at the same scale as Spotify, but we did. Other than some growing pains - that GCP team is already aware of - this has been a very smooth migration for us. We use Google Cloud {Storage, BigTable, Container Engine, BigQuery, Monitoring, DNS, Compute} and may be some services I'm missing. I'd be happy to answer any questions.
agentgt 11y ago
I have some questions based on previous experience many moons ago experience (3 or 4 yrs). When I tried to port some of our platform to AppEngine I had some issues with various Java libraries (writing tmp files, opening sockets, etc). Once I sort of got past that and used some of Googles own APIs (which is sort of annoying that I had to couple my stuff) I had timeout issues. See we have to integrate with all these enterprise third parties (SOAP/REST) and these guys can be really slow. We also couldn't use our own pub/sub (AMQP/RabbitMQ) so that was going to have to be ported as well. Things must have changed because its hard for me to imagine Spotify being able to get all their needs met on a rather coupling platform. * How does one write code for Google Cloud while being agnostic of Google Cloud? * Maybe appengine supports AMQP? * Or maybe the container engine fixes these problems? (ie need something more custom run it in a docker image). Our current platform runs on Rackspace and Digital Ocean. These guys are IaaS so your basically doing DevOps which is a pain... but Rackspace does have kick ass customer service.
- dragonwriter 11y ago
  AppEngine is Google Cloud Platform's PaaS offering. Its the oldest product on Google Cloud Platform, but its not one of the ones mentioned that Spotify is using (the Spotify post doesn't talk much about the products, but Google's [0] does.) The Google Cloud Platform services they are identified as using are: Cloud Storage, Compute Engine, Cloud Datastore, Cloud Bigtable, Direct Peering, Cloud VPN, Cloud Router, Cloud Pub/Sub, Cloud Dataflow, BigQuery, and Cloud Dataproc. So, its not surprising that they might be doing something that Google AppEngine doesn't support particularly well. [0] http://googlecloudplatform.blogspot.com/2016/02/Spotify-chooses-Google-Cloud-Platform-to-power-data-infrastructure.html http://googlecloudplatform.blogspot.com/2016/02/Spotify-choo...
  - agentgt 11y ago
    I guess they are not entirely on Google Cloud but rather their data warehousing / big data stuff is now?/going? to be on GCP. I don't really call big data processing "backend" but rather the microservices the backend. Its not clear to me if those are going to be ported over or if they are already. Does spotify plan on porting everything over or just the heavy computing? A detailed techy case study I hope will follow soon as I am interested (the DevOps pain is very real).
dmourati 11y ago
Does Google Cloud offer an equivalent to AWS Direct Connect?
- boulos 11y ago
  Yes. We have both "Carrier Interconnect" (https://cloud.google.com/interconnect/ https://cloud.google.com/interconnect/) and "Direct Peering" (https://cloud.google.com/interconnect/direct-peering https://cloud.google.com/interconnect/direct-peering).
peterfschaadt 11y ago
We host a lot of our applications on Google Compute Engine but have had many issues with even simple features available from other cloud providers. For example, Google's HTTP(S) Load Balancer offering does not support SNI [1], HTTP to HTTPS forwarding [2], or sticky sessions [3], making it unusable for us. The support we pay for has been pretty helpful, but all we hear is "we have a feature request for this but I cannot provide an ETA on when and if this will be implemented". Many of the issues in the Compute Engine public issue tracker [4] haven't been updated in many months or over a year in some cases. Additionally, it can be very confusing finding out which beta services can only be accessed via the gcloud tool and not the web console. Without the ability to even schedule a window for maintenance on our VMs, we often feel like we do not have the control we need over our hosting environment. Please increase your transparency around addressing Google Cloud issues, all the Docker updates have been great, but many of these issues are over a year old and still show-stoppers for some users. Thanks again for churning out new features/services constantly, but please help close out some of these older feature requests! [1] https://code.google.com/p/google-compute-engine/issues/detail?id=289 https://code.google.com/p/google-compute-engine/issues/detai... [2] https://code.google.com/p/google-compute-engine/issues/detail?id=255 https://code.google.com/p/google-compute-engine/issues/detai... [3] https://code.google.com/p/google-compute-engine/issues/detail?id=240 https://code.google.com/p/google-compute-engine/issues/detai... [4] https://code.google.com/p/google-compute-engine/issues/list https://code.google.com/p/google-compute-engine/issues/list
jhgg 11y ago
At work we moved to GCE at the beginning of this year, from Linode after they were having stability issues over the christmas break. No complaints from us. So far have been very happy with it. We were considering moving to AWS, but to realize the same pricing as GCE we'd have to purchase reserved instances - the sustained usage discounts have been huge for us. Additional things we liked: - gcs is way more responsive than S3. also, was fairly painless to migrate our S3 buckets to GCS via the web console. - peering with cloudflare lets us save on bandwidth costs! - network load balancer has shown itself to be very reliable and solid for holding open A LOT sustained websocket connections. - http load balancer has shown itself to be very capable of ssl termination & routing (love that we can route /api to our API servers, and the rest to our static servers). additionally, no pre-warming was required. when we did the datacenter switch, it was just 5 mins of downtime. didn't have to worry about pre-warming for our production load like we would have on ELB. Things we didn't like: - salt-cloud's driver for GCE is still lacking. Can't specify disk size for provisioned storage. Can't specify local SSD storage either. Also parallel provisioning didn't work. Something about PyCrypto not liking the way it forked. Not GCE's fault - billing support non-existent unless you purchase a support plan. - documentation still needs some work.
- jpatokal 11y ago
  Billing support is provided for free to all customers. If you had trouble with a particular ticket, please feel free to ping me. Disclaimer: I work in Cloud Support.
  - jhgg 11y ago
    Thanks for the reply! I don't remember the specific issue, but we had a billing inquiry and were told we had to upgrade to a paid support plan to get an answer.
    - jpatokal 11y ago
      Please feel free to forward the ticket to me at jani at google, and I'll look into it.
  - 11y ago
hanief 11y ago
Om Malik sees it as a first step toward acquisition by Google (or rather, Alphabet). https://twitter.com/om/status/702194246489546752 https://twitter.com/om/status/702194246489546752
- PhantomGremlin 11y ago
  This has been rumored before. E.g here's a headline from 2014: http://www.wsj.com/articles/google-considers-buying-spotify-but-finds-the-price-too-high-1406061732 http://www.wsj.com/articles/google-considers-buying-spotify-...
balls187 11y ago
I wonder if this has to do with Amazon's streaming music competitor being more of a threat to Spotify than Google's?
- ascagnel_ 11y ago
  Amazon's streaming music competitor is awful. The apps are bad, the web UI is bad, the selection isn't as big, and the audio quality is middle-of-the-pack. In general, it's a "oh right, I have that Amazon service because I have Prime"; a similar comparison would be Prime Video to Netflix. Spotify is better off in most ways (their desktop app still sucks, but the web UI is better).
  - balls187 11y ago
    I wouldn't judge a competitive threat solely on a V1 implementation, given it's from a company with engineering chops like Amazon.
doxcf434 11y ago
A little birdy told me that if you find a bug in GCE's live migration they'll buy you a bottle of scotch. Apparently the team gets blamed a lot for problems that turn out not to something else. :)
tomschlick 11y ago
Still no IPv6 on Google's cloud... why?
crimsonalucard 11y ago
Why google over aws? Why isn't this man answering the real question?
ergo14 11y ago
I think I could try moving to GAE if they had Postgresql offering for database storage.

715 points by dmichel 11y ago

dkubb 11y ago

(Serious question) Isn't it risky to rely on any Google product as a foundation for your business? They've shut down more popular services that people depend on than Google Cloud. They tend to enter a market, undercut all the competitors, and then once they are dominant in the niche they terminate or change the service in a way that makes it not fulfill the original promise.

mastazi 11y ago
Well Amazon killed the Fire Phone though ;-) https://fortune.com/2015/09/09/amazon-killing-fire-phone/ https://fortune.com/2015/09/09/amazon-killing-fire-phone/ Jokes aside, probably you are thinking about free products like Google Reader, because this: > once they are dominant in the niche they terminate or change the service would not make any sense if the product in question makes $$$

obulpathi 11y ago

> Google has long been a thought-leader in this space, and this shows in the sophistication and quality of its data offerings. From traditional batch processing with Dataproc, to rock-solid event delivery with Pub/Sub to the nearly magical abilities of BigQuery, building on Google’s data infrastructure provides us with a significant advantage where it matters the most. Big Data is the core strength of Google Cloud. Good to see this move by Spotify! > What really tipped the scales towards Google for us, however, has been our experience with Google’s data platform and tools. Good infrastructure isn’t just about keeping things up and running, it’s about making all of our teams more efficient and more effective, and Google’s data stack does that for us in spades. What I really really liked about Google Cloud is the ease of use. Spin up a VM, start Cloud shell, SSH into your instance, install a bunch of software and you will know what I mean. It's "Quality" Cloud.

sigmar 11y ago

>"...feature testing and more intelligent user-facing features," the Google staffer wrote. I have to wonder if Spotify is doing this so that they can (eventually) use Google's deep learning technology in improving their recommendation engine. They purchased Echo Nest in 2014 to achieve this.

atrudeau 11y ago

This reads like they've been acquired by Google and are integrating. Now Google just has to write the check and own music streaming.

simonswords82 11y ago
Hmmm...interesting point but Google already has an awful lot of music infrastructure in the form of Google Music and YouTube does it not? Would it really make sense for them to purchase Spotify too?
- txru 11y ago
  Buying brand loyalty, perhaps?
- toomuchtodo 11y ago
  They'd be buying the users.
  - simonswords82 11y ago
    That's about the only real value. But then what? Do they run the two entities separately? Or merge Google's offering with Spotify and risk pissing customers off who are loyal to the Spotify brand and app?
    - toomuchtodo 11y ago
      No idea whatsoever. Maybe merge Spotify into Google Fiber and include it as part of a content offering?
- mcintyre1994 11y ago
  I use Google's offering and it's probably pretty solid with YouTube Red, but I'm guessing at this point they're third behind Spotify and Apple?
- zymhan 11y ago

dang 11y ago

Also on this: http://www.wired.co.uk/news/archive/2016-02/23/spotify-move-to-google-cloud http://www.wired.co.uk/news/archive/2016-02/23/spotify-move-... and http://googlecloudplatform.blogspot.com/2016/02/Spotify-chooses-Google-Cloud-Platform-to-power-data-infrastructure.html http://googlecloudplatform.blogspot.com/2016/02/Spotify-choo...

shockzzz 11y ago

Smells like a puff-piece

skj 11y ago
A puff piece would be if some tech reporter wrote about the wonders of GCP. This is the Spotify team, and they've clearly put their money where their mouth is by actually making the move.
- wcummings 11y ago
  Marketing blog spam is the new puff piece.
  - vgt 11y ago
    Would you prefer testaments like these from Spotify engineers? https://twitter.com/plamere/status/702168809445134336 https://twitter.com/plamere/status/702168809445134336 https://twitter.com/plamere/status/702169733454495745 https://twitter.com/plamere/status/702169733454495745 https://twitter.com/mpjme/status/702167231623516160 https://twitter.com/mpjme/status/702167231623516160
    - shockzzz 11y ago
      Kind of. We all know that companies cut deals with providers in exchange for blog posts, tweets, and other sorts of promotion. I'm always wary of posts like these, especially when they lack detail or awareness of competing products.
      - vgt 11y ago
        I think if Spotify dressed down Google Cloud Platform competition, no matter the merit, there would be many more complaints (ex. see comments on this technical post https://news.ycombinator.com/item?id=11029032 https://news.ycombinator.com/item?id=11029032). Spotify did say that they really enjoy specific aspects of Google Cloud Platform, through their own engineering blog. Spotify engineers have also been independently vocal on Twitter. Netflix has been very similarly vocal of their support of AWS. I'd love to hear your thoughts on a better mindshare model, perhaps we can all learn to do things a bit better.

shockzzz 11y ago

Smells like a puff-piece

ed_blackburn 11y ago

I always used to think Facebook would make a move for Spotify..

ed_blackburn 11y ago
-1 :( Zuckerberg used to wax lyrically about Spotify and I always assumed Facebook would one day like to move to the subscription streaming market. A barrier to entry for Spotify to stream movies I always suspected would be cash. Moving to google cloud won't stop a Spotify sale but it may discourage some?
ed_blackburn 11y ago
-1 :( Zuckerberg used to wax lyrically about Spotify and I always assumed Facebook would one day like to move to the subscription streaming market. A barrier to entry for Spotify to stream movies I always suspected would be cash. Moving to google cloud won't stop a Spotify sale but it may discourage some?

waffle_ss 11y ago

I tried Google Container Engine (GKE) and really liked it - it's the best cloud solution for deploying Docker to production in my opinion, mainly due to its use of Kubernetes. Unfortunately in my Web apps I make heavy use of Postgres-specific features, and since Cloud SQL only supports MySQL, Google Cloud is a total non-starter for me. So for now I'm on AWS, using Postgres on RDS and deploying containers with ECS. ECS is a lot simpler than Kubernetes, but since my apps are pretty simple (a half dozen task definitions), it's not a big deal. I really hope Google adds Postgres to Cloud SQL at some point.

stevenleeg 11y ago
Is there any reason you can't mix and match Google Container Engine and Amazon RDS?
- dabeeeenster 11y ago
  Latency
  - MBCook 11y ago
    And bandwidth charges
  - stevenleeg 11y ago
    Got it, thanks! What about running a postgres container on GKE?
    - andrewstuart2 11y ago
      There's nothing stopping anybody for sure, but it's definitely not going to be as hands-off for getting to high availability, backups, scaling well, etc would be with managed SQL as a service.
    - crb 11y ago
      Nice writeup on that here: https://blog.oestrich.org/2015/08/running-postgres-inside-kubernetes/ https://blog.oestrich.org/2015/08/running-postgres-inside-ku...

dublinben 11y ago

When did Spotify abandon their original peer-to-peer architecture?[0] [0]http://www.csc.kth.se/~gkreitz/spotify-p2p10/spotify-p2p10.pdf http://www.csc.kth.se/~gkreitz/spotify-p2p10/spotify-p2p10.p...

philo23 11y ago
Quite a while ago. Not sure on the exact time, but judging from this article, I'd guess sometime in 2014 https://torrentfreak.com/spotify-starts-shutting-down-its-massive-p2p-network-140416/ https://torrentfreak.com/spotify-starts-shutting-down-its-ma...
- hengheng 11y ago
  Seems a great growth hack. Too bad it's not more.
- api 11y ago
  I'd be curious if anyone knows the details about why: complexity, customer complaints, reliability?
  - natarius4k 11y ago
    Too much complexity for a world that's moving to mobile clients that can't be active peers of the network.
    - api 11y ago
      Seems to be a pattern: wimpy mobile endpoint devices drive everything to the cloud. If so this is perhaps a dated phenomenon. Moore's Law is still a thing and today's generation of mobile devices are getting fatter and fatter.
      - LeoPanthera 11y ago
        Moore's Law has not been improving batteries, which is the real problem. Mobile P2P nodes need to be awake most/all the time, which is a real battery killer. Modern devices have improved battery life by being asleep almost all the time.

flurpitude 11y ago

Now how about Spotify fixing their front end so that Premium customers on Android can just hit "play" to play the album they want to play, in the right order? And how about an app in the Windows Store so that you can use it on a touchscreen? And how about some improved metadata that work for classical music? And a way to report corrupt or mistagged tracks and get them fixed? /rant

voltagex_ 11y ago
I'm on mobile so can't directly link you but there's a support form for getting songs fixed.
- flurpitude 11y ago
  I have never found a form, but I have found forums where people complain about persistent issues and nothing gets done. They seem to go out of their way to make it difficult for users to report issues. In addition they seem to have contempt for user experience, or they're incompetent at it, or they are under weird nonsensical licensing pressures that require them to make things difficult for their users. Their front ends need an overhaul and they could easily introduce a "flag this track as having a problem" feature.

icaromedeiros 11y ago

I was wondering about Spotify's contributions to Hadoop stack such as Snakebite and Luigi. What will happen to these projects if they're moving to BigQuery?

thesandlord 11y ago
Google Cloud has a managed Hadoop service called Dataproc: https://cloud.google.com/dataproc/ https://cloud.google.com/dataproc/
crb 11y ago
Spotify are not moving to BigQuery alone. They are also using Dataproc, Google's managed Hadoop service (which co-incidentally went GA yesterday), which lets them get value out of all the orchestration tools they already use, like Luigi. (They are also adopting Cloud Dataflow, Cloud Pub/Sub and other parts of our Big Data stack.) Luigi has had BigQuery support for some time: https://github.com/spotify/luigi/pull/1002 https://github.com/spotify/luigi/pull/1002 Spotify will be talking more about the specifics of their data pipelines at GCP Next in SF in March, - I would expect to see a lot more on the new Google Cloud Big Data blog at https://cloud.google.com/blog/big-data/ https://cloud.google.com/blog/big-data/ too. [Disclosure, GCP guy, etc]
dundun 11y ago
Nothing in this move should affect contributions of Snakebite and Luigi. If anything, it will just make them easier to use with cloud environments in addition to bare metal. With over 20K jobs/day, Hadoop will be a part of Spotify's data processing stack for quite a while. BigQuery is just a (awesome) piece of the full puzzle. [spotifier]

lukasm 11y ago

Isn’t it risky to move to competition's infrastructure?

beambot 11y ago
IIRC, Netflix predominantly hosted on AWS. It's definitely a consideration, but focusing on core competency isn't bad either.
Splines 11y ago
Not really - many people rely on Google Cloud and if Google made it harder for Spotify to compete (via, say, raising prices for Google Cloud for them), they'd have to be pretty specific to Spotify to not cause other customers to move to AWS or Azure. Spotify has much bigger risks in their licensing deals.
- lukasm 11y ago
  cost it not a problem and google wouldn't do it. I'm talking about stealing recommendation algorithms and models. Have a peek new features before release etc.
  - lukasm 11y ago
    Seems googlers are down voting massively.
  - zenlikethat 11y ago
    Recommendation algorithms and models don't do you any good if you don't have users.
    - lukasm 11y ago
      Google Music has a lot of users?

latchkey 11y ago

What Google Cloud needs is a better sales story. AWS consistently beats out Google in this respect. This story is welcomed and there should be more success stories published like this. My business is a heavy GC user (AppEngine, Datastore, CloudStorage, BQ, ComputeEngine, ManagedVM) and I couldn't imagine achieving what we have achieved in such a short amount of time on any other platform. We (myself and another guy) started at zero and shipped our product in 3 months on what is effectively an infinitely auto-scalable platform where we don't have to do any devops or carry a pager. I'd much rather work on features than chores. =) 4 months later and we've had profitable growth and zero downtime (knock on wood) and we're hiring. Thanks Google!

tn13 11y ago
One of the reasons I avoid Google is because they have taken lot of pain to make their customer care totally suck. I mean even IRS and Comcast support does better compared to Google. I published a very critical release to my Android app few weeks back and realized that it was not getting published at all. It took me a month to get the problem resolved. I don't want to buy a cloud service where the support might be nothing more than bunch of "Help yourself" pages.
- latchkey 11y ago
  They have a paid support service that is actually pretty good. It sounds like you're avoiding the generic Google in the same way you'd not buy something from Amazon vs. using AWS.
  - tn13 11y ago
    As I said Google's reputation is what made me not even bother to try. We all know that AWS and AZURE both are state of art and very competitively priced. If Google wants to beat them they have to win the perception war too. (OR reduce prices substantially).
    - boulos 11y ago

misiti3780 11y ago

I had no clue spotify operated their own data centers. Does anyone have a link to that compares google cloud vs aws on a per price + per service basis ?

crb 11y ago
Last year, we commissioned an independent 3rd-party vendor to prepare a whitepaper comparing GCP pricing with AWS: https://cloud.google.com/files/esg-whitepaper.pdf https://cloud.google.com/files/esg-whitepaper.pdf We also have a handy guide for AWS professionals who are looking at Google's cloud: https://cloud.google.com/docs/google-cloud-platform-for-aws-professionals https://cloud.google.com/docs/google-cloud-platform-for-aws-... [Disclosure, I work for GCP, but I love all TLAs equally]
- mikeklaas 11y ago
  There are some half-truths in that whitepaper, like "reserved instances cannot be resized" (They can, within the same "class" of instances)
saturnism 11y ago
there are a few resources out there, the most recent one: http://www.zdnet.com/article/what-google-says-to-aws-price-cuts-our-cloud-is-still-way-cheaper/ http://www.zdnet.com/article/what-google-says-to-aws-price-c... But GCP also has a pretty nice pricing calculator: https://cloud.google.com/pricing/tco/ https://cloud.google.com/pricing/tco/ Cheers,

an4rchy 11y ago

I'm curious regarding the economics of this but could anyone approximate the cost associated with this move or expenses going forward. It sounds like this would be a pretty big account for Google.

jamesblonde 11y ago
You can bet your bottom dollar Spotify are getting heavy, heavy discounts. Google are ramping up here in Stockholm, partly to get this account. This has been in the works since before last summer, so they've taken time before announcing it. The precedent for Spotify was switching from Cloudera to Hortonworks a couple of years ago. Rumours are that they got their licenses for close to nothing from Hortonworks, they were so desparate to close them. Same rumours going around vis-a-vi GCE...
- jkestelyn 11y ago
  Just with respect to some details: Spotify changed from being a self-supporting (free) user of Cloudera's open source platform to being a paid support customer of Hortonworks. So, it was never actually a Cloudera customer (despite what was widely "reported"). Disclaimer: Cloudera emp here.
  - jamesblonde 11y ago
    Nice, reinforces the point that Spotify likes and Spotify gets discounts on infrastructure :)

simonebrunozzi 11y ago

I want to commend some of the Google Compute Platform employees that commented in this thread. I was at AWS for 6 years (left 2 years ago... today!), and I've always been a proponent of being more open and communicative with developers, but it rarely happened - I guess that AWS' PR policy and such are a big showstopper for these kind of discussions. Although, some individuals did their best (e.g. Jeff Barr) to try to share as much as possible. It seems that the Google guys know how to do it. Keep doing it. It will help your business a lot.

milesward 11y ago
Thx boss!
abrookewood 11y ago
AWS employees may not speak out very often, but they hardly need to. AWS is a marketing machine, from Jeff's regular blog posts, to the volume of free information they provide and to the conferences that they are running everywhere. I think Microsoft is starting to catch on with Azure promotions & conferences, but it's like Google didn't even know that the there was a race .... and they left their running gear at home.
- rifung 11y ago
  This might be true but I think the issue is that many AWS employees actually want to speak out or respond to issues personally. I used to work at AWS and the things we could say were very controlled. It could just be my team/managers though.
  - simonebrunozzi 11y ago
    No, I agree with your last statement - at AWS you always feel the invisible shadow of PR :)
- jsolson 11y ago
  For me personally[0], the difference isn't being able to speak in terms of marketing, but in terms of engaging with users who have problems. It may not translate to much in terms of run rate, but I like to think (or at least hope) the presence of GCP staff on HN, Stack Overflow, and in any other community we happen to be part of will in the long run pay off in terms of goodwill. I am free to comment on posts about GCP here on HN. If there are workarounds, I can post them. If it's a bug I can comment as such and file it internally. I can express opinions as long as they're clearly marked as my own. I didn't have to take any training or end up on a whitelist; Google expressed their faith in my good judgement when they hired me. If nothing else, it was a morale boost to feel trusted to act responsibly. At the other end of the spectrum, GCP has started building out in terms of large-scale engagement. Cloud has been a big part of Google I/O; now there's GCP NEXT as a Cloud-specific user conference: https://cloudplatformonline.com/NEXT2016.html https://cloudplatformonline.com/NEXT2016.html. Building more in this space seems easier to me than a cultural change which encourages (rather than prohibits) customer and community engagement on the individual level, but perhaps I'm being naïve. [0]: I currently work at Google on Compute Engine; I formerly worked at Amazon, although not on AWS.

snissn 11y ago

In the same way that facebook shut down parse, I am afraid that google is going to shut down google cloud. And reluctantly have become okay with relying on AWS.

ariwilson 11y ago
Google Cloud interacts with the core of Google's business (web services that needs scalable data infrastructure) in a way Parse never did for Facebook.
- snissn 11y ago
  Google's core business is ads. Parse makes a lot of sense strategically for facebook, a big moat that they have built is around being your identity on the internet. Parse's platform encourages app developers to use facebook login as a core component of their apps. So even despite being ancillary to their business, they shut it down. Building a technology company around an ecosystem is potentially a lot riskier than people take it as. Especially one with a lot of proprietary bells and whistles..
  - ariwilson 11y ago
    Google's core business is providing web services and putting ads on them. That core business means Google focuses a lot on running those web services effectively, which is why they are now focusing hard on the business angle of running other people's web services. Google spends $2+B per quarter on capital expenditures; where do you think that money is going if not datacenters? Using Facebook as a default login was never core to Parse (e.g. PFUser: https://parse.com/docs/osx/api/Classes/PFUser.html https://parse.com/docs/osx/api/Classes/PFUser.html) nor is Parse nearly on the same scale as Google's investment in its datacenters / web software efficiency.
    - snissn 11y ago
      And at some point your company might decide that it's already its biggest customer for its cloud products and close it to the public.. or it might not.. but it's up for me to decide where I host :)

jamesblonde 11y ago

The big point of this move is the hit it will have on on-premises Hadoop offerings: Cloudera, Hortonworks, MapR. It's a massive vote of no-confidence in their offerings. Spotify are effectively ditching a 2000 server, 90 PB Hadoop cluster to go GCE. As Spotify say themselves, that's big news....

nrh 11y ago
Spotifier here. This is an important point. I have nothing bad to say about Cloudera or HWX (disclaimer: we're an HWX customer - we've had a pretty good experience), but I don't really see a compelling reason at this stage to manage your own cluster(s) (HIPAA/regulatory constraints, maybe?) Getting shared-storage and indepedently operated/scaled compute clusters on top of that storage isn't easily achievable with the standard Hadoop stack, and building that on top of HDFS is non-trivial.
- jamesblonde 11y ago
  In fact, I don't think large orgs like you (Spotify) really want independently operated clusters. That prevents easy sharing of data, causing data silos to appear. You really want to have true multi-tenancy, which isn't in Hadoop yet. Hadoop has worked more on Kerberos support at the cost of features like easy-to-use access control - Apache Ranger or Sentry anybody!?!?
  - ec109685 11y ago
    Yahoo's Hadoop clusters operate in a multi tenant fashion for precisely this reason: to ease sharing of data between groups.

pdknsk 11y ago

I notice App Engine isn't mentioned in their stack. I'm a bit worried that Google abandons it once Snapchat moves off it. It has already been in maintenance mode for some years now (save for Managed VMs). The writing was on the wall when Guido left the team. Apparently Google still uses it internally (as the recent switch to Monorail made evident), so perhaps it's safe for a few more years.

tschellenbach 11y ago
With Lambda entering the space, I don't think they will remove app engine. Sounds like they should work hard on improving it.
- jganetsk 11y ago
  Have you seen Google Cloud Functions? https://cloud.google.com/functions/docs https://cloud.google.com/functions/docs Disclaimer: Google Cloud Pub/Sub engineer here.
nulltype 11y ago
While Google could potentially abandon App Engine, it's much more likely it will simply remain unchanged for a number of years (as indeed it has roughly been for the past few years, save for Managed VMs). GCP is now offering an increasingly larger percentage of what App Engine used to offer, so really the future may look a lot more like just build your apps on GCE + PubSub + Cloud Datastore and don't even mess with App Engine at all.
arouzrokh 11y ago
Hi pdknsk, I'm a Product Manager on App Engine and wanted to reach out and say that 100% we are invested in app engine and are working to make it your favorite platform. For example, we heard that GAE documentation needs enhancements and we've kicked off a large project to do just this. It's an ongoing effort as you can imagine, but please take a look at the new landing page and managed VM content as well as the left-hand nav reorg we did. Here is the link. More to come of course. https://cloud.google.com/appengine/docs https://cloud.google.com/appengine/docs Please give the new set of docs a try and please give us feedback when you see errors or generally when the content is not up to par with what you expect. There is a feedback tool on the top right hand side of every page. Thank you! Amir

KaiserPro 11y ago

Sorry but spotify's infrastructure is not a shiny beacon of correctness. They've compounded mistake upon "Oooshiny lets use that" here is a post where someone dissects spotify's infrastructure bit by bit: http://www.secretbatcave.co.uk/software/docker-and-you/ http://www.secretbatcave.co.uk/software/docker-and-you/ This isn't an anti google compute, its got some really great features. But, spotify's backend infrastructure ain't one Yes, I appreciate that that post is from a while ago, and they might have completely paid back that massive technical debt/innovation token debt.

rohansingh 11y ago
To be fair, the presentation that that article is using as a reference for Spotify's infrastructure was from 3 years ago, and during most of the presentation I talked about what was wrong with how we were doing things. How we do things is quite different now. Check out my DockerCon 2014 presentation or our tech blog for details.
nrh 11y ago
Spotifier here. For the record, I hate puppet with the fiery intensity of a thousand suns. It's also pretty hard to magically make go away. I could probably do a whole talk on why puppet is difficult to kill, it's the kudzu of config management. I think it is the worst. We've got our warts and a pile of tech debt, and I wouldn't want anyone to think otherwise. Containers are a part of a long-term strategy to get away from puppet and onto more idempotent units of deployment, and move a lot of what is considered to be "configuration" back into the build process where it belongs.
- laxk 11y ago
  Is there any public available information where we can read about deployment process in the company?

boulos 11y ago

This was mentioned somewhere deep in a thread, but Spotify (and others!) will be speaking at our upcoming customer conference (GCP Next) in a few weeks: https://cloudplatformonline.com/NEXT2016-schedule.html https://cloudplatformonline.com/NEXT2016-schedule.html You'll be able to livestream the sessions here: https://cloudplatformonline.com/NEXT2016-Live.html https://cloudplatformonline.com/NEXT2016-Live.html Disclosure: I work on Compute Engine (and I'll be at Next).

derEitel 11y ago

I would be interested to get some more technical info. OK so you use GC, but how? What products do you use, how do you build your scalable infrastructure so anywhere in the world I can have all the music in the world? I love reading those type of posts from Netflix and was hoping to find something similar here

boulos 11y ago
As I mentioned in a top-level comment (https://news.ycombinator.com/item?id=11159840 https://news.ycombinator.com/item?id=11159840), they will be speaking at GCP Next (which you can livestream) and should have some more detail to share.

verbatim 11y ago

Has anyone seen stats published on how big of an infrastructure Spotify runs? Or how it compares to Netflix, which runs on AWS?

jonesb6 11y ago

Great now they can free up some engineers to bring their Linux client out of beta, which has been in beta since dinosaurs.

kingosticks 11y ago
Or work on a successor to libspotify. Ha! Who am I kidding?

halayli 11y ago

Congrats to the sales team at Google Cloud. I wonder how much Google's new AI services played a role in convincing Spotify to use them.

vgt 11y ago
Hey guys it's Tino. When will you contribute to our Big Data blog with your excellent ideas? :) https://cloud.google.com/blog/big-data/ https://cloud.google.com/blog/big-data/
- halayli 11y ago
  I am more than happy to contribute but I have a feeling you've mistaken me with someone else? :)
  - vgt 11y ago
    yup sorry posted in wrong thread.. but please do :)

matt_wulfeck 11y ago

The GCE model is the future. Google knows this because they've already discovered it internally by running their own services. They know containers are he future and they built a lot of the LXC kernel updates that makes things like docker work. Most people don't need the future yet. Really people are just trying to get out of the business of managing and scaling their hardware with as little work as possible. Containerization is much farther down the line. So I posit the future GCE customers are not people "moving to the cloud" like AWS, but rather people "moving to the small, contained, service" -- indeed, it's current AWS customers.

NearAP 11y ago

I've run a few small side projects on GAE (Google App Engine). I use it as a very simple and fast way to get an MVP off. My major issue is that their documentation tends to lag (sometimes way behind) when they change stuff, especially things that are integrated to other services they provide. For example after dropping support for OpenID 2.0, the documentation for 'Authentication Type' on Google App Engine was not updated to reflect this, neither was the documentation for securing your urls. This cost me considerable time and effort.

arouzrokh 11y ago
Hi NearAP, I'm a Product Manager on App Engine and I truly apologize for the trouble you went through when you tried the platform. We have kicked off a large initiative to update and enhance our docs, and we just recently launched our new App Engine docs, an ongoing project of course. You can check them out here: https://cloud.google.com/appengine/docs https://cloud.google.com/appengine/docs Please give the new set of docs a try and please give us feedback when you see errors or generally when the content is not up to par with what you expect. There is a tool on the top right hand side of every page that explains this. Thank you! Amir

frakkingcylons 11y ago

The short and sweet of my user story as a hobbyist fullstack developer and wannabe data analyst is AWS has standalone services for your apps that either don't exist on GCP or are better than what GCP offers. I'm talking about pay-as-you-go compute functions with Lambda (I'm aware that GC Functions are on the way, Lambda has been out for more than a year already), managed Postgres with RDS, and managed task queues with SQS. But GCP's console UX is killer compared to AWS, and using BigQuery feels too good to be true compared to how difficult it is to accomplish the same thing with Hadoop/Spark.

mratzloff 11y ago

Today: Highly publicized move onto Google Cloud Later: Silent, gradual move away from Google Cloud after experiencing a number of issues and bugs, constant connectivity issues with services like Cloud Storage, silent failures that do not return error messages on a random basis, detailed billing info only available through arcane APIs and not the administrative UI, broken functionality and rigid inflexibility of BigQuery, and other non-obvious negative behaviors

tschellenbach 11y ago

I wonder if they are still using Cassandra or moved that over as well.

ceocoder 11y ago

We (Sojern) moved to 100% GCP as well this month, admittedly not nearly at the same scale as Spotify, but we did. Other than some growing pains - that GCP team is already aware of - this has been a very smooth migration for us. We use Google Cloud {Storage, BigTable, Container Engine, BigQuery, Monitoring, DNS, Compute} and may be some services I'm missing. I'd be happy to answer any questions.

agentgt 11y ago

I have some questions based on previous experience many moons ago experience (3 or 4 yrs). When I tried to port some of our platform to AppEngine I had some issues with various Java libraries (writing tmp files, opening sockets, etc). Once I sort of got past that and used some of Googles own APIs (which is sort of annoying that I had to couple my stuff) I had timeout issues. See we have to integrate with all these enterprise third parties (SOAP/REST) and these guys can be really slow. We also couldn't use our own pub/sub (AMQP/RabbitMQ) so that was going to have to be ported as well. Things must have changed because its hard for me to imagine Spotify being able to get all their needs met on a rather coupling platform. * How does one write code for Google Cloud while being agnostic of Google Cloud? * Maybe appengine supports AMQP? * Or maybe the container engine fixes these problems? (ie need something more custom run it in a docker image). Our current platform runs on Rackspace and Digital Ocean. These guys are IaaS so your basically doing DevOps which is a pain... but Rackspace does have kick ass customer service.

dragonwriter 11y ago
AppEngine is Google Cloud Platform's PaaS offering. Its the oldest product on Google Cloud Platform, but its not one of the ones mentioned that Spotify is using (the Spotify post doesn't talk much about the products, but Google's [0] does.) The Google Cloud Platform services they are identified as using are: Cloud Storage, Compute Engine, Cloud Datastore, Cloud Bigtable, Direct Peering, Cloud VPN, Cloud Router, Cloud Pub/Sub, Cloud Dataflow, BigQuery, and Cloud Dataproc. So, its not surprising that they might be doing something that Google AppEngine doesn't support particularly well. [0] http://googlecloudplatform.blogspot.com/2016/02/Spotify-chooses-Google-Cloud-Platform-to-power-data-infrastructure.html http://googlecloudplatform.blogspot.com/2016/02/Spotify-choo...
- agentgt 11y ago
  I guess they are not entirely on Google Cloud but rather their data warehousing / big data stuff is now?/going? to be on GCP. I don't really call big data processing "backend" but rather the microservices the backend. Its not clear to me if those are going to be ported over or if they are already. Does spotify plan on porting everything over or just the heavy computing? A detailed techy case study I hope will follow soon as I am interested (the DevOps pain is very real).

dmourati 11y ago

Does Google Cloud offer an equivalent to AWS Direct Connect?

boulos 11y ago
Yes. We have both "Carrier Interconnect" (https://cloud.google.com/interconnect/ https://cloud.google.com/interconnect/) and "Direct Peering" (https://cloud.google.com/interconnect/direct-peering https://cloud.google.com/interconnect/direct-peering).

peterfschaadt 11y ago

We host a lot of our applications on Google Compute Engine but have had many issues with even simple features available from other cloud providers. For example, Google's HTTP(S) Load Balancer offering does not support SNI [1], HTTP to HTTPS forwarding [2], or sticky sessions [3], making it unusable for us. The support we pay for has been pretty helpful, but all we hear is "we have a feature request for this but I cannot provide an ETA on when and if this will be implemented". Many of the issues in the Compute Engine public issue tracker [4] haven't been updated in many months or over a year in some cases. Additionally, it can be very confusing finding out which beta services can only be accessed via the gcloud tool and not the web console. Without the ability to even schedule a window for maintenance on our VMs, we often feel like we do not have the control we need over our hosting environment. Please increase your transparency around addressing Google Cloud issues, all the Docker updates have been great, but many of these issues are over a year old and still show-stoppers for some users. Thanks again for churning out new features/services constantly, but please help close out some of these older feature requests! [1] https://code.google.com/p/google-compute-engine/issues/detail?id=289 https://code.google.com/p/google-compute-engine/issues/detai... [2] https://code.google.com/p/google-compute-engine/issues/detail?id=255 https://code.google.com/p/google-compute-engine/issues/detai... [3] https://code.google.com/p/google-compute-engine/issues/detail?id=240 https://code.google.com/p/google-compute-engine/issues/detai... [4] https://code.google.com/p/google-compute-engine/issues/list https://code.google.com/p/google-compute-engine/issues/list

jhgg 11y ago

At work we moved to GCE at the beginning of this year, from Linode after they were having stability issues over the christmas break. No complaints from us. So far have been very happy with it. We were considering moving to AWS, but to realize the same pricing as GCE we'd have to purchase reserved instances - the sustained usage discounts have been huge for us. Additional things we liked: - gcs is way more responsive than S3. also, was fairly painless to migrate our S3 buckets to GCS via the web console. - peering with cloudflare lets us save on bandwidth costs! - network load balancer has shown itself to be very reliable and solid for holding open A LOT sustained websocket connections. - http load balancer has shown itself to be very capable of ssl termination & routing (love that we can route /api to our API servers, and the rest to our static servers). additionally, no pre-warming was required. when we did the datacenter switch, it was just 5 mins of downtime. didn't have to worry about pre-warming for our production load like we would have on ELB. Things we didn't like: - salt-cloud's driver for GCE is still lacking. Can't specify disk size for provisioned storage. Can't specify local SSD storage either. Also parallel provisioning didn't work. Something about PyCrypto not liking the way it forked. Not GCE's fault - billing support non-existent unless you purchase a support plan. - documentation still needs some work.

jpatokal 11y ago
Billing support is provided for free to all customers. If you had trouble with a particular ticket, please feel free to ping me. Disclaimer: I work in Cloud Support.
- jhgg 11y ago
  Thanks for the reply! I don't remember the specific issue, but we had a billing inquiry and were told we had to upgrade to a paid support plan to get an answer.
  - jpatokal 11y ago
    Please feel free to forward the ticket to me at jani at google, and I'll look into it.
- 11y ago

hanief 11y ago

Om Malik sees it as a first step toward acquisition by Google (or rather, Alphabet). https://twitter.com/om/status/702194246489546752 https://twitter.com/om/status/702194246489546752

PhantomGremlin 11y ago
This has been rumored before. E.g here's a headline from 2014: http://www.wsj.com/articles/google-considers-buying-spotify-but-finds-the-price-too-high-1406061732 http://www.wsj.com/articles/google-considers-buying-spotify-...

balls187 11y ago

I wonder if this has to do with Amazon's streaming music competitor being more of a threat to Spotify than Google's?

ascagnel_ 11y ago
Amazon's streaming music competitor is awful. The apps are bad, the web UI is bad, the selection isn't as big, and the audio quality is middle-of-the-pack. In general, it's a "oh right, I have that Amazon service because I have Prime"; a similar comparison would be Prime Video to Netflix. Spotify is better off in most ways (their desktop app still sucks, but the web UI is better).
- balls187 11y ago
  I wouldn't judge a competitive threat solely on a V1 implementation, given it's from a company with engineering chops like Amazon.

doxcf434 11y ago

A little birdy told me that if you find a bug in GCE's live migration they'll buy you a bottle of scotch. Apparently the team gets blamed a lot for problems that turn out not to something else. :)

tomschlick 11y ago

Still no IPv6 on Google's cloud... why?

crimsonalucard 11y ago

Why google over aws? Why isn't this man answering the real question?

ergo14 11y ago

I think I could try moving to GAE if they had Postgresql offering for database storage.