10 ms·
How Spotify ran a large Google Dataflow job for Wrapped 2019
- fs111 7y agowhy is this link doing a redirect through some ad network?
- Swtrz 7y agoI wonder why I never see this behavior despite every other person mentioning it
- corndoge 7y agoublock origin shows a confirm page when it happens
- jdormit 7y agoIt's really quick. Open the network tab and check the "persist logs" checkbox to ensure that the request logs don't disappear after every redirect, then clear your cookies for advertising.com and guce.techcrunch.com and reload the page. You'll see the request for techcrunch.com redirect to guce.techcrunch.com, which redirects to guce.advertising.com, which redirects back to techcrunch.com. It happens so fast it's not noticeable on page load.
- dang 7y agoWe've since changed the link, which originally was https://techcrunch.com/2020/02/18/how-spotify-ran-the-largest-google-dataflow-job-ever-for-wrapped-2019/ https://techcrunch.com/2020/02/18/how-spotify-ran-the-larges....
- C14L 7y agoBecasue more and more browsers are limiting access to cookies not only depending on first-party context but also third-party context. So tracking users by web bugs becomes less reliable. By redirecting throught the domain, they can set and access cockies in a first-party context.
- stilisstuk 7y agoNo tech crunch... You can't have my cookies..
- fmjrey 7y agoThis may be a more appropriate source, from the source: https://labs.spotify.com/2019/11/12/spotifys-event-delivery-life-in-the-cloud/ https://labs.spotify.com/2019/11/12/spotifys-event-delivery-...
- stingraycharles 7y agoMuch better article, thanks for sharing.
- mackey 7y agoThis is correct link https://labs.spotify.com/2020/02/18/wrapping-up-the-decade-a-data-story/ https://labs.spotify.com/2020/02/18/wrapping-up-the-decade-a...
- dang 7y agoOk, we've changed to that from https://techcrunch.com/2020/02/18/how-spotify-ran-the-largest-google-dataflow-job-ever-for-wrapped-2019/ https://techcrunch.com/2020/02/18/how-spotify-ran-the-larges.... Thanks all!
- gabagool 7y agoThe new Spotify blog only states that "the Wrapped Campaign data pipeline had one of the largest Dataflow jobs to ever run on GCP," without claiming that it was the largest ever. I didn't see any additional evidence in the TechCrunch article to support this being the largest either. Not sure if a better title is warranted ("How Spotify ran its massive Google Dataflow job for Wrapped 2019", "How Spotify ran one of the largest Google Dataflow jobs ever for Wrapped 2019"?).
- dang 7y agoOk, we've knocked the largest down to size in the title above. I always tell startups not to use superlatives on HN. Modest language sounds stronger.
- data4lyfe 7y agoOne massive SQL query across a billion plus users.
- ipnon 7y agoDatabases are the one area of computer science that makes me realize these machines can do magical things.
- downerending 7y agoImpressive, but I'd be more impressed if they fixed their random shuffle.
- nvarsj 7y agoWhat's wrong with the spotify shuffle? edit: Did a search, seems like there's quite a few problems (only playing recently added songs, only playing 100 songs out of the playlist, etc.). I know google music has also had long standing issues with shuffle play - and in fact I left it over these kind of issues. Is it really difficult to implement a shuffle?!
- fuzzmz 7y agoIt's not really random, in the sense that if you have a playlist and hit shuffle it'll always play in the same order instead of randomizing the play order each time you listen to that playlist. Basically, with the current behavior, once you learned the order of the shuffled songs you can always know what comes next.
- mrkeen 7y agoIt may be the case that 100 tracks are sent to the device and the shuffle logic chooses from them locally.
- kingosticks 7y agoNot sure why you are being down voted, this is essentially how Spotify's shuffle works. At least, if you MITM the official client and load a large playlist/context you'll only see a small window worth of tracks being loaded. And you won't see any request from the client when you then shuffle that playlist, it's done locally. This may, of course, have changed. My experiments while (badly) implementing librespot's shuffle functionality were a few years ago now.
- dvtrn 7y agoI thought we had a thing about preserving post titles from the source?
- jsnell 7y agoThe source changed, the title didn't.
- capableweb 7y agoThat's still true, submission used to link to https://techcrunch.com/2020/02/18/how-spotify-ran-the-largest-google-dataflow-job-ever-for-wrapped-2019/ https://techcrunch.com/2020/02/18/how-spotify-ran-the-larges... See https://news.ycombinator.com/item?id=22359865 https://news.ycombinator.com/item?id=22359865
- dna_polymerase 7y agoBasically the perfect use case for cloud computing. Tons of compute for a short time. In this case there can’t possibly be people arguing for their own datacenter over cloud.
- wrkronmiller 7y ago> Basically the perfect use case for cloud computing. Tons of compute for a short time. I completely agree. > In this case there can’t possibly be people arguing for their own datacenter over cloud. Devil's advocate time: This solution was great for the cloud because it was designed for the cloud. There might be equally good or even superior solutions designed for on-prem or even on-device computing. For example, this ceases to be a big-data problem if you are simply aggregating listening metrics for a single user on a single device.
- gen220 7y agoIMO, this is a great example for how the policy of “owning your own data” actually leads to objectively “better” Engineering solutions. If Spotify leveraged my phone to calculate these statistics of my listening history (owned and stored locally), this article would have been written about an app update. No need for a massive ad-hoc job with high-bandwidth round trips, just a simple app update. It’s funny to imagine how engineers of the future might look back on our pride over this kind of computing similar to how we look back in horror on how wasteful we once were with mining oil back in the 1910s, etc.
- joshuamorton 7y ago> If Spotify leveraged my phone to calculate these statistics of my listening history (owned and stored locally), this article would have been written about an app update. Then the article would be about the challenges of battery life on users' phones, and trying to coordinate listening history on PC vs. phone.
- gen220 7y ago
- gwittel 7y agoInteresting. I wish it had more details as far as inputs/outputs, data sizes in different phases. One thing that I wonder about is how much work could they do to collect this data on a forward moving basis. Often I see huge lookback jobs that answer predictable/static questions -- prime candidates for aggregation during ingest.
- wobblykiwi 7y agoThis is the thing I was most forward to reading about in the article, but there were no figures about how large the "largest Google Dataflow job ever" is. There are a bunch of relative figures, 5x 2018 - but what does that translate to? How long did it take?
- tylerl 7y agoYa, concrete details were conspicuously missing. Like petabytes? Exabytes? I suspect that the "largest dataflow job ever" is significantly smaller than the kind of crap Google regularly throws at the backend that dataflow runs on. With that infrastructure at their fingers, I suspect engineers regularly fire off jobs orders of magnitude larger than necessary simply because it's not worth the 3 hours of human effort it'd take to narrow down the input set.
- rsmets 7y agoI thought this was such a marvel! However, my excitement level was tapered when I realized the playlist Best of the Decade was not created by only my music listening habits. Seems as though users were pinned to some general playlist that had characteristics similar to listening habits? Still hats off from an engineering perspective. I as well wish there was more technical detail provided. The year recap playlists though are fun personal snapshot of time.
- aabeshou 7y agoIt's interesting to confirm that because anecdotally my best of the decade playlist sucked lol. It had songs that I really don't think I listened to that much or liked that much. It was weird.
- paxys 7y agoI think the decade lists were a bit underwhelming considering not too many people were actually using Spotify all that much 10 years ago. I still got a ton of my music from CDs, iTunes downloads and other more nefarious places.
- thejosh 7y agoI found my decade wrapped interesting, as I have been using Spotify for most of the decade and loved my change of music tastes. And the reaction to the music as I remembered which projects I was working on as I listened to those on repeat.
- matsemann 7y agoI became a paying customer (Premium subscriber) oct 5, 2009. Everyone at my school was using Spotify at the time, albeit the free version. (Norway)
- huseyinkeles 7y agoInteresting, maybe it was more popular back then in the nordics as spotify is a swedish company?
- booboolayla 7y agoThis smells like an initiative to boost "women in tech" storyline - the task at hand is something many, many companies do daily, the blog has no technical substance, and also just take a look at the team members at the bottom of the page. Kind of reminds me of the time a trans person from Google "calculated the number Pi" and we were all supposed to cheer it as some kind of an accomplishment.
- dang 7y agoThere's more info at https://techcrunch.com/2020/02/18/how-spotify-ran-the-largest-google-dataflow-job-ever-for-wrapped-2019/ https://techcrunch.com/2020/02/18/how-spotify-ran-the-larges.... (via https://news.ycombinator.com/item?id=22359528 https://news.ycombinator.com/item?id=22359528)
- drdoooom 7y agoWas a neat little feature, too bad the share functionality didn't actually work.
- deepsun 7y agoI'd recommend them to check out Clickhouse for exactly the same purposes. Works well for Cloudflare, Yandex, Sentry. Another idea is to run probabilistic queries instead of exact ones, could bring down costs way more.
- matlin 7y agoI'm curious how much data this involves per user. This is clearly a massive undertaking when you're talking about ~250 million users but I bet it would be easy to provide the same info if all the data was local on a device and each user ran their own query. This assumes that the space required to store all of your listening history fits on device which I think is a safe bet.
- paxys 7y ago> This assumes that the space required to store all of your listening history fits on device which I think is a safe bet Space-wise, yes, but users are likely using multiple devices and may have switched phones, reinstalled the app, wiped data etc. Then you have to consider that the scripts would have to be individually written for each platform, and would have to be careful about power consumption, CPU usage etc., especially on mobile devices. And there's not just data mining but also video encoding (for the stories). And then there's this part: > To bring you a Decade Wrapped, we had to process these data stories over 10 years’ worth of data for all of our monthly active users
- herbstein 7y ago> And there's not just data mining but also video encoding I was under the impression that the stories were live graphics. They certainly where on PC, as I had issues running the WebGL because of my script blockers.
- xrjn 7y agoI made a GDPR request for my data shortly after the law was enacted, and they provided me with 280mb of data for the past 90 days of me listening to music.
- matlin 7y agowow that's impressive. I did a CCPA request and got 2.2MB (466KB zipped) of data from the last year which included listening history, playlists, and search history.
- swagonomixxx 7y agoThis is interesting, but what I actually find even more interesting than this is Spotify continuing it's usage of Google Cloud products even after being acquired by Microsoft. Can anyone shed some light as to why this is the case? Has that acquisition not been a "traditional" MS acquisition?
- dajohnson89 7y agoi... don’t think microsoft acquired microsoft
- lucasverra 7y agoSpotify is not a Microsoft asset
- npad 7y agoMicrosoft doesn’t own Spotify. It seems there were rumours they might be acquired by MS around the time of the IPO but nothing came of it.
- luhn 7y agoThere was some news about Microsoft acquiring Spotify in April 2018, but as far as I can tell that never went through.
- dundun 7y agoThere was also a breaking story about Google acquiring Spotify on the exact same date a year later!
- deleted 7y ago[deleted]
- kevsim 7y agoAssuming you mean the “acquisition” that was rumored around the time of Spotify’s IPO, you may want to check the date of that article: April 1, 2018 https://www.digitalmusicnews.com/2018/04/01/spotify-microsoft-acquisition-billions/ https://www.digitalmusicnews.com/2018/04/01/spotify-microsof...
- justlexi93 7y agoIn early December, Spotify launched its annual personalized Wrapped playlist with its users’ most-streamed sounds of 2019. That has become a bit of a tradition and isn’t necessarily anything new, but for 2019, it also gave users a look back at how they used Spotify over the last decade. Because this was quite a large job, Spotify gave us a bit of a look under the covers of how it generated these lists for its ever-growing number of free and paid subscribers.
- deleted 7y ago[deleted]
- spyke112 7y agoThis they can do, but you can't change your display name unless you hook up with Facebook. [0] [0] https://community.spotify.com/t5/Live-Ideas/Account-Change-Username/idi-p/703799 https://community.spotify.com/t5/Live-Ideas/Account-Change-U...
- cuddlecake 7y agoYou also have to be online with good connectivity within the app to view the menu for removing an album from the device. Presumably, because offline access means access only if you are really, truly, completely offline.
- simfoo 7y agoJust opening the context menu of a song (to add it to the queue for example) now requires connectivity and does at least one round-trip to their servers. Unless you actually cut the connection (e.g. air plane mode), then it suddenly works fine
- ddoeth 7y agoThey should just have set the timeout to half a second and no one would have noticed
- Hasnep 7y agoYou can make a new account and contact the support team who can transfer almost all your info to the new account including followers and playlists. Not ideal though.
- mjdude 7y agoHaving gone through this, they cant transfer your history and recommendations take a few weeks to adjust.
- spyke112 7y agoI find the biggest value of Spotify is the listening history they have gathered on me throughout the years. I would hate to lose all those excellent recommendation they give me based on that data.