7 ms·
Spotify Optimized the Largest Dataflow Job Ever for Wrapped 2020
- max_streese 6y agoHi not sure if I am just completely off here but I am wondering how this relates or compares to processing things with Kafka and Kafka Streams? If I am reading things correctly with Kafka the workflow equivalent to what's written in the article would be to have your producer produce via hash-based-round-robin (the default partitioning algorithm) based on the key you are interested in into some topic and then your consumer would just read it and your data would already be sorted for the given keys (because within a partition Kafka has sorting guarantees) and also be co-partitioned correctly if you need to read some other topic in with the same number of partitions and the same logical keys produced via the same algorithm. No?
- jsjsbdkj 6y agoThis is the most basic pattern for distributed joins - you hash on the join key in both tables and shuffle data based on hash ranges. In some systems like Redshift you can designate the key for distribution so that "related" records are already co-located on a single shard. > our data would already be sorted for the given keys (because within a partition Kafka has sorting guarantees) It's been a while since I used Kafka but I don't remember "sorting guarantees". Consumers see events "in order" based on when they were produced, because each partition is a queue.
- max_streese 6y agoYes I guess my point is when using Kafka in combination with Kafka Streams and you produce things partitioned in a way that you need them for consumption then you do not need to do any shuffling in the instance where you want to join because data is already partitioned correctly.
- dvdhnt 6y agoYou seem to know what you're talking about. Any recommendations on learning resources for this type of flow? Or really understanding which platform works for in each situation? I'm learning proper data flow in real time as I look to transition ETL of product data into Postgres to a more applicable system. Finding the right learning resources is difficult! Cheers.
- vdm 6y agohttps://dataintensive.net/ https://dataintensive.net/
- gripfx 6y agoYet, I, as a user still cannot see my playcounts via app or API.
- marliechiller 6y agothis is a massive gripe of mine. I remember being able to toggle so many stats in itunes back in the day which gave me such great insight into my listening habits and other cool tidbits. Spotify, for all its ease of listening has removed some of the magic there
- deleted 6y ago[deleted]
- utucuro 6y agoThat might be explained as guarding their information, yet I still have trouble believing that they are unable to distinguish two artists with the same name from each other. Each week, my Release Radar playlist is basically invaded by 0 listener rappers who "feat" defunct 70s rock bands who do not manage their artist pages... Response from Spotify: please report them individually, from the desktop app.
- sbarre 6y agoThat's some interesting product hacking in a way. I've run into artist naming collisions enough to know it's a thing but I've never seen anyone (ab)use it intentionally.. Neat, but still annoying I'm sure. =)
- thejoeflow 6y agothis. They don't even do an exact string match of the artist name, the reason why I can't fathom. I could make a new artist account named "drake" (lowercase d) today and probably show up in a million Release Radar playlists by next week.
- asutekku 6y agoThis is one of the reasons why I still keep using lastfm. Stats in Spotify are next to useless and even then it would not track my local listening.
- iamacyborg 6y agoAnd they’re still less useful than the data last.fm makes available to you.
- dewey 6y agoUseful how? What actionable insights do you get from Last.fm? It's vanity metrics optimized for sharing on social networks and there's nothing wrong with it.
- iamacyborg 6y agoWell, for one it gives me the yearly summary once the year is actually over. It's not particularly actionable to know you listen to more music on a Saturday than a Sunday, but it is mildly interesting for those who are curious about such things.
- sbarre 6y agoWrapped is a marketing product, not a data product. Spotify itself is not a data product. The first rule of avoiding disappointment is managing your own expectations to align with reality. That said, you can export more granular data Spotify has on you from your settings page[0] if you want to do your own deeper analysis on your trends and usage, etc.. 0: https://www.spotify.com/account/privacy/ https://www.spotify.com/account/privacy/
- fooblat 6y agoWhen this report came out it was the straw that broke the camel's back for me in terms of my data privacy. Most people seem to have found Wrapped 2020 entertaining but I found it creepy. I miss being able to do something simple like listen to music or watch a movie without all my actions being recorded and saved. So I'm back to buying physical media and DRM free downloads. I'm convinced that it is now important to hold on to older appliances that work without internet access or data collection this plus right to repair gives me hope for the future.
- 88 6y agoWhat’s your concern, that your listening history will be linked to your personal identity and somehow used against you? If that’s the case, why not just sign up using a single-use email address, pay via gift cards purchased with cash, and if you’re really concerned, use a VPN?
- avh02 6y agoBecause going through all that just to listen to music is bonkers
- marliechiller 6y agobut also far less effort and money than buying physical copies of music
- avh02 6y agoOh I totally agree, for me this is a matter of an absolutely backwards industry. Very often I WANT to give my money but can't find where to buy a DRM-free track, preferably in flac. And registering for 8 different stores is not a valid solution.
- spotyoufi462881 6y agoThis is what relentless back-and-forth made someone realise how much Spotify loves him. https://twitter.com/steipete/status/1025024813889478656 https://twitter.com/steipete/status/1025024813889478656
- staticelf 6y agoSpotify is a company that feels like they want to be a "big tech company" when in reality they do not need to. All they need to do is provide a great service with as much music as possible.
- throwaway3699 6y agoThey clearly value the free+recommendations+ads stack to drive their business more than adding basic features.
- horseRad 6y agoI for one love discover weekly and hope they will continue working on improving it. I consider it a core feature :)
- staticelf 6y agoSure, that is a great feature but that doesn't really need that much data besides listen data that they need to save anyway (I am assuming).
- Hacman123 6y agoAgree 100%. I always find great songs there. A cool feature to discover new music. But I guess we are not that sophisticated. Predicting which songs I'm gonna like is probably noit that difficult.
- onion2k 6y agoAll they need to do is provide a great service with as much music as possible. I wonder if adding 'as much music as possible' would drive any growth without the AI music discovery stuff. There has to be a diminishing return to adding new songs - if Spotify adds an artist that only a few thousand people have heard of then that's only going attract a few thousand new customers at most. No one else is going to listen to that artist unless Spotify recommends the songs to people who might like them.
- danielscrubs 6y agoWouldn’t surprise me if they shuffle more data than Twitter. Is it because it’s not a Silicon Valley darling that it gets all the snide remarks?
- soheilpro 6y agoSorry if it's off-topic, but if anyone's interested, I'm launching volt.fm next week. It connects to your Spotify account and generates a nice public page with your stats (top artists, top tracks), playlists and etc. You can reserve your username now: https://volt.fm https://volt.fm
- pohy 6y agoHi, how is this different/better than last.fm?
- soheilpro 6y agoIt helps you promote your playlists and discover new ones.
- saberience 6y agoWhy would I want to promote my playlists?
- polskibus 6y agoI wonder if they could publish dollar cost of that job before and after the optimization, as provided by GCP billing. I know it could be a bit unfair (some costs may be static, regardless of job size, etc.) but it would improve decision making for others if discussions of public cloud usage optimizations also include the cost.
- sbarre 6y agoThey do refer to savings in percentages.. I feel like giving away actual dollar costs would potentially break contractual agreements (because I doubt Spotify pays public list price), and potentially give away competitive information about how much data they have on their customers etc.. I agree with you that it would be interesting to know, I just don't think it's realistic for them to release that information.
- spotyoufi462881 6y agoAnyone wanna know how much Spotify wanna know about you? https://twitter.com/steipete/status/1025024813889478656 https://twitter.com/steipete/status/1025024813889478656
- float4 6y agoI wonder whether they use the bluetooth device logging purely to develop their social graph. A person streaming music to their own bluetooth headphone using a friend's computer and Spotify account can be detected this way. I can't really come up with another purpose for it. Other than that, I'm not surprised by what they log. Virtually every company stores search queries, oauth grants, play history, ad interactions etc. Doesn't make it right of course.
- Jonnax 6y agoWhat's scary about this? They're complying with GDPR. Isn't that a good thing? The "scary" thing in that tweet is that they store the manufacturer of their bluetooth headphones?
- spotyoufi462881 6y agoThat's not what one could call complying. He had to follow them like a dog for a long while just to get his rightful data.
- Jonnax 6y agoThat was from 2018. Almost 3 years ago.
- yaqoob 6y agoWoo. Thats incredible. Seriously!
- npteljes 6y agoI'm already thinking that every service hoovers as much up as they can. Nice to see proof that it is actually happening!
- greatthx 6y agoGreat! Now will they stop scanning my entire hard drive with their desktop app? Also stop opening sockets directly to advertiser IP address. And stop paying off data thieves instead of disclosing to their users that their passwords were leaked. And to stop being sellouts too!
- jeofken 6y agoDo you have links to further readings about this?
- meibo 6y agoWrapped works so well because it panders to you. Everyone likes to be acknowledged for listening to the weird indie band they discovered earlier this year. I enjoy it anyways, and Spotify is still a great service for now - I wonder if it'll meet the same fate as Netflix at some point, with publishing houses going for their own streaming services instead.
- onion2k 6y agoI wonder if it'll meet the same fate as Netflix at some point, with publishing houses going for their own streaming services instead. I don't think so because the way that people consume music and the way they consume films and television are very different. With a film you might block out a few hours to watch that specific provider. With music you're more likely to want to interleave content from several providers at the same time (eg a playlist). Unless all the providers are available on the same platform it wouldn't work well.
- hnlmorg 6y agoA considerable amount of music is distributed by a small subset of providers. So in that regard it's not that different to the TV / film situation and you could theoretically still interleave different artists. There's also a common use case where people will just play a specific artist for an hour. Or even an album. Frankly, I hope services like Spotify don't disappear. It's a great loss to consumers just how fragmented video streaming services have become. I'd hate to see the same happen to music as well.
- alvarlagerlof 6y agoI don't think that behavior is all to common. I see people listening to a very wide range of artists and almost never reach for stuff outside of it around here.
- hnlmorg 6y agoI use music streaming services to play specific albums (as I tend to play older artists who created albums designed to be played as a whole entity rather than singles) My wife uses them to shuffle singles by specific artists (she’s more into pop music). I’d wager if my wife and I both coincidentally follow the same pattern despite doing so for different reasons, that it’s then likely a more common pattern than first assumed. Please also bare in mind that I’m not suggesting our use cases are how the majority of people consume music, but I’d be surprised if it was small enough to be a rounding error.
- matsemann 6y agoI came here expecting to read about the tech in the article or how others do big data processing stuff. Instead I get off topic Spotify rants.. Did you read the article or just see Spotify in the headline and decided your gripes therefore are relevant?
- chishaku 6y agoYou discover that people have different thoughts than you and decided your gripe about this is therefore relevant? Upvote, downvote or move on.
- matsemann 6y agoI agree that my post is just additional noise. But it's noise not disturbing a good signal. Let me just point out that I think discussing sides of what's linked can be interesting and relevant. In this instance I think discussing privacy around the amount of data Spotify stores is a relevant subdiscussion worth exploring. But complaints about their UI doesn't feel very relevant. Etc. Now that's only my opinion. What made me write my comment was that it was literally no one discussing the concepts of the article in the first 15 or so comments. Which I found a bit disappointing as I thought the tech is interesting.
- chishaku 6y ago> But it's noise not disturbing a good signal. The signal will always be weak early on. Look at the comments now and it's clear that the upvote/downvote mechanisms are working sufficiently to address your concern. This should not be surprising. It's typically faster to not read the article and spew superficial off-topic comments than to write on-topic, substantive, and technical comments.
- matsemann 6y agoGood point, although I felt it was more than normal in this case. Having my meta comment on top is also no good, hopefully it can be demoted somehow. Will try to flag this thread.
- deleted 6y ago[deleted]
- mouldysammich 6y agolistenbrainz has gotten pretty good in the last while for keeping track of your music stats. Its got a nice weekly stats page etc. Its also not ad laden like last.fm
- iliekcomputers 6y agoGlad that you like it! I'm part of the (small) team working on ListenBrainz. If you have any feedback or feature requests, I'd be happy to hear it. My email is me [at] param.codes.
- saberience 6y agoOne thing I can never understand about Spotify is that despite it's insane budget, huge amount of employees/talent, they still can't create better personalized playlists than either Pandora OR last.fm. To this day when I want a recommended playlist based on my taste/history, I always use last.fm because it's just plain better. Why? The "Discover" etc playlists on Spotify are just crap.
- feintruled 6y agoReally? I find my 'Discover Weekly' playlist to be astounding. I have to confess for a long time I thought it was a human curated playlist from someone who just happened to have exactly the sames tastes as me.
- adamhp 6y agoI have to second this. I can't list the number of artists I have discovered over the last few years using Discover Weekly alone. It's really incredible.
- iamacyborg 6y agoIt's been the opposite for me. It regularly surfaces songs I've listened to in the past from artists I listen to frequently. As a music discovery algo Spotify has been nothing but poor in my opinion.
- H8crilA 6y agoRemember the Netflix Challenge fiasco? It's pretty clear that this problem is either ridiculously hard, or it is fundamentally "unsolvable" (because, perhaps, it depends on highly variable mood).
- deleted 6y ago[deleted]
- CyberRabbi 6y ago“How <moderately large startup> did <some obscure tech thing>” <... low substance and poorly written synopsis of the obscure tech thing ...> “Enjoy work like this? Want to make a big impact as a lowly programmer who will never start their own company? Well at <moderately large startup>, lowly programmers who will never start their own companies have the freedom to make a big impact.”
- rrdharan 6y agoSpotify is no longer a startup.
- CyberRabbi 6y agoGood point. That’s changes the substance of the post.
- BorisTheBrave 6y agoI'm having a hard time understanding this article. It seems to be a bit too low level on the specifics of Beam for general consumption. From what i undestand, Spark has the same feature built in. If the planner knows that the source data is partitioned and/or sorted appropriately, it can skip shuffling/sorting it, instead having each executor directly requesting the one file it needs. It's a nice optimization, but it's not game changing. You often end up having to shuffle anyway, as you are joining on a different key, or for performance reason you need more executors than the set amount of partitions, or the shuffle needed to write the data doesn't justify the savings on the readers. Maybe it's better with their additional optimizations? Spark does not do those, mostly.
- texasbigdata 6y ago50% cost reduction though
- enigmo 6y agoHive also has had this optimization for as long as I can remember. As others have noted it's not particularly new or novel, it's just not part of the Beam SDK.
- person_of_color 6y agoCould be done with a bash script...
- rovr138 6y agoAnd paper and pencil What’s your point?
- shaicoleman 6y agoMy off-topic rant: I'd really wish Spotify would focus on improving the core player experience. It has barely seen any improvements in years. * Not overwrite/delete my listening history everytime I switch devices * Allow tabs, or some way to resume what I've been listening to in different contexts * Option to open only one instance, instead of having multiple instances that mess with each other * Playing local files crashes/not working on Linux * Change playback speed, not just for podcasts * Jump back/forward, not just for podcasts * Have some visibility when the song was last played / play count * Liked songs not always appearing in search results * Sorting search results not working * Add basic functionality to the dbus interface (e.g. seeking) * Ability to report songs (e.g. wrong titles/badly split tracks/etc.)
- TheRealSteel 6y agoI have been begging them for years to add a 'resume playlist' feature but they won't do it.
- deleted 6y ago[deleted]
- krakmh 6y agoYou are absolutely right. Even more, people are using it for free using mods like these https://bestforandroid.com/apk/spotify-premium-mod-apk/ https://bestforandroid.com/apk/spotify-premium-mod-apk/
- yaqoob 6y agoYup, well said.
- bootlooped 6y agoAlmost all of these strike me as only benefiting a very small sliver of users, like well under 1%. An infinitesimal portion of Spotify listeners even know what dbus is. How much engineer time is it worth to improve something like that?
- 6y ago
- paulsutter 6y agoLargest data flow job ever? I’m sure Google would beg to differ. At Quantcast we process 50PB every day, and that’s nothing compared to real scale like Google. And merge joins from sorted data? Joins have been done that way since the punched card days on mainframes (and by any scaled data system)
- sbarre 6y agoSurely you read the article before posting, right? From literally the first sentence: > from our largest Dataflow job
- ascar 6y agoSurely you have read the guidlines before posting, right? > Please don't comment on whether someone read an article. "Did you even read the article? It mentions that" can be shortened to "The article mentions that." https://news.ycombinator.com/newsguidelines.html https://news.ycombinator.com/newsguidelines.html The hackernews title and the article title say "the". Critizing this clickbait is more than warranted.
- sbarre 6y agoFair point, I could have worded it better. But I think it's a reach to call this "clickbait". Article titles are shortened all the time, and you can't expect them to have all the context in the title. However, one should reasonably expect people participating in a discussion about an article (particularly when posting criticism) to have actually read it. Complaining about something that is provably false in the first sentence of the article is the bigger sin here, is it not?
- davweb 6y agoIt's largest Dataflow[1] job ever, with a capital D, not "data flow". [1]: https://cloud.google.com/dataflow https://cloud.google.com/dataflow
- enigmo 6y agoMisleading headline, the first sentence is "how Spotify optimized and sped up elements from *our largest Dataflow job*". Surely it's not the largest ever run, even on Dataflow.
- henron 6y agoThis technique of using distributed storage for large joins instead of shuffling between compute nodes also helps make your job robust to spot instance kills. Until disaggregated shuffle services are widely adopted, it can be really handy.
- a254613e 6y agoPerhaps a bit off-topic. But a lot of users (myself included) reported wildly inaccurate data in the spotify wrapped this year with seemingly no explanation, no shared accounts, no re-used passwords, no weird listening history, etc. I wonder if some of the data in the "We worked with the maintainer of these data sets to convert a year’s worth of data to SMB format." step got corrupted or just wrongly converted/lost. I'm not sure how else explain that I have to google artists in my top 10 because I never heard of them.
- philip1209 6y agoI'd be curious how this compares in load to Google's internal applications. I'm also curious what the capacity of Google's infrastructure goes to Google vs. GCE - has combined GCE usage even passed the compute needs of Google internally yet?
- jeffbee 6y agoThe only even remotely concrete information in this post is their input was 1PB, and they typically have 500 bigtable tablet servers. In 2008, Google said they processed 20PB per day through mapreduce jobs. For the last ten years the only thing they've said about the size of their public web index is that it is over 100PB.
- gabagool 6y agoRegarding the title, how do we know this is THE largest dataflow job? The article body itself only makes mention that this is THEIR largest dataflow job. This post doesn't make any quantifiable claims either one could use to support, this is all I found: > "We estimate around a 50% decrease in Dataflow costs this year compared to previous years’ Bigtable-based approach. Additionally, we avoided scaling the Bigtable cluster up two to three times its normal capacity (up to around 1,500 nodes at peak" The official Spotify Engineering Tweet similarly only makes mention that this is Spotify's largest dataflow job ever: https://twitter.com/SpotifyEng/status/1359887825047613442 https://twitter.com/SpotifyEng/status/1359887825047613442. I'm fairly sure a similar accidental unsourced exaggeration was made last year. Maybe the title should be Spotify Optimized Their Largest Dataflow Job Ever For Wrapped 2020?
- sailfast 6y agoFor those that would like to dig more into what SMB is - here's the link to the paper from the article: http://kth.diva-portal.org/smash/get/diva2:1334587/FULLTEXT01.pdf http://kth.diva-portal.org/smash/get/diva2:1334587/FULLTEXT0...
- deleted 6y ago[deleted]
- shoulderfake 6y agoWhatever theyre doing with data means nothing when their client apps are absolute dogshit.
- blt 6y agoThis article fails to make a clean problem statement for the general audience. It jumps right into jargon and names from some framework/library. It reads like it was an internal report from a programmer to their team, and someone decided to make it public with no changes.