7 ms·

Automatic music playlist generation via simulation-based reinforcement learning

55 points by saeedesmaili 3y ago

p1esk 3y ago
“Our recommender system aims at maximizing user satisfaction and, as a proxy, we consider the completion count (and rate) at the session level, i.e., the amount of tracks recommended by the policy that are completed by the user. For this task we first train an agent offline using a non-sequential world model (CWM) as user simulator, and then deploy the agent online to serve recommendations to the users. The outcomes of our user model are primarily consumption-focused and summarize the probability of user-item interactions. Specifically, CWM has been optimized for three targets: completion, skip and listening duration. The reward for the agent is computed as the sum of the probability of completion for each track in an episode. … All policies take as inputs features of the user who made the request and the pool-provided set of tracks with their associated features. The goal of each policy is to select and order a list of tracks from the pool that maximizes expected user satisfaction, which we measure by counting the number of tracks completed by the user.”
- doctorhandshake 3y ago
  Came here to excerpt that same snip and to comment that maybe I’m an outlier but based on my own behavior I’d want to reinforce the model’s propensity to suggest tracks that the user likes/favorites/saves/adds to library (all past names for the same action due to some INSANE UX struggle going on at Spotify, topic for another rant) or adds to a playlist, or plays again outside of the context of a recommendation playlist. I think, for my purposes, ‘plays to completion’ is a weak proxy for whether I liked the track, as I’ll sometimes just put on a recommended playlist and leave the room, or tune out and work, etc.
  - bckr 3y ago
    I’ve thought this exact thing many times. However, I wonder if they do this because they’re not actually optimizing for user satisfaction, but for complete plays, for financial reasons.
    - lukasb 3y ago
      What financial reasons?
      - bckr 3y ago
        Others have speculated on this in the thread, and I concur with them.
    - bowmessage 3y ago
      Interesting thought, but doesn't Spotify get my same monthly membership fee, whether I play 1 track or 1,000? Although, I suppose this line of thinking makes sense for ad-supported streaming, which is likely used by the majority of users.
      - delusional 3y ago
        Financially, I think they count a stream after 30 seconds, and they pay per stream, therefore they'd either want you to skip before 30 seconds (making it free for them) or otherwise spend the maximum time with the song (such that they only pay for 1 stream). The most expensive way for you to listen to spotify is to skip the song every 31 seconds.
        nerdponx 3y ago
        I wonder how this works with "grindcore" bands like Agoraphobic Nosebleed.
        numbp4d 3y ago
        This is not how they pay out subscription $. They do not pay per stream. While the payout algorithm is highly complicated, essentially it is a % of subscription dollars distributed pro-rata by stream count to copyright holders (where a 1 stream is at least 30s of listening).
        delusional 3y ago
        Ah, so consumption is entirely decoupled downstream payment? That's interesting. Thanks for telling me.
  - flutas 3y ago
    > all past names for the same action due to some INSANE UX struggle going on at Spotify, topic for another rant I've felt this exact same sentiment and can't help but wonder if it's them being partially afraid of Apple Music in some way. Since Apple Music launched with "add to library" as their shortcut action and "love" buried a bit.
    - m3kw9 3y ago
      I have a feeling Spotify is trying to make the UI as different to Apple as possible so that the switching cost on the UI will be huge and people will not be comfortable to switch. This is why Spotify will choose some weird UI.
psychphysic 3y ago
Is this Hang the DJ but for your Spotify playlist?
rectang 3y ago
> maximizing user satisfaction I'm never going to be satisfied with Spotify's recommendations until it's easy for me to curate the input to the recommender. As things stand, to prevent Spotify from giving me crap recommendations, I have to keep secrets from it! If I want to listen to Led Zeppelin once but don't want Spotify to think "oh I should recommend classic rock from now until the end of time", then I have to go to a different app (YouTube or whatever). There's a "private session" feature in Spotify which allows me to prevent searches and listens from entering into history, but Spotify really really doesn't want me to use it — it's cumbersome to access and it gets turned off automatically after a few hours. EDIT: The satisfaction achieved by whole-account recommendations determined by algorithm described in this post is going to be crushed by some app which gives me a UI where I can say "give me recommendations based on these N arbitrary things". Not just "this one artist" or "everything in my history". Let me design my own recommended channel.
- halfdan 3y ago
  Add to that having kids at home that sometimes listen to Paw Patrol and you will end up with a daily mix that consists of (in my case): Death Metal, news, random clip from paw patrol, German rap (I only like one artist not that genre in general). I’ve been longing for a “kids mode” for a long time.
  - saeedesmaili 3y ago
    The "exclude from your taste profile" option in playlists might be helpful in your case.
  - Epa095 3y ago
    I gave my kid it's own Spotify account (we have a family subscription), and I installed "island" in android enabling me to install a separate Spotify instance logged into their account. Now I have 2 Spotifys installed in parallel, one for me and one for them.
- robbiemitchell 3y ago
  You can do this in Spotify now by creating a playlist and then playing the station driven from that playlist.
SoftTalker 3y ago
What I want to listed to depends greatly on my mood. Might be classic rock, might be jazz, might be contemporary pop, might be alternative or punk or metal or even classical. Might also depend on who else is in the room or car. If I'm playing music for myself my selections will be different from what I'd play if my wife was listening, because our musical tastes are different. Or a group of friends, etc. I've never found that recommendations or automatic playlists are very good, on any of the services, though my experience is mainly with YouTube music.
- city41 3y ago
  I used to really like Pandora's a lot. So much so that I once emailed them commending them on this and they sent me a t-shirt. If memory serves the difference was that Pandora's associations were human curated.
fredley 3y ago
Just let me exclude songs from recommendations, please. There are songs that are superficially similar to those I enjoy, but that I never want to hear. It's infuriating.
bglazer 3y ago
Maybe I’m misunderstanding something fundamental, but I’m struggling to see the reason for this work. They have a (presumably) supervised model that estimates the likelihood that a user will like a song. Then they train an RL model using the supervised model as an oracle. They note that the optimal result for the RL model is just making the same prediction as the supervised model. What are we doing here folks? Training models to match other models? Why? That’s good enough to get into KDD?
- pizza 3y ago
  Premises: - the offline model data has a non-sequential world model - the offline model is a great starting point but slow to react to realtime changes in user preferences - the in-streaming of new data points makes the reward prediction inherently sequential - hence you could use a cheap fast online model for fast reward prediction updates, that derives from a good starting point The big question is: what do you do to adapt to just one new data point? You're trying to learn theta' := train(old data ++ new data) ≈ train(old data) ++ dTrain(new data) ≈ theta_offline ++ dTheta(new data) If you change the offline model: well it's a big base model and there are always M users with N new data points coming in. Rerouting all training into the same common model theta, dynamically and live, in response to vast amounts of incoming data, while balancing batching-latency overheads or catastrophic forgetting or reversion to the mean preference, would be tricky. If you change the online model: the delta per-user can be cheap to track individually with essentially increment-only/append-only per-user data structures. You can get away with approximating user_theta' ≈ offline_theta + delta(train(new data)) ≈ offline_theta + online_train(new_data). This can be easily parallelized and personalized and zero latency. It's a good achievement.
navels 3y ago
Any open-source recommenders out there that people like? Could be easy enough to hook one up to spotify via their api.
scrps 3y ago
Spotify's rating and recommendation system leaves quite a bit to be desired. They need granularity, and they need to explain the algorithm to some extent. If I like a track what exactly am I liking? The genre? The artist? Only specifically that song? Some maths that approximate a certain properties in the track? Some combination? How does liking song X influence Spotify's recommendation system on average? I have experienced the same as other posters WRT listening to a genre for a session only to have Spotify fixate on that genre until I've skewed the average in another direction. Small hack I've found if you make a playlist with a large and diverse sample size you can nudge their playlist enhancer to stay with the mood you are going for. On granularity, I would want to be able to say: Let x = track, genre, artist I don't like X, never play it. Exclude X just for this listening session. Skew toward more X I'd also imagine this would produce much richer data for future recommendation systems. If Spotify were to train on the data I've generated on their platform it would end introducing bias by the nature of me working around their systems and still not getting what I want, granted I may be an outlier. Edit: Formatting & typo
BonoboIO 3y ago
I‘m shocked how unbelievable bad the recommendations from Spotify are. They never were any good, but they also don’t improve, despite how often they brag about some new recommendation engine, feature or improvement on their engineering blog. A multi billion dollar company that fails so hard at this. The best recommendations I got were from Pandora, unbelievable good and on point and that was 15 years ago. I can not use them anymore in Europe so I m stuck with Spotify. Maybe this GPT4 based project will give me better recommendations. https://github.com/Railly/spotigen-chat-gpt-plugin https://github.com/Railly/spotigen-chat-gpt-plugin
morkalork 3y ago
I tried out the new "Personal DJ" mix on Spotify last week and bounced when the way over the top AI voice stated my username as a part of the most cringe inducing intro. The feature was awful and there wasn't even any notion of fast-forwarding through the bad parts. 0/10. Will not try again.
- neura 3y ago
  I agree so much with this. I was excited for the feature until the first 10 seconds. I don't want a fake DJ "talking to me" or attempting to emulate a radio DJ. It's not like all I have to look at is a dial with numbers on it. I can see all the details of what you're playing, thanks. Shut up and just let me see if I like what you think I'll like, enough that it makes me feel that your recommendations won't be immediate rejections. Or just have the dumb AI talk to me about it, so I know that I absolutely do not want to use the feature. sigh 0/10 indeed
MrYellowP 3y ago
That makes me a doomsayer, basically, but I consider this to be evidence that every single aspect of our lives will eventually be somehow controlled by AI. Sure, you can claim that there's always the choice of not doing what the AI suggests, but that'll not apply to the kids growing up in a world where they're being breast-fed literally everything, including their playlist. And their reason will be "because it's good", which is related to "because it's fun". When AI is perfectly pleasing, people will be fully enslaved by fun. Fun already seems to be the driving factor for most people, so I'm not sure how to find anything positive in any of this. Disclaimer: At the peak, I had around 300 gigs worth of language models stored on my machine. Now I'm down to around 120. :p

55 points by saeedesmaili 3y ago

“Our recommender system aims at maximizing user satisfaction and, as a proxy, we consider the completion count (and rate) at the session level, i.e., the amount of tracks recommended by the policy that are completed by the user. For this task we first train an agent offline using a non-sequential world model (CWM) as user simulator, and then deploy the agent online to serve recommendations to the users. The outcomes of our user model are primarily consumption-focused and summarize the probability of user-item interactions. Specifically, CWM has been optimized for three targets: completion, skip and listening duration. The reward for the agent is computed as the sum of the probability of completion for each track in an episode. … All policies take as inputs features of the user who made the request and the pool-provided set of tracks with their associated features. The goal of each policy is to select and order a list of tracks from the pool that maximizes expected user satisfaction, which we measure by counting the number of tracks completed by the user.”

doctorhandshake 3y ago
Came here to excerpt that same snip and to comment that maybe I’m an outlier but based on my own behavior I’d want to reinforce the model’s propensity to suggest tracks that the user likes/favorites/saves/adds to library (all past names for the same action due to some INSANE UX struggle going on at Spotify, topic for another rant) or adds to a playlist, or plays again outside of the context of a recommendation playlist. I think, for my purposes, ‘plays to completion’ is a weak proxy for whether I liked the track, as I’ll sometimes just put on a recommended playlist and leave the room, or tune out and work, etc.
- bckr 3y ago
  I’ve thought this exact thing many times. However, I wonder if they do this because they’re not actually optimizing for user satisfaction, but for complete plays, for financial reasons.
  - lukasb 3y ago
    What financial reasons?
    - bckr 3y ago
      Others have speculated on this in the thread, and I concur with them.
  - bowmessage 3y ago
    Interesting thought, but doesn't Spotify get my same monthly membership fee, whether I play 1 track or 1,000? Although, I suppose this line of thinking makes sense for ad-supported streaming, which is likely used by the majority of users.
    - delusional 3y ago
      Financially, I think they count a stream after 30 seconds, and they pay per stream, therefore they'd either want you to skip before 30 seconds (making it free for them) or otherwise spend the maximum time with the song (such that they only pay for 1 stream). The most expensive way for you to listen to spotify is to skip the song every 31 seconds.
      - nerdponx 3y ago
        I wonder how this works with "grindcore" bands like Agoraphobic Nosebleed.
      - numbp4d 3y ago
        This is not how they pay out subscription $. They do not pay per stream. While the payout algorithm is highly complicated, essentially it is a % of subscription dollars distributed pro-rata by stream count to copyright holders (where a 1 stream is at least 30s of listening).
        delusional 3y ago
        Ah, so consumption is entirely decoupled downstream payment? That's interesting. Thanks for telling me.
- flutas 3y ago
  > all past names for the same action due to some INSANE UX struggle going on at Spotify, topic for another rant I've felt this exact same sentiment and can't help but wonder if it's them being partially afraid of Apple Music in some way. Since Apple Music launched with "add to library" as their shortcut action and "love" buried a bit.
  - m3kw9 3y ago
    I have a feeling Spotify is trying to make the UI as different to Apple as possible so that the switching cost on the UI will be huge and people will not be comfortable to switch. This is why Spotify will choose some weird UI.

psychphysic 3y ago

Is this Hang the DJ but for your Spotify playlist?

rectang 3y ago

> maximizing user satisfaction I'm never going to be satisfied with Spotify's recommendations until it's easy for me to curate the input to the recommender. As things stand, to prevent Spotify from giving me crap recommendations, I have to keep secrets from it! If I want to listen to Led Zeppelin once but don't want Spotify to think "oh I should recommend classic rock from now until the end of time", then I have to go to a different app (YouTube or whatever). There's a "private session" feature in Spotify which allows me to prevent searches and listens from entering into history, but Spotify really really doesn't want me to use it — it's cumbersome to access and it gets turned off automatically after a few hours. EDIT: The satisfaction achieved by whole-account recommendations determined by algorithm described in this post is going to be crushed by some app which gives me a UI where I can say "give me recommendations based on these N arbitrary things". Not just "this one artist" or "everything in my history". Let me design my own recommended channel.

halfdan 3y ago
Add to that having kids at home that sometimes listen to Paw Patrol and you will end up with a daily mix that consists of (in my case): Death Metal, news, random clip from paw patrol, German rap (I only like one artist not that genre in general). I’ve been longing for a “kids mode” for a long time.
- saeedesmaili 3y ago
  The "exclude from your taste profile" option in playlists might be helpful in your case.
- Epa095 3y ago
  I gave my kid it's own Spotify account (we have a family subscription), and I installed "island" in android enabling me to install a separate Spotify instance logged into their account. Now I have 2 Spotifys installed in parallel, one for me and one for them.
robbiemitchell 3y ago
You can do this in Spotify now by creating a playlist and then playing the station driven from that playlist.

SoftTalker 3y ago

What I want to listed to depends greatly on my mood. Might be classic rock, might be jazz, might be contemporary pop, might be alternative or punk or metal or even classical. Might also depend on who else is in the room or car. If I'm playing music for myself my selections will be different from what I'd play if my wife was listening, because our musical tastes are different. Or a group of friends, etc. I've never found that recommendations or automatic playlists are very good, on any of the services, though my experience is mainly with YouTube music.

city41 3y ago
I used to really like Pandora's a lot. So much so that I once emailed them commending them on this and they sent me a t-shirt. If memory serves the difference was that Pandora's associations were human curated.

fredley 3y ago

Just let me exclude songs from recommendations, please. There are songs that are superficially similar to those I enjoy, but that I never want to hear. It's infuriating.

bglazer 3y ago

Maybe I’m misunderstanding something fundamental, but I’m struggling to see the reason for this work. They have a (presumably) supervised model that estimates the likelihood that a user will like a song. Then they train an RL model using the supervised model as an oracle. They note that the optimal result for the RL model is just making the same prediction as the supervised model. What are we doing here folks? Training models to match other models? Why? That’s good enough to get into KDD?

pizza 3y ago
Premises: - the offline model data has a non-sequential world model - the offline model is a great starting point but slow to react to realtime changes in user preferences - the in-streaming of new data points makes the reward prediction inherently sequential - hence you could use a cheap fast online model for fast reward prediction updates, that derives from a good starting point The big question is: what do you do to adapt to just one new data point? You're trying to learn theta' := train(old data ++ new data) ≈ train(old data) ++ dTrain(new data) ≈ theta_offline ++ dTheta(new data) If you change the offline model: well it's a big base model and there are always M users with N new data points coming in. Rerouting all training into the same common model theta, dynamically and live, in response to vast amounts of incoming data, while balancing batching-latency overheads or catastrophic forgetting or reversion to the mean preference, would be tricky. If you change the online model: the delta per-user can be cheap to track individually with essentially increment-only/append-only per-user data structures. You can get away with approximating user_theta' ≈ offline_theta + delta(train(new data)) ≈ offline_theta + online_train(new_data). This can be easily parallelized and personalized and zero latency. It's a good achievement.

navels 3y ago

Any open-source recommenders out there that people like? Could be easy enough to hook one up to spotify via their api.

scrps 3y ago

Spotify's rating and recommendation system leaves quite a bit to be desired. They need granularity, and they need to explain the algorithm to some extent. If I like a track what exactly am I liking? The genre? The artist? Only specifically that song? Some maths that approximate a certain properties in the track? Some combination? How does liking song X influence Spotify's recommendation system on average? I have experienced the same as other posters WRT listening to a genre for a session only to have Spotify fixate on that genre until I've skewed the average in another direction. Small hack I've found if you make a playlist with a large and diverse sample size you can nudge their playlist enhancer to stay with the mood you are going for. On granularity, I would want to be able to say: Let x = track, genre, artist I don't like X, never play it. Exclude X just for this listening session. Skew toward more X I'd also imagine this would produce much richer data for future recommendation systems. If Spotify were to train on the data I've generated on their platform it would end introducing bias by the nature of me working around their systems and still not getting what I want, granted I may be an outlier. Edit: Formatting & typo

BonoboIO 3y ago

I‘m shocked how unbelievable bad the recommendations from Spotify are. They never were any good, but they also don’t improve, despite how often they brag about some new recommendation engine, feature or improvement on their engineering blog. A multi billion dollar company that fails so hard at this. The best recommendations I got were from Pandora, unbelievable good and on point and that was 15 years ago. I can not use them anymore in Europe so I m stuck with Spotify. Maybe this GPT4 based project will give me better recommendations. https://github.com/Railly/spotigen-chat-gpt-plugin https://github.com/Railly/spotigen-chat-gpt-plugin

morkalork 3y ago

I tried out the new "Personal DJ" mix on Spotify last week and bounced when the way over the top AI voice stated my username as a part of the most cringe inducing intro. The feature was awful and there wasn't even any notion of fast-forwarding through the bad parts. 0/10. Will not try again.

neura 3y ago
I agree so much with this. I was excited for the feature until the first 10 seconds. I don't want a fake DJ "talking to me" or attempting to emulate a radio DJ. It's not like all I have to look at is a dial with numbers on it. I can see all the details of what you're playing, thanks. Shut up and just let me see if I like what you think I'll like, enough that it makes me feel that your recommendations won't be immediate rejections. Or just have the dumb AI talk to me about it, so I know that I absolutely do not want to use the feature. sigh 0/10 indeed

MrYellowP 3y ago

That makes me a doomsayer, basically, but I consider this to be evidence that every single aspect of our lives will eventually be somehow controlled by AI. Sure, you can claim that there's always the choice of not doing what the AI suggests, but that'll not apply to the kids growing up in a world where they're being breast-fed literally everything, including their playlist. And their reason will be "because it's good", which is related to "because it's fun". When AI is perfectly pleasing, people will be fully enslaved by fun. Fun already seems to be the driving factor for most people, so I'm not sure how to find anything positive in any of this. Disclaimer: At the peak, I had around 300 gigs worth of language models stored on my machine. Now I'm down to around 120. :p