11 ms·

Incident report: Spotify outage on March 8

120 points by charlieegan3 5y ago

hericium 5y ago
> On March 8, Google Cloud Traffic Director experienced an outage. This in coordination with a bug in a client (gRPC) library caused the Spotify outage that affected many of our users: if you were logged out of a Spotify app, you were unable to log back in.
- superjan 5y ago
  But they don’t explain why I was logged out. I did not log out myself. Or is this “logged out” in the sense that I did not have their app active at the time of the disruption?
  - detinho 5y ago
    Maybe some token expired and could not be renewed. They also mentioned a bug on a client grpc library.
    - RL_Quine 5y ago
      Everybody was instantly logged out, which doesn’t make sense if their backend just went down, as the app normally works fine offline.
      - addandsubtract 5y ago
        Well, not everybody, since I was logged in all week and didn't experience any downtime.
  - benatkin 5y ago
    Logging out on a 401 is pretty common in authentication, because it gives the user an opportunity to sign in again. Maybe the problem resulted in a 401 being returned, when it would have been better to return a 500.
    - sususu 5y ago
      In the web at least it was returning 500, I guess that the desktop/mobile apps make some check in intervals. When the backend returned 500 the app thought that the current login could not be authenticated and logged out everybody
      - benatkin 5y ago
        I think some will aggressively log out even on 500s. If you log out 500s that's not the same as logging out failed network requests. The server has to actually send a response, so while it will erroneously log out sometimes, in a reliable api, or one that sends errors in 200s but is otherwise reliable, it won't be so often that a whole lot of users give up. Plus some API middlewares will default to hiding error response codes for security and always send 500 no matter what the upstream response code (this assumes the error codes aren't needed by the devs because they can look at the logs). I've seen logging out on failed network requests, but that is highly annoying and frequently occurs and typically just a coding mistake.
jmacd 5y ago
Just here to share that if you haven't tried Apple Music for its lossless and Dolby Atmos sound quality then you really, really, should. Apple Music does not have a great UI but it is definitely worth the trade off. Tidal sounds great, but they make you pay extra for lossless and have a smaller catalogue.
- flats 5y ago
  Apple Music’s best feature for me is its real focus on building up your own library. On top of that, it integrates seamlessly with iTunes in the Cloud. Using the same app, I can stream anything from my iTunes library, which includes a decent amount of music not available on any DSP.
- ge96 5y ago
  > lossless and Dolby Atmos sound quality General headphone recommendation? Best I have is an MDR-7506 but I don't think it's meant for listening rather producing.
  - lelandfe 5y ago
    Depends on price range. DT 770 PRO's would be a nice step up and serve both purposes pretty well. They're over-ear like your MDR's. You can usually find them on sale for like $140-160 USD Fairly neutral response (dunno if you'd prefer something bassier for listening) and very comfortable
    - ge96 5y ago
      Yeah I don't really have any specific demands other than it is meant for music... I bought these ATH-MX-20/40X from reviews and apparently they're monitoring headphones?... so yeah. Those are definitely flat. Thanks I'll put those on a list. Definitely one of those things have to try on in person to buy but recommendations help too. Edit: not too bassy actually, neutral would be nice like works for almost anything.
  - Simon_O_Rourke 5y ago
    With some of the banal EDM that's out there now, some retro 1980s walkman headphones would do.
wizwit999 5y ago
So the bigger news here is GCP had a global outage. How'd that happen? It mentions a pipeline deployment introduced a bug (https://status.cloud.google.com/incidents/LuGcJVjNTeC5Sb9pSJ9o https://status.cloud.google.com/incidents/LuGcJVjNTeC5Sb9pSJ...) but why weren't they doing staged deployments to region(s) at a time?
- politelemon 5y ago
  This is now 'par for the course' with GCP. A recent example, Firefox's Jan 2022 outage was due to an unannounced change deployed by GCP to their load balancers. https://hacks.mozilla.org/2022/02/retrospective-and-technical-details-on-the-recent-firefox-outage/ https://hacks.mozilla.org/2022/02/retrospective-and-technica... I've read several GCP related stories over the years so my opinion of them has fallen quite a bit. For some reason GCP does not get the flak they deserve, and their role in outages tend to be downplayed, while AWS and Azure are scrutinized and raked over the coals for smaller incidents.
  - toomuchtodo 5y ago
    They don’t get flak because most folks are in AWS or Azure. You notice when AWS or Azure hiccups because of this. GCP, not so much (besides Spotify and Snap, although I believe Snap’s 5yr $2B GCP commit deal recently ended). https://info.flexera.com/CM-REPORT-State-of-the-Cloud https://info.flexera.com/CM-REPORT-State-of-the-Cloud https://thenewstack.io/long-running-study-finds-uptake-of-private-clouds-from-azure-aws-and-gcp/ https://thenewstack.io/long-running-study-finds-uptake-of-pr...
    - politelemon 5y ago
      Ah, good point about higher usage therefore flak, I hadn't considered that. Thanks.
      - MetricExpansion 5y ago
        Bjarne Stroustrup’s classic quote “There are only two kinds of languages: the ones people complain about and the ones nobody uses” certainly applies here.
ipython 5y ago
Hey, for once it wasn’t DNS!
- Crosseye_Jack 5y ago
  There is always an exception to the rule!
- bluepizza 5y ago
  Traffic Director replaces DNS in GCP
toastal 5y ago
Just a reminder that downloaded audio files work offline, can have very high or lossless quality not restricted by bandwidth, have a concept of ‘ownership’ with no DRM, don’t have ads or tries to sell your data (though voluntarily contributing to ListenBrainz is admirable), don’t require a monthly fee, and if bought legally gives a bigger slice of the money pie to artists. Also podcasts are just RSS audio.
- myself248 5y ago
  Got a favorite source for same? I'm bewildered by too many options and not enough comparisons.
  - jmcphers 5y ago
    I get my music on Bandcamp, if the artist sells it there. They compensate artists far better than Spotify[1], and offer lossless, DRM-free downloads. Some major labels don't sell on Bandcamp; for those I often buy the MP3s on Amazon. I don't love Amazon, but they do sell high quality, DRM-free music now. [1] https://pitchfork.com/thepitch/how-much-more-money-artists-earn-from-bandcamp-compared-to-spotify-apple-music-youtube/ https://pitchfork.com/thepitch/how-much-more-money-artists-e...
    - DougMellon 5y ago
      I would also recommend checking Qobuz which is where I find most of my larger label FLAC files. https://www.qobuz.com/us-en/shop https://www.qobuz.com/us-en/shop
- hk__2 5y ago
  According to Last.fm I’ve listened to 156k songs from 16k artists (35k albums) over a little less than 10 years. Last time I bought music it was on iTunes and the lowest price was €0.79 per song. That would be a total of €123k. Over the same period, I gave exactly €1200.93 to Spotify [1]. [1] note: it’s a pure coincidence that this is ~ €10 * 12 months * 10 years; the price has not always been the same.
dotnet00 5y ago
Over the past year I've taken to just buying my music and using plex/plexamp for the streaming convenience. I'd have a hard time going back to other streaming services after getting used to the convenience of having control over the entire chain. Buying music also serves as a nice way to spend digital credit from Amazon. It ends up costing me about as much as streaming.
- belval 5y ago
  > getting used to the convenience of having control over the entire chain What convenience is that? How do you consider that your setup is in anyway more convenient that opening the Spotify app and listening to basically anything? If you are into very niche music or something maybe, but it's pretty hard to beat just clicking "play". Now if you said you simply pirate the songs I could see the convenience of saving money, but in your case you buy the music anyway?
  - justinhj 5y ago
    Because of outages and or censorship of content
    - alar44 5y ago
      I've never experienced a Spotify outage in the 5 years I've been using it and I use it daily. Not once.
    - belval 5y ago
      I probably listen to music on Spotify ~5h a day while I work and I haven't had any downtime ever since I subscribed. As for censorship there was that thing with Joe Rogan, but as far as I know Spotify sided with him so I don't see where the censorship is?
      - justinhj 5y ago
        You are in a thread about a Spotify outage. The White House press secretary called for censorship of Joe Rogan and I expect it will happen soon as they find a reason. Personally I cant stand the user interface and many times I am away from a network connection and something I downloaded is not available or Spotify is just stalled trying to connect. Its probably just not a good fit for my use case
dastbe 5y ago
Does anyone have more context on why “unable to retrieve their xDS configurations.” led to ” Traffic Director-managed clients deprogrammed as the configuration was removed” ? For example, Envoy will keep its configuration forever unless 1) the configuration has a ttl or 2) the server itself sent back empty results. Was it the case that TD config uses ttls, the server was actually sending back empty results (frightening), or is this a behavior specific to grpc?
- jhgg 5y ago
  We suffered from similar issue to Spotify. Traffic Director pushed a mostly empty configuration to envoy. Very frightening indeed. It continued to push that invalid configuration over the course of the outage.
w4rh4wk5 5y ago
The main problem of the outage was that users were logged out and thus unable to play what songs they had downloaded / cached.
- trutannus 5y ago
  That and the accounts appearing locked, with password reset links not working. For the first few minutes of the outage, I thought I my accounts were what was compromised. Caused genuine concern for a while there since it appeared I had lost control over my Spotify account, and there's credit card data in there.
  - carstenhag 5y ago
    My better half thought that her account was hacked. So yeah, the logout thing is even worse than not being able to log in. Doesn't help that the error message they have is not very useful (there should be a "yeah it's our fault" message)
    - trutannus 5y ago
      For sure. The "invalid password reset link" one really caused worry. It took until I hit the support page to figure out there was an account issue. Probably should have issued an email notice so I could have seen it when I tried to reset my password.
- ngngngng 5y ago
  Yeah I was surprised that wasn't clearly stated in the incident report. They made it sound like I wasn't logged in when the outage occurred when I'm always logged in across all my devices.
dijonman2 5y ago
I’d love for outages to not be newsworthy.
tjpnz 5y ago
Has anyone else noticed that the quality of the Android app has really gone downhill lately? There's often a lag of a few seconds now when switching between tracks and I've noticed an issue recently where the song title/cover art doesn't always update to reflect the currently playing track. Sometimes the play/pause/skip buttons become completely unresponsive. This doesn't appear to be an issue on my end, I haven't had problems with any of the other streaming apps I use and I've confirmed the same set of issues after a fresh install of the app. For the time being it's more of a nuisance but I would be prepared to consider alternatives if things don't start improving soon.
- vorpalhex 5y ago
  It's bad. Unfortunately last time I tried the main competitors, TIDAL and Deezer, their app was somehow worse.
  - peterburkimsher 5y ago
    It surprises many colleagues and friends that I still use an iPod Video from 2005-6. Life without adverts, week-long battery life, instant track skipping, ratings, playlists (and OTG playlists) make the music experience better than any streaming services, which are really just radio replacements. Embedded systems might not be "cool" any more, but they're repairable, and have other features too (iPod notes, games, contacts, calendar), and a thriving mod firmware scene (iPodLinux and Rockbox). There is an alternative, if you'd like to come to the hardware side :)
    - hiptobecubic 5y ago
      You can also do this with your phone. You don't have to stream everything. Just add your files and use an app that can play music locally.
      - BuildTheRobots 5y ago
        Or something like plex/embly/jellyfin and I can stream any of my home media to my phone/tablet that won't let me have removable storage.
daguava 5y ago
I recently went to pay for a music service and wanted to give spotify a chance, but they have a huge information leak somewhere on when new accounts are created. On the day of my free trial ending I received the first and only phishing email I've ever had from spotify telling me my trial was ending and it was time to swipe. Up until then I had been happy with the service, but then noticed I was about to get owned.... Down to the day of expiration... I emailed their support about how this was sketchy and they needed to do better information masking and got told their team did not consider any of this a vulnerabilty. With this in tow, it really cast some shade on their serious engineering abilities for me.

120 points by charlieegan3 5y ago

> On March 8, Google Cloud Traffic Director experienced an outage. This in coordination with a bug in a client (gRPC) library caused the Spotify outage that affected many of our users: if you were logged out of a Spotify app, you were unable to log back in.

superjan 5y ago
But they don’t explain why I was logged out. I did not log out myself. Or is this “logged out” in the sense that I did not have their app active at the time of the disruption?
- detinho 5y ago
  Maybe some token expired and could not be renewed. They also mentioned a bug on a client grpc library.
  - RL_Quine 5y ago
    Everybody was instantly logged out, which doesn’t make sense if their backend just went down, as the app normally works fine offline.
    - addandsubtract 5y ago
      Well, not everybody, since I was logged in all week and didn't experience any downtime.
- benatkin 5y ago
  Logging out on a 401 is pretty common in authentication, because it gives the user an opportunity to sign in again. Maybe the problem resulted in a 401 being returned, when it would have been better to return a 500.
  - sususu 5y ago
    In the web at least it was returning 500, I guess that the desktop/mobile apps make some check in intervals. When the backend returned 500 the app thought that the current login could not be authenticated and logged out everybody
    - benatkin 5y ago
      I think some will aggressively log out even on 500s. If you log out 500s that's not the same as logging out failed network requests. The server has to actually send a response, so while it will erroneously log out sometimes, in a reliable api, or one that sends errors in 200s but is otherwise reliable, it won't be so often that a whole lot of users give up. Plus some API middlewares will default to hiding error response codes for security and always send 500 no matter what the upstream response code (this assumes the error codes aren't needed by the devs because they can look at the logs). I've seen logging out on failed network requests, but that is highly annoying and frequently occurs and typically just a coding mistake.

jmacd 5y ago

Just here to share that if you haven't tried Apple Music for its lossless and Dolby Atmos sound quality then you really, really, should. Apple Music does not have a great UI but it is definitely worth the trade off. Tidal sounds great, but they make you pay extra for lossless and have a smaller catalogue.

flats 5y ago
Apple Music’s best feature for me is its real focus on building up your own library. On top of that, it integrates seamlessly with iTunes in the Cloud. Using the same app, I can stream anything from my iTunes library, which includes a decent amount of music not available on any DSP.
ge96 5y ago
> lossless and Dolby Atmos sound quality General headphone recommendation? Best I have is an MDR-7506 but I don't think it's meant for listening rather producing.
- lelandfe 5y ago
  Depends on price range. DT 770 PRO's would be a nice step up and serve both purposes pretty well. They're over-ear like your MDR's. You can usually find them on sale for like $140-160 USD Fairly neutral response (dunno if you'd prefer something bassier for listening) and very comfortable
  - ge96 5y ago
    Yeah I don't really have any specific demands other than it is meant for music... I bought these ATH-MX-20/40X from reviews and apparently they're monitoring headphones?... so yeah. Those are definitely flat. Thanks I'll put those on a list. Definitely one of those things have to try on in person to buy but recommendations help too. Edit: not too bassy actually, neutral would be nice like works for almost anything.
- Simon_O_Rourke 5y ago
  With some of the banal EDM that's out there now, some retro 1980s walkman headphones would do.

wizwit999 5y ago

So the bigger news here is GCP had a global outage. How'd that happen? It mentions a pipeline deployment introduced a bug (https://status.cloud.google.com/incidents/LuGcJVjNTeC5Sb9pSJ9o https://status.cloud.google.com/incidents/LuGcJVjNTeC5Sb9pSJ...) but why weren't they doing staged deployments to region(s) at a time?

politelemon 5y ago
This is now 'par for the course' with GCP. A recent example, Firefox's Jan 2022 outage was due to an unannounced change deployed by GCP to their load balancers. https://hacks.mozilla.org/2022/02/retrospective-and-technical-details-on-the-recent-firefox-outage/ https://hacks.mozilla.org/2022/02/retrospective-and-technica... I've read several GCP related stories over the years so my opinion of them has fallen quite a bit. For some reason GCP does not get the flak they deserve, and their role in outages tend to be downplayed, while AWS and Azure are scrutinized and raked over the coals for smaller incidents.
- toomuchtodo 5y ago
  They don’t get flak because most folks are in AWS or Azure. You notice when AWS or Azure hiccups because of this. GCP, not so much (besides Spotify and Snap, although I believe Snap’s 5yr $2B GCP commit deal recently ended). https://info.flexera.com/CM-REPORT-State-of-the-Cloud https://info.flexera.com/CM-REPORT-State-of-the-Cloud https://thenewstack.io/long-running-study-finds-uptake-of-private-clouds-from-azure-aws-and-gcp/ https://thenewstack.io/long-running-study-finds-uptake-of-pr...
  - politelemon 5y ago
    Ah, good point about higher usage therefore flak, I hadn't considered that. Thanks.
    - MetricExpansion 5y ago
      Bjarne Stroustrup’s classic quote “There are only two kinds of languages: the ones people complain about and the ones nobody uses” certainly applies here.

ipython 5y ago

Hey, for once it wasn’t DNS!

Crosseye_Jack 5y ago
There is always an exception to the rule!
bluepizza 5y ago
Traffic Director replaces DNS in GCP

toastal 5y ago

Just a reminder that downloaded audio files work offline, can have very high or lossless quality not restricted by bandwidth, have a concept of ‘ownership’ with no DRM, don’t have ads or tries to sell your data (though voluntarily contributing to ListenBrainz is admirable), don’t require a monthly fee, and if bought legally gives a bigger slice of the money pie to artists. Also podcasts are just RSS audio.

myself248 5y ago
Got a favorite source for same? I'm bewildered by too many options and not enough comparisons.
- jmcphers 5y ago
  I get my music on Bandcamp, if the artist sells it there. They compensate artists far better than Spotify[1], and offer lossless, DRM-free downloads. Some major labels don't sell on Bandcamp; for those I often buy the MP3s on Amazon. I don't love Amazon, but they do sell high quality, DRM-free music now. [1] https://pitchfork.com/thepitch/how-much-more-money-artists-earn-from-bandcamp-compared-to-spotify-apple-music-youtube/ https://pitchfork.com/thepitch/how-much-more-money-artists-e...
  - DougMellon 5y ago
    I would also recommend checking Qobuz which is where I find most of my larger label FLAC files. https://www.qobuz.com/us-en/shop https://www.qobuz.com/us-en/shop
hk__2 5y ago
According to Last.fm I’ve listened to 156k songs from 16k artists (35k albums) over a little less than 10 years. Last time I bought music it was on iTunes and the lowest price was €0.79 per song. That would be a total of €123k. Over the same period, I gave exactly €1200.93 to Spotify [1]. [1] note: it’s a pure coincidence that this is ~ €10 * 12 months * 10 years; the price has not always been the same.

dotnet00 5y ago

Over the past year I've taken to just buying my music and using plex/plexamp for the streaming convenience. I'd have a hard time going back to other streaming services after getting used to the convenience of having control over the entire chain. Buying music also serves as a nice way to spend digital credit from Amazon. It ends up costing me about as much as streaming.

belval 5y ago
> getting used to the convenience of having control over the entire chain What convenience is that? How do you consider that your setup is in anyway more convenient that opening the Spotify app and listening to basically anything? If you are into very niche music or something maybe, but it's pretty hard to beat just clicking "play". Now if you said you simply pirate the songs I could see the convenience of saving money, but in your case you buy the music anyway?
- justinhj 5y ago
  Because of outages and or censorship of content
  - alar44 5y ago
    I've never experienced a Spotify outage in the 5 years I've been using it and I use it daily. Not once.
  - belval 5y ago
    I probably listen to music on Spotify ~5h a day while I work and I haven't had any downtime ever since I subscribed. As for censorship there was that thing with Joe Rogan, but as far as I know Spotify sided with him so I don't see where the censorship is?
    - justinhj 5y ago
      You are in a thread about a Spotify outage. The White House press secretary called for censorship of Joe Rogan and I expect it will happen soon as they find a reason. Personally I cant stand the user interface and many times I am away from a network connection and something I downloaded is not available or Spotify is just stalled trying to connect. Its probably just not a good fit for my use case

dastbe 5y ago

Does anyone have more context on why “unable to retrieve their xDS configurations.” led to ” Traffic Director-managed clients deprogrammed as the configuration was removed” ? For example, Envoy will keep its configuration forever unless 1) the configuration has a ttl or 2) the server itself sent back empty results. Was it the case that TD config uses ttls, the server was actually sending back empty results (frightening), or is this a behavior specific to grpc?

jhgg 5y ago
We suffered from similar issue to Spotify. Traffic Director pushed a mostly empty configuration to envoy. Very frightening indeed. It continued to push that invalid configuration over the course of the outage.

w4rh4wk5 5y ago

The main problem of the outage was that users were logged out and thus unable to play what songs they had downloaded / cached.

trutannus 5y ago
That and the accounts appearing locked, with password reset links not working. For the first few minutes of the outage, I thought I my accounts were what was compromised. Caused genuine concern for a while there since it appeared I had lost control over my Spotify account, and there's credit card data in there.
- carstenhag 5y ago
  My better half thought that her account was hacked. So yeah, the logout thing is even worse than not being able to log in. Doesn't help that the error message they have is not very useful (there should be a "yeah it's our fault" message)
  - trutannus 5y ago
    For sure. The "invalid password reset link" one really caused worry. It took until I hit the support page to figure out there was an account issue. Probably should have issued an email notice so I could have seen it when I tried to reset my password.
ngngngng 5y ago
Yeah I was surprised that wasn't clearly stated in the incident report. They made it sound like I wasn't logged in when the outage occurred when I'm always logged in across all my devices.

dijonman2 5y ago

I’d love for outages to not be newsworthy.

tjpnz 5y ago

Has anyone else noticed that the quality of the Android app has really gone downhill lately? There's often a lag of a few seconds now when switching between tracks and I've noticed an issue recently where the song title/cover art doesn't always update to reflect the currently playing track. Sometimes the play/pause/skip buttons become completely unresponsive. This doesn't appear to be an issue on my end, I haven't had problems with any of the other streaming apps I use and I've confirmed the same set of issues after a fresh install of the app. For the time being it's more of a nuisance but I would be prepared to consider alternatives if things don't start improving soon.

vorpalhex 5y ago
It's bad. Unfortunately last time I tried the main competitors, TIDAL and Deezer, their app was somehow worse.
- peterburkimsher 5y ago
  It surprises many colleagues and friends that I still use an iPod Video from 2005-6. Life without adverts, week-long battery life, instant track skipping, ratings, playlists (and OTG playlists) make the music experience better than any streaming services, which are really just radio replacements. Embedded systems might not be "cool" any more, but they're repairable, and have other features too (iPod notes, games, contacts, calendar), and a thriving mod firmware scene (iPodLinux and Rockbox). There is an alternative, if you'd like to come to the hardware side :)
  - hiptobecubic 5y ago
    You can also do this with your phone. You don't have to stream everything. Just add your files and use an app that can play music locally.
    - BuildTheRobots 5y ago
      Or something like plex/embly/jellyfin and I can stream any of my home media to my phone/tablet that won't let me have removable storage.

daguava 5y ago

I recently went to pay for a music service and wanted to give spotify a chance, but they have a huge information leak somewhere on when new accounts are created. On the day of my free trial ending I received the first and only phishing email I've ever had from spotify telling me my trial was ending and it was time to swipe. Up until then I had been happy with the service, but then noticed I was about to get owned.... Down to the day of expiration... I emailed their support about how this was sketchy and they needed to do better information masking and got told their team did not consider any of this a vulnerabilty. With this in tow, it really cast some shade on their serious engineering abilities for me.