4 ms·

Spotify's Kafka-Based Event Delivery System

122 points by vgt 11y ago

kinofcain 11y ago
Interesting to look at the chart of event volumes at the end. Did Spotify just accidentally reveal the effect Apple Music has had on their growth rate? Steep drop around mid-2015, some recovery since then. Hard to glean anything concrete, but interesting nonetheless.
- brazzledazzle 11y ago
  I would avoid reading into that since the text surrounding that chart indicate that the system was struggling with the load. That drop could have simply been them prioritizing and removing certain event sources in the clients while they figured out how to fix things.
serialpreneur 11y ago
Interesting blog post. I don't have a good idea of the requirements at Spotify, but looks to me that using a streaming system like Storm or Spark Streaming would solve the 30 min event delivery delay they are experiencing w/ unstructured text -> Avro conversion. The latency for delivery would go down to sub-second levels.
- vgt 11y ago
  Stay tuned for Parts 2 + 3 :)
- raphar 11y ago
  Also the article doesn't say WHY they didn't use Kafka to persist the events. Kafka it's designed to do that. Once persisted the consumers can just read kafka data and send them to hadoop, with less latency. Or you can plug storm o spark in as you said and do the analisys there real time. Or both. I'm just intrigued why.
  - RickHull 11y ago
    They do talk about this: > When the system was built, one of the missing features from Kafka 0.7 was the ability of the Kafka Broker cluster to behave as a reliable persistent storage. This influenced a major design decision to not keep persistent state between the producer of data, Kafka Syslog Producer, and Hadoop. An event is considered reliably persisted only when it gets written to a file on HDFS.
shockzzz 11y ago
Nice!
pheeney 11y ago
I wish they went more in-depth into the particular events. They seem to imply the events are user actions, which I took to mean user played this song, user viewed this playlist, and other analytics. However do they, or would you, use a system like Kafka as an event source instead of the db. So you would also capture events like user added this song to their playlist that would get persisted to the event source db and then eventually a view model instead of directly to a relational db. It doesn't sound like they do that due to the delays, but I can't find a lot of real usage blog posts about how much to put into something like kafka.
- dundun 11y ago
  You certainly can use a db for an event source. This article does a really good job of explaining how: http://www.confluent.io/blog/turning-the-database-inside-out-with-apache-samza/ http://www.confluent.io/blog/turning-the-database-inside-out... As mentioned in the post, we've pushed Kafka to at least 700,000 events per second. We have room to push it to much more, but stay in tune for post 2 and 3 to see what we're doing instead.
  - pheeney 11y ago
    That is actually the post that got me into event sourcing / streams. As far as user analytic type events go this makes complete sense to me. What I haven't been able to discern is whether its useful to use this architecture on a much much smaller scale for things that may not be user events. I love the thought of throwing everything into a stream and populating the read models, analytics, search index, etc with the data. However, for example if you had a CMS / ecom for a smaller organization, should the admin actions also be events? If you have an event source db, they would have to be, and you get all the benefits outlined from the article. At what point do you decide what to put in the stream and what to build without? Are there events that should never be in a stream? Those are the questions I have been researching but I haven't found a lot of resources or discussions around making these decisions.
    - pheeney 11y ago

122 points by vgt 11y ago

Interesting to look at the chart of event volumes at the end. Did Spotify just accidentally reveal the effect Apple Music has had on their growth rate? Steep drop around mid-2015, some recovery since then. Hard to glean anything concrete, but interesting nonetheless.

brazzledazzle 11y ago
I would avoid reading into that since the text surrounding that chart indicate that the system was struggling with the load. That drop could have simply been them prioritizing and removing certain event sources in the clients while they figured out how to fix things.

serialpreneur 11y ago

Interesting blog post. I don't have a good idea of the requirements at Spotify, but looks to me that using a streaming system like Storm or Spark Streaming would solve the 30 min event delivery delay they are experiencing w/ unstructured text -> Avro conversion. The latency for delivery would go down to sub-second levels.

vgt 11y ago
Stay tuned for Parts 2 + 3 :)
raphar 11y ago
Also the article doesn't say WHY they didn't use Kafka to persist the events. Kafka it's designed to do that. Once persisted the consumers can just read kafka data and send them to hadoop, with less latency. Or you can plug storm o spark in as you said and do the analisys there real time. Or both. I'm just intrigued why.
- RickHull 11y ago
  They do talk about this: > When the system was built, one of the missing features from Kafka 0.7 was the ability of the Kafka Broker cluster to behave as a reliable persistent storage. This influenced a major design decision to not keep persistent state between the producer of data, Kafka Syslog Producer, and Hadoop. An event is considered reliably persisted only when it gets written to a file on HDFS.

shockzzz 11y ago

Nice!

pheeney 11y ago

I wish they went more in-depth into the particular events. They seem to imply the events are user actions, which I took to mean user played this song, user viewed this playlist, and other analytics. However do they, or would you, use a system like Kafka as an event source instead of the db. So you would also capture events like user added this song to their playlist that would get persisted to the event source db and then eventually a view model instead of directly to a relational db. It doesn't sound like they do that due to the delays, but I can't find a lot of real usage blog posts about how much to put into something like kafka.

dundun 11y ago
You certainly can use a db for an event source. This article does a really good job of explaining how: http://www.confluent.io/blog/turning-the-database-inside-out-with-apache-samza/ http://www.confluent.io/blog/turning-the-database-inside-out... As mentioned in the post, we've pushed Kafka to at least 700,000 events per second. We have room to push it to much more, but stay in tune for post 2 and 3 to see what we're doing instead.
- pheeney 11y ago
  That is actually the post that got me into event sourcing / streams. As far as user analytic type events go this makes complete sense to me. What I haven't been able to discern is whether its useful to use this architecture on a much much smaller scale for things that may not be user events. I love the thought of throwing everything into a stream and populating the read models, analytics, search index, etc with the data. However, for example if you had a CMS / ecom for a smaller organization, should the admin actions also be events? If you have an event source db, they would have to be, and you get all the benefits outlined from the article. At what point do you decide what to put in the stream and what to build without? Are there events that should never be in a stream? Those are the questions I have been researching but I haven't found a lot of resources or discussions around making these decisions.
  - pheeney 11y ago