You can just think of Kafka as a kind of pipe, specifically for real time data (think little messages that tell you an event happened). It takes this data from

You can just think of Kafka as a kind of pipe, specifically for real time data (think little messages that tell you an event happened). It takes this data from anything that produces it and moves it to anything that you want to consume that data stream. As an example, say you’ve got a service you built to keep track of the people installing your app, which platform they’re using, device version, etc. You’ve also got a service that tracks their screen views and what they’re doing in the app. Now, you want to be able to join those two data sources together so you can get better insights like ‘what do people on iOS do in my app first compared to those on Android?’. So you start thinking you should stick the data from these two services in one data warehouse, but how do you move the data where you need it to be? One thing you could do is to make a simple script that runs every 24 hours, to export the data from your two service databases to files, and import it to your data warehouse that way. This works fine, but what if you can’t wait up to 24 hours to get new info about installs and activity? Maybe you’re spending a ton of money on ads and want to be sure you’re not wasting it during a particular spike in traffic. Well then you might think about using Kafka to do it in real time. You can have one Kafka stream of install events—so every time an install happens, your install tracking service sends that event down the pipe labelled ‘installs’. Similarly, every time someone views a page in the app it gets sent down the pipe labelled ‘activity’ by the activity tracking service. At the other end of the pipes, your data warehouse gets the events and sticks them in the relevant tables, so you can start querying them very quickly after they happen. But! Your data warehouse doesn’t have to be the only thing sitting at the end of this pipe. You can have as many services as you want consuming the events. Want to trigger an email when someone installs your app? No problem, just make a service that consumes events from your existing ‘installs’ pipe and send an email whenever you get an event from it. This helps you decouple services, because your installs tracking service doesn’t need to know anything about your email service, or any other service at the other end of the pipe. Nor does it have to know about the activity tracking service. It just has to send messages for the type of events that are within its own scope. What other services do with those messages is up to them.