That's a pattern to reduce on queue size (the one between broker and consumer), but it still means the broker (here Kafka) would need to queue up everything tha

That's a pattern to reduce on queue size (the one between broker and consumer), but it still means the broker (here Kafka) would need to queue up everything that any consumer might potentially need. It can work, and the data is at least limited by the amount and speed of producers. But it can also get a bottlneck. > How do you even solve it, in principle for systems like this? What should happen if a client can't process all messages? Do you drop some? Should the server persist messages on disk? (And how / when?) This will all pretty much depend on the problem domain. If one can't drop messages, then it pretty much all doesn't help. But usually in systems clients do not need all the past history - they might just need the newest data. So it's possible to detect in various parts of the system (even the server or broker before the TCP stack) that some data is superseeded with a newer version, and then drop the old variant instead of sending it too. If purely "latest state" has to be transferred to a client, the max amount of data which needs to be queued is always sizeof(state). now this gets again tricky, because the state could be big and completely resending it might also be expensive. So a compromise solution that is sometimes used is sending the client one snapshat of the complete state when connecting, and then falling back to sending events/diffs. If one client falls behind on processing diffs, the will likely reconnect and get a new starting point. Obviously also has its gotchas, since you don't want to get to a point where there's constant fetching of initial state which is most costly. But at least it limits the amount of data that is queued anywhere in the system to "whatever is required between 2 state snapshots".