> 1) Where's the REST/Kafka boundary? I.e. when should something write to the Kafka log as opposed to POSTing directly to another service? E.g. If a user sets t

> 1) Where's the REST/Kafka boundary? I.e. when should something write to the Kafka log as opposed to POSTing directly to another service? E.g. If a user sets their payment method, do we update some DB immediatley, or do we write the fact that they set their payment method onto Kafka, and have another service read it? I believe the term that's emerging for this issue is "collapsing CQRS" and how you handle this is application-dependent (are you using plain synchronous HTTP requests? websockets?) In my case, the HTTP server has a producer that writes to a Kafka topic and a consumer that consumes the answers. The HTTP request waits until the answer appears on the answer topic. > 2) Services which had to replay from the beginning of time took a while to start up, so we had to find ways to get them not to. Kafka Streams local state makes this fast, unless you need to reprocess. > 3) You need to be serious about versioning. Since many services read messages from Kafka, you can't just change the implementation of those messages. We explicitly versioned every event in its class name. Yes, this is tricky. In my case, I either add fields in a backwards compatible manner, or rebase/rewrite event topics and roll them out while unwinding the previous version of the topic that may still be in use. The former is obviously the simpler option.