In case you haven't already read it, do read the classic LinkedIn "Log" article: https://engineering.linkedin.com/distributed-systems/log-what-every-software-en

In case you haven't already read it, do read the classic LinkedIn "Log" article: https://engineering.linkedin.com/distributed-systems/log-what-every-software-engineer-should-know-about-real-time-datas-unifying https://engineering.linkedin.com/distributed-systems/log-wha... I just learned Kafka this year to augment an existing system at work. One thing I have found is, there's a good amount of smarts in the Kafka clients. Look carefully at the Kafka clients available for the runtimes you work with, and evaluate them for robustness, features, and development activity (sadly, for example, kafka-node seems to have lost all momentum).