One piece caught my eye: > In terms of the best integration architecture, what seems to me the only long-term solution is something like the unified log archit

One piece caught my eye: > In terms of the best integration architecture, what seems to me the only long-term solution is something like the unified log architecture that Jay Kreps wrote about back in 2013. All incoming writes need to go into a centralized log, such as Kafka, and then from there the various databases can pull what they need, with each team making its own decisions about what it needs from that central log. Then, from the linked Jon Kreps post, 2013 [0]: > In order to allow horizontal scaling we chop up our log into partitions. Each partition is a totally ordered log, but there is no global ordering between partitions (other than perhaps some wall-clock time you might include in your messages). The assignment of the messages to a particular partition is controllable by the writer, with most users choosing to partition by some kind of key (e.g. user id). Partitioning allows log appends to occur without co-ordination between shards and allows the throughput of the system to scale linearly with the Kafka cluster size. In practice, wouldn't the end "centralized log" be essentially a collection of the logs from each original/logical database, now managed by a specialized log team? To read the log, the interested consumer team would have to create the appropriate read-only replica of the original database, using the respective database technology / binaries / schemas. As such, isn't the overall system better described as "database replica as a service", as opposed to the finer grained implication of the "unified log as a service" of the article / Jon Kreps post? I'm trying to figure out if the implication of such decisions are organizational, "who's responsible for the logs, who's responsible for the replicas", vs. architectural, "where does data bit X go, how does it maintain consistency with data bit Y, how do we read its current state?". [0] https://engineering.linkedin.com/distributed-systems/log-what-every-software-engineer-should-know-about-real-time-datas-unifying https://engineering.linkedin.com/distributed-systems/log-wha...