Architectural decisions explained here: https://github.com/eBay/beam/blob/master/docs/central_log_arch.md https://github.com/eBay/beam/blob/master/docs/central_

Architectural decisions explained here: https://github.com/eBay/beam/blob/master/docs/central_log_arch.md https://github.com/eBay/beam/blob/master/docs/central_log_ar... The architecture is: materialized views from a central append log. I don't see a reference to Apache Bookkeeper[1][2] in the above. My recommendation to anyone who wants to build their own variant of this architecture is to build it on top of Bookkeeper instead of rolling your own. It is designed for this precise workload [2] and addresses the write throughput limitations of the OP 'Beam'. [edit: pull quote from [2]]: "what makes BookKeeper unique is its ability to offer a short-tailed, low-latency, distributed scale-out storage solution. Although this is a CP system, its greater availability makes it almost a C(A)P system. It is an apt storage for immutable data." [1]: https://bookkeeper.apache.org/ https://bookkeeper.apache.org/ [2]:https://www.linux.com/news/event/apachecon/2017/4/high-performance-logging-apache-bookkeeper https://www.linux.com/news/event/apachecon/2017/4/high-perfo...