This approach to processing real-time and historical data works very well. We use a similar approach at Morgan Stanley and have a PL/compiler for real-time ind

This approach to processing real-time and historical data works very well. We use a similar approach at Morgan Stanley and have a PL/compiler for real-time index construction and query processing on top (https://github.com/Morgan-Stanley/hobbes https://github.com/Morgan-Stanley/hobbes). In addition to the general model of log writing + reactive processing, I've also found that we spend a lot of time focused on getting the _type structure_ of our data right (which allows for flexible/generic downstream processes that can also be efficient and type-checked against the structure of the data ahead of time). I wonder if/how this aspect of the problem has been considered by the Facebook team.