Wow, this is a great idea. I recently worked on a team building streaming data pipelines, and we built a bespoke system to do exactly this: end-to-end test our

Wow, this is a great idea. I recently worked on a team building streaming data pipelines, and we built a bespoke system to do exactly this: end-to-end test our software. We had past messages written to a >300TB sharded file, and wrote a microservice to read each shard and publish it to the message queue for our staging instance, and then run data validation/anomaly detection on the output. It was useful but incredibly painful to use and maintain, and Batch would have been a fantastic solution for accomplishing this.