Yeah recently at work I created a compose.yml for our kafka pipeline. We have a collector and an enricher which produce to and consume from a kafka topic, respe

Yeah recently at work I created a compose.yml for our kafka pipeline. We have a collector and an enricher which produce to and consume from a kafka topic, respectively. The enricher then runs some functions on the data it consumes and produces to another topic. These services are all super microservicey, and so testing locally outside of docker was difficult (not to mention the PITA that is setting up a zookeeper broker on Windows) The compose.yml runs everything on a docker network and uses redpanda for the kafka broker. Iteration became SUPER easy. Previously, we were just writing config files locally and uploading them to S3, then restarting the dev instance of whatever we were working on in k8s to re-ingest. This was, as I'm sure you can tell by now, tedious