We use Celery with Airflow. Recently had an incident where the celery workers were getting thrashed because, by default, the workers all heartbeat to each other

We use Celery with Airflow. Recently had an incident where the celery workers were getting thrashed because, by default, the workers all heartbeat to each other, via Redis. The networking on our redis instance was maxed out all the time and we had a secret soft-cap on the number of workers that could be active. The documentation even says that as-is this "gossiping" doesn't even do anything! Unsure if it's worth it for us to write our own, but I felt inspired to share my Celery horror-story.