> Kafka has log compaction so it may help alleviate some of the rebuilding process I've seen jobs spend 6+ hours rebuilding from Kafka when the local store is

> Kafka has log compaction so it may help alleviate some of the rebuilding process I've seen jobs spend 6+ hours rebuilding from Kafka when the local store is blown away for all partitions (using log compaction) -- if it's just a partition or 10 (single node failure), it takes way less time. When it comes to production, Samza's host affinity support is probably the most important thing to have working for when a job fails. https://samza.apache.org/learn/documentation/latest/yarn/yarn-host-affinity.html https://samza.apache.org/learn/documentation/latest/yarn/yar...