4 ms·

One potential downside is disk space. If you actually want to retain all records, that's going to be an order of magnitude more space if your workload is updat

One potential downside is disk space. If you actually want to retain all records, that's going to be an order of magnitude more space if your workload is update-heavy. If it's primarily inserts not a big deal. The other potential downside I can see is if lost your views for any reason. If your views are not tolerant to partitions, then you face a long time to rebuild it the further out you go, so you may have to snapshot your views at certain intervals and sync with the specific record of the last kafka insert during snapshot, otherwise you have the same problem that people have when they clone a huge long-running git repo from the beginning. (edit: Kafka has log compaction so it may help alleviate some of the rebuilding process)

fintler 10y ago
> Kafka has log compaction so it may help alleviate some of the rebuilding process I've seen jobs spend 6+ hours rebuilding from Kafka when the local store is blown away for all partitions (using log compaction) -- if it's just a partition or 10 (single node failure), it takes way less time. When it comes to production, Samza's host affinity support is probably the most important thing to have working for when a job fails. https://samza.apache.org/learn/documentation/latest/yarn/yarn-host-affinity.html https://samza.apache.org/learn/documentation/latest/yarn/yar...