3 ms·
Admittedly, the specific mechanics of how the log interacts with data partitions was glossed over a bit in that post. However, to answer your question, the inp
by freels 9y ago
Admittedly, the specific mechanics of how the log interacts with data partitions was glossed over a bit in that post.
However, to answer your question, the input log is partitioned: Transactions for each epoch are striped across the log partitions. A data partition will need to read a batch from each log partition in order to be able to process the transactions for any given epoch.