11 ms·
It could also be a sharding/checkpointing issue. Imagine you want to scan every file in every user account, for example for blank documents (maybe because yo
by londons_explore 4y ago
It could also be a sharding/checkpointing issue. Imagine you want to scan every file in every user account, for example for blank documents (maybe because you want to downrank blank 'untitled documents' in search results, because they are easily accidentally made).
You write your scanner to divide up the list of user accounts into chunks, process all chunks in different worker machines, and combine the results. Simple mapreduce.
However, if there is one huge user account, then you either have to wait for the entire process to take far longer, or you need to have multiple workers working on the single account (adding a lot of complexity to every operation you wish to run across all accounts).