If it's this random it might be worth stepping back and trying to figure out how to get the portion of the results that are desired using a more effective mecha

If it's this random it might be worth stepping back and trying to figure out how to get the portion of the results that are desired using a more effective mechanism. Sometimes the source data structure is just a bad fit for the expressed data. I've seen this often in combinatorial expansions of data stored in hierarchal structure to a 2d table. Sometimes all, or a majority, of the data is going to be touched anyway so it's more costly to avoid a full read and instead more optimal to make the results of that read parallel; either by sorting to a more suitable structure or by working on streams / permutations / parallel processes from a dispatch design.