>removing duplicates is as expensive as counting uniques. I don't think so. You do all kinds of things (unzip, sort, uniq, plus apparently 1-2 passes through t

>removing duplicates is as expensive as counting uniques. I don't think so. You do all kinds of things (unzip, sort, uniq, plus apparently 1-2 passes through the whole data). If you did it daily, you only need to dedup day 30 vs a sorted version of the rest of the month, which takes around a 30th of the time, and you don't need to unzip/sort/uniq 29/30ths worth of data. I don't know your exact structure but I don't see why most of the computation can't be done earlier. Think of it this way: on day 30 with the script described in the post, for the first $X amount of time you're reading the beginning of the files and don't touch any day 30 data until later in those 16 hours. So there's definitely some kind of processing that could be done prior to day 30.