I am currently on a task of processing 4 million records to produce a report. It takes me about 10 seconds in the code. You could always use a sliding window

I am currently on a task of processing 4 million records to produce a report. It takes me about 10 seconds in the code. You could always use a sliding window type algorithm to keep up with the grown of the data.