You should try using approximate algorithms. They don't quite load all the data, but are able to give approximate (near perfect) statistical results whilr consu

You should try using approximate algorithms. They don't quite load all the data, but are able to give approximate (near perfect) statistical results whilr consuming orders of magnitude less data. Count sketches, Reservior sampling and similar methods come to mind.