Most of the work I do involves recent data--cycles are six months at most and 3 months on average. 3 months of data, sifting by a pretty strict filter, it's not

Most of the work I do involves recent data--cycles are six months at most and 3 months on average. 3 months of data, sifting by a pretty strict filter, it's not unsurprising that hundreds of terabytes of claims over years and years gets filtered into a few GB. Start to finish on jobs is a few hours (though it can be a few days if the filter is less strict or the task is more complicated), including pulling the data from the warehouse. Without a doubt, the bottleneck of the process is the data warehouse query. I'm sure having a more distributed database (it's DB2--I've been more pleased with Teradata's speed) could make the queries faster, but things are slow to change. Second to the query, the latency from working with a remote server (I'm in CA--the server's in Minnesota) adds latency if there's something I need to pull to the local machine. The actual SAS code (Is it still "high level" if the syntax models Fortran? I kid, I kid.) takes a negligible amount of time compared to the query.