I agree with your reasoning and your insight about missed value is valuable, but this is not a Python problem. It's a scaling problem. You can solve scaling pr

I agree with your reasoning and your insight about missed value is valuable, but this is not a Python problem. It's a scaling problem. You can solve scaling problems in many different ways with different costs. Any reasonably sized company has a Kubernetes cluster, just launch 10 instances and you're down to 24 minutes instead of 4 hours. You could also buy faster hardware. You might increase single-node performance by rewriting in a performant language (this won't help much if I/O is the bottleneck though, which often is the case). Talking to your data supplier to check if they can send the data in a machine-readable format is probably also something to consider. Or you might conclude that any effort is not worthwhile at all. Who knows? Scaling problems are not be avoided as a negative thing, they're beautiful and ought to be celebrated. What's not beautiful is letting the thought of potential scaling problems lead you to make decisions that optimize for long-term results instead of short-terms results.