Since the original post criticizes the behavior of Panda runtime where it loads everything into memory before processing, I think replacing the underlying Panda

Since the original post criticizes the behavior of Panda runtime where it loads everything into memory before processing, I think replacing the underlying Panda runtime with Spark runtime is very much a valid solution. You can keep using Panda code for exploratory work, then with a little effort move it to production with Spark runtime.