Good point. I used Spark and DataFrames at my last job just because the data I needed was conveniently available in that environment. Using a high-memory single

Good point. I used Spark and DataFrames at my last job just because the data I needed was conveniently available in that environment. Using a high-memory single server seems better on general principles, until your data won't fit.