I'm not sure how valid the last point is (my Data is 5TB or bigger) when you can create a 2TB (memory) x1 instance https://aws.amazon.com/ec2/instance-types/#me

I'm not sure how valid the last point is (my Data is 5TB or bigger) when you can create a 2TB (memory) x1 instance https://aws.amazon.com/ec2/instance-types/#memory-optimized https://aws.amazon.com/ec2/instance-types/#memory-optimized . Also someone else mentioned using Spark's Catalyst engine and DataFrames data structure. My previous experience with Spark it would automatically swap the data so it would be feasible to work with 5tb of data.