Thanks. Can you clarify what you mean by your last point? One of the main advantages of spark over Hadoop is that it doesn't need to spill to disk/ serialise be

Thanks. Can you clarify what you mean by your last point? One of the main advantages of spark over Hadoop is that it doesn't need to spill to disk/ serialise between steps, i.e. it is pitched as an in memory big data platform, so I'm a bit confused.