The difference is that every variable is sharded by default. So if you have 100 machines with 8gb memory each, you can keep an 800gb array in memory. Some tradi

The difference is that every variable is sharded by default. So if you have 100 machines with 8gb memory each, you can keep an 800gb array in memory. Some traditional databases can do this for one operation at a time, like querying a sharded index in Mongo or doing a range query in Cassandra. But usually you want to do a whole pipeline of these operations. In that case, your computation engine should own the memory for efficiency, and seamlessly redistribute based on access patterns. Thus, Spark and friends.