4 ms·
Honest question: what does "in memory" mean exactly? Do traditional databases not process data using memory? Doesn't Spark spill to disk when it reached the lim
by luckydata 10y ago
Honest question: what does "in memory" mean exactly? Do traditional databases not process data using memory? Doesn't Spark spill to disk when it reached the limit of the system memory? The in-memory thing always puzzled me.
- ihodes 10y agoI think that "in memory" is in contrast to traditional MR jobs that wrote to disk a lot more than Spark jobs do.
- cle 10y agoYou can "cache" a dataset in-memory and continue to perform operations on the dataset without having to persist it to disk, unlike MapReduce. Traditional databases can do a lot of that too, but aren't as scalable.
- btown 10y agoThe difference is that every variable is sharded by default. So if you have 100 machines with 8gb memory each, you can keep an 800gb array in memory. Some traditional databases can do this for one operation at a time, like querying a sharded index in Mongo or doing a range query in Cassandra. But usually you want to do a whole pipeline of these operations. In that case, your computation engine should own the memory for efficiency, and seamlessly redistribute based on access patterns. Thus, Spark and friends.
- abelo_the_cool 10y agohack tanki