4 ms·
While pithy, the implication that you are going to process 12 TB of data in RAM using mostly single-threaded tools doesn't reflect reality.
by randyzwitch 7y ago
While pithy, the implication that you are going to process 12 TB of data in RAM using mostly single-threaded tools doesn't reflect reality.
- lotyrin 7y agoWhere exactly does that implication come from? Are you from a world where you need a map reduce framework + cluster to have parallelism of any kind?
- randyzwitch 7y agoNo, but I frequently see people implying that you can do your data science in Python and R as long as you can fit the data in RAM. As you mention, it's not RAM that's the limiting factor for larger data volumes, it's finding tools that exploit parallelism.
- lotyrin 7y agoI have done plenty of parallel work in python and R, so I'm still not sure what you mean.
- randyzwitch 7y agoYes, that's my point. It's too simplistic to say "well, the data fits in RAM", you have to add parallelism to make the workload tolerable. In the past, some people have done that using MapReduce or Spark, GNU parallel or just writing parallel code in their favorite language. But RAM by itself isn't the only limiting factor to whether a problem is solvable in a reasonable amount of time.
- Aeolun 7y agoIf you are using python and R, can’t you make your own parallelism?