3 ms·
60T sadly isn't that much. We tried (and failed) to use spark on a several PB dataset and it failed miserably.
by SEJeff 10y ago
60T sadly isn't that much. We tried (and failed) to use spark on a several PB dataset and it failed miserably.
- placeybordeaux 10y agoDo you have anymore details? What version?
- SEJeff 10y agoI want to say it was the highest 1.6.x around Feb/March of this year with a few PB (a sample of the real dataset) over an infiniband network. It just broke miserably. Also, java was never terribly good when you want to speak native ibverbs as the jni stuff is just slow.