37 ms·
Do you mean 1TB of data? or cluster with total memory capacity less than 1TB?
by iamcreasy 3y ago
Do you mean 1TB of data? or cluster with total memory capacity less than 1TB?
- qsort 3y agoI mean data size. It's anecdotally the threshold where "normal" ETL methods start having problems. Obviously performance is hard and the only correct answer is "it depends", but I have never seen Spark improve things on datasets that small.