4 ms·
"time is counted as total processing time minus disk time." Can anyone explain why that's a valid benchmark for him to use? Certainly the Hadoop version had si
by MichaelGG 13y ago
"time is counted as total processing time minus disk time."
Can anyone explain why that's a valid benchmark for him to use? Certainly the Hadoop version had significant disk access time?
- Bill_Dimm 13y agoI'm guessing: Total time will vary dramatically depending on whether the database is on HD, SSD, or in memory, so he is separating out that component. Of course, he might be optimizing something that doesn't help much if it takes forever to read the data from disk.
- dragontamer 13y agoNot just the disk, but also transferring it over to the GPU is often a bottleneck. If there is back and forth talk between the GPU and CPU, things will slow down considerably.
- wmf 13y agoArguably if data fits in memory but particular software (Hadoop) doesn't keep the data in memory then that particular software can be considered unoptimized. But he should really be comparing against Shark anyway.
- dpe82 13y agoMuch of the point of using something like Hadoop is to increase raw disk IO by parallelizing it over many machines. So no, it's not a terribly valid benchmark.