6 ms·
Congratulations, well done! Our team has built an in-memory DB (hybrid, both column and row store available, also using advanced compression). We have investig
by mrich 15y ago
Congratulations, well done!
Our team has built an in-memory DB (hybrid, both column and row store available, also using advanced compression). We have investigated GPUs for scans, we found that while they can outperform CPUs, the bottleneck is the transfer of the data to the card. Since the card's memory is only in the GB's, and our databases are typically much larger this is kind of limiting, since you have to copy data back and forth.
Do you know what the maximum amount of GPU memory for a blade-like system currently is?
- antonmks 15y agoIt is 6 GB for Tesla GPU. PCI Express transfer speed is 5GB/s, with average compression rate averaging 4/1 the speed becomes 20GB/s so transfer speed is not a bottleneck anymore.
- alexchamberlain 15y agoDo you compress per row/column or over a given set?
- antonmks 15y agoBy column. FOR(frame of reference) compression for integers and decimals, FOR-DELTA for already sorted columns, dictionary for strings.
- mrich 15y agoI see. So for a large dataset you would still hit a bottleneck, compared to a main mem store which also uses compression. But nonetheless it's a good idea to do certain operations on hot data on the GPU. Which operations benefit the largest in your experience? I would guess scans with simple predicates and expressions. What about aggregations?
- alexchamberlain 15y agoExcept that hard drives have a much lower transfer rate?
- orcadk 15y agoBut we're already working on the premise that data is in the main memory, thus the bottleneck being the GPU<->memory transfer rate.
- binarymax 15y agoI had investigated aggregates using gpu for building dwarf petacubes (http://www.cs.umd.edu/~nick/projects/Dwarf.pdf http://www.cs.umd.edu/~nick/projects/Dwarf.pdf), and yes it looks viable.
- marshray 15y agoI had not heard of that data structure before, thanks! Sadly, it appears to be patented and thus of no use to me. http://www.cs.umd.edu/~nick/projects/Dwarf.html http://www.cs.umd.edu/~nick/projects/Dwarf.html I think it's really quite crass and mercantile for UMD to patent basic computer science algorithms and data structures like that.
- Symmetry 15y agoThis makes me think of CPU/GPU architectures where both are on the same die, but of course that won't work with CUDA.
- grundprinzip 15y agocan you share a link to the project?
- mrich 15y agoYou can find info here (unfortunately not just an easy download) http://www.sap.com/hana/index.epx http://www.sap.com/hana/index.epx
- grundprinzip 15y agoI know that one pretty well :)