3 ms·

They don't help on large volumes of data which cannot fit into GPU memory. Obviously. PCIe is even slower than standard CPU memory, pulling non-trivial amount

by paulmd 10y ago

They don't help on large volumes of data which cannot fit into GPU memory. Obviously. PCIe is even slower than standard CPU memory, pulling non-trivial amounts of data across the PCIe bus is catastrophic for any GPU program. GPUs shine because their VRAM has enormous volumes of bandwidth (up to ~512 GB/s for high-end modern GPUs). That's an order of magnitude more than CPU memory. So basically you throw indexes and perhaps a few key columns in VRAM, but as little actual data as possible, and just return the row IDs (or ranges) of interesting data. That lets you cross-reference back into host memory, and at that point you would perhaps perform some further limitation criteria based on something that's not available in GPU memory. GPU memory isn't super-abundant, but high-end cards have up to 24 GB, and if you stack a bunch of them in a rack then you are talking about a reasonable amount of memory for most use-cases. Four cards in a box is easy, and you can put four cards in a 1U chassis if you use GP100 mezzanine cards. So now we're talking 96GB of VRAM per box or per 1U, which works out to quite a bit of index data. The other use-case would be something that's really computationally intensive to perform, but for some reason cannot be pre-computed and cached. So - for example find me all rows where "sha256(INTEGER my_table.my_val AND now() AS INTEGER) > 256". Which as you can imagine mostly trends towards crypto kinds of stuff.

dogma1138 10y ago
Pascal supports upto 49bit memory addressing which covers 512TB worth of addresses, AST between the GPU and system memory addresses is done in hardware which means prefetches via usage hints / branch prediction and page faults are handled by the GPU. In this case you are only bound by PCIE bandwidth (or NVLINK if you are using a supporting GPU) and performance wise it's pretty near native speeds. You can also access network and other types of storage from within the GPU since MIMO is universal these days. AMD will add similar functionality starting with VEGA. This means that with both vendors you are no longer bound by the GPU onboard memory. And if PCIE x16 isn't enough then NVLINK comes with upto 200GT/s these days and has an infiniband end point.
- paulmd 10y ago
  The problem is "PCIe bandwidth" is really bad. PCIe 3.0x16 is only 16 GB/s, which again is less than half of you would get from any old DDR4 memory. You might as well do that on the CPU then. NVLink is fine as far as it goes... the problem is it's a two-party transaction, NVLink still has to be fed by something and that's either another GPU or the CPU or something like a SSD via DMA. With a database type program, the CPU and GPU use-cases are pointless because you could just be doing the search there and returning results directly instead of device-to-device DMA. And NVMe SSDs are usually limited to PCIe 2.0x4 which is 2 GB/s. Absolute maximum, 16 GB/s, same as any other PCIe-attached device. Remember, GPUs aren't always used on compute-bound tasks. There is a very valid use-case for I/O bound tasks where you are just using it as a super-cache. GPU database programs often (but not always) fall into that category. You are just searching indexes/target columns like crazy and throwing the resulting row IDs back over the wall to the CPU.
  - dogma1138 10y ago
    NVLINK is GPU to GPU or GPU to CPU link, if you are using it as a GPU to CPU link you are taking to the CPU at native NVLINK speeds either through a native NVLINK bus if you are using a POWER CPU which supports it or via an Infiniband/OmniPath interconnect on Intel/AMD CPUs. 16GB/s is a bit tight but that's where prefetches based on pre-defined usage hints as well as the branch predictor in the GPU and the CUDA Compiler/Runtime that feeds the GPU without stalling as much as possible. OFC you can I/O bound a GPU that's is actually pretty easy these days even in normal (gaming) workloads (e.g. draw call limit). Let's say you are building a 4 GPU DB, if you are using NVLINK you each GPU can see the entire memory space of the other GPUs connected on the same NVLINK (there is also RDMA which can be done over the network but this is a complicated scenario) in this case you have 96GB of high speed VRAM + which ever amount of memory you can put on your CPU which can be upto 1.5TB or so per socket these days, you also have 64GB/s of bandwidth to feed that 96GB of memory with which means that it takes a second and a half to completely refill the memory completely (and I'm still not sure that the same lossless compression schemes that GPUs use these days for won't work for your data which would give you much higher effective bandwidth that the compression/decompression if free on the GPU side). With these figures I can't see a reason why you can't optimize your memory residency to have the best of both worlds fast key-value/hash or index lookups for IO bound tasks and then programmatically prefetching the workload or if the branch prediction is good enough just letting it roll in processing bound tasks. So yes it's not "perfect" but nothing ever will, the question is it much faster or even orders of magnitude faster for certain tasks / workloads / implementations than a CPU only DB and clearly it is. How well would it compete against other contenders like Intel's Xeon Phis, programmable FPGAs and other emerging technologies that aim to solve this problem from another direction i don't know.

by paulmd 10y ago

dogma1138 10y ago

Pascal supports upto 49bit memory addressing which covers 512TB worth of addresses, AST between the GPU and system memory addresses is done in hardware which means prefetches via usage hints / branch prediction and page faults are handled by the GPU. In this case you are only bound by PCIE bandwidth (or NVLINK if you are using a supporting GPU) and performance wise it's pretty near native speeds. You can also access network and other types of storage from within the GPU since MIMO is universal these days. AMD will add similar functionality starting with VEGA. This means that with both vendors you are no longer bound by the GPU onboard memory. And if PCIE x16 isn't enough then NVLINK comes with upto 200GT/s these days and has an infiniband end point.

paulmd 10y ago
The problem is "PCIe bandwidth" is really bad. PCIe 3.0x16 is only 16 GB/s, which again is less than half of you would get from any old DDR4 memory. You might as well do that on the CPU then. NVLink is fine as far as it goes... the problem is it's a two-party transaction, NVLink still has to be fed by something and that's either another GPU or the CPU or something like a SSD via DMA. With a database type program, the CPU and GPU use-cases are pointless because you could just be doing the search there and returning results directly instead of device-to-device DMA. And NVMe SSDs are usually limited to PCIe 2.0x4 which is 2 GB/s. Absolute maximum, 16 GB/s, same as any other PCIe-attached device. Remember, GPUs aren't always used on compute-bound tasks. There is a very valid use-case for I/O bound tasks where you are just using it as a super-cache. GPU database programs often (but not always) fall into that category. You are just searching indexes/target columns like crazy and throwing the resulting row IDs back over the wall to the CPU.
- dogma1138 10y ago
  NVLINK is GPU to GPU or GPU to CPU link, if you are using it as a GPU to CPU link you are taking to the CPU at native NVLINK speeds either through a native NVLINK bus if you are using a POWER CPU which supports it or via an Infiniband/OmniPath interconnect on Intel/AMD CPUs. 16GB/s is a bit tight but that's where prefetches based on pre-defined usage hints as well as the branch predictor in the GPU and the CUDA Compiler/Runtime that feeds the GPU without stalling as much as possible. OFC you can I/O bound a GPU that's is actually pretty easy these days even in normal (gaming) workloads (e.g. draw call limit). Let's say you are building a 4 GPU DB, if you are using NVLINK you each GPU can see the entire memory space of the other GPUs connected on the same NVLINK (there is also RDMA which can be done over the network but this is a complicated scenario) in this case you have 96GB of high speed VRAM + which ever amount of memory you can put on your CPU which can be upto 1.5TB or so per socket these days, you also have 64GB/s of bandwidth to feed that 96GB of memory with which means that it takes a second and a half to completely refill the memory completely (and I'm still not sure that the same lossless compression schemes that GPUs use these days for won't work for your data which would give you much higher effective bandwidth that the compression/decompression if free on the GPU side). With these figures I can't see a reason why you can't optimize your memory residency to have the best of both worlds fast key-value/hash or index lookups for IO bound tasks and then programmatically prefetching the workload or if the branch prediction is good enough just letting it roll in processing bound tasks. So yes it's not "perfect" but nothing ever will, the question is it much faster or even orders of magnitude faster for certain tasks / workloads / implementations than a CPU only DB and clearly it is. How well would it compete against other contenders like Intel's Xeon Phis, programmable FPGAs and other emerging technologies that aim to solve this problem from another direction i don't know.