3 ms·

With a 4B parameter model that probably ran through 8GBs of RAM multiple times to run. At a certain point we should seriously talk about CUDA accelerating Post

With a 4B parameter model that probably ran through 8GBs of RAM multiple times to run. At a certain point we should seriously talk about CUDA accelerating Postgres instead.

soerxpso 16d ago
I would think it's possible to make it so that the 4B model only needs to be called during an initial phase, and then the same queries it constructed can just be re-used with values replaced, unless you're generating a lot of unique on-the-fly query shapes.
- setr 16d ago
  With query hints finally being added it’d probably be doable as an extension
- williamdclt 16d ago
  Postgres takes the actual values into account when generating a query plan. The same query with different params can (and should) result in different query plans. It looks at statistics on the actual data stored.
bt1a 16d ago
pardon but aren't disks usually the bottleneck? im all for CUDA acceleration and CUDA accelerating culture
- voganmother42 16d ago
  I remember projects like PG-Strom back in the day, very cool stuff
- tomnipotent 16d ago
  Which is why a good query plan is so important, so that as much disk I/O can be avoided as possible (predicate push down, index elimination, join ordering, partition/scan pruning). Like the old CTE optimization fence problem.
- dragontamer 16d ago
  Parent post was talking about an 8GB dataset. 8GB isn't even CPU RAM these days. That's GPU super-mega-awesome ram. Ordinary Server CPUs are regularly pushing 2TB capacities. GPUs are in the 8GB to 32GB typically, at least for smaller and more regular GPUs. This GPU RAM is also well known to be at least 10x the bandwidth of CPU RAM.
  - saghm 16d ago
    Yeah, I have a GPU from almost six years ago in my desktop that has twice that much VRAM. Less than a year ago my wife got a 5070 Ti with the same for around $750 without needing to wait for it to be in stock or anything. I'm inclined to think that for a server that needs a GPU, even 32 GB would probably be considered small.
- fuy 16d ago
  not necessarily, no. With SSDs you get much better IOPS for cold data, and many datasets fit in RAM. So a lot of (OLTP/HTAP) workloads can become CPU-bound due to sorting/hashing - bread and butter of joins.
  - CodesInChaos 16d ago
    Though many use network block storage (e.g. EBS), which is significantly slower than an SSD.
eloisius 16d ago
What would you accelerate? Is there a lot of linear algebra you could throw cuda at in Postgres?
- dragontamer 16d ago
  You know that GPUs are more flexible than just linear algebra, right? GPUs are simply faster at fundamental algorithms like sorting (which has huge parallelism), and hashing. This is because both sorting and hashing benefit from endless growth of parallelism, offering enough "work" for these 10,000 SIMD-core systems to crunch work upon. And because of modern algorithms/libraries with 'Mergepath sort' (a GPU-SIMD parallel sorting algorithm), its not even that difficult to implement anymore. Naturally, this then leads to parallel Sort Merge Join, as well as parallel Hash-Join (two ways to implement left or right joins in a GPU that benefit from significant parallelism). So yeah, Joins. https://www.kenchoi.dev/papers/gpu-joins.pdf https://www.kenchoi.dev/papers/gpu-joins.pdf (This paper also has a description of "Mergepath sort", a GPU parallel way of sorting) --------- Even if GPUs weren't fundamentally faster at these kinds of operations... the RAM is simply 10x higher bandwidth and we all know its a RAM-constrained problem. Your typical SQL query is going to need multiple joins, probably a sort and possibly some "group" operations. As long as you have more than 10,000 elements or so (IE: can saturate all 10,000+ SIMD-units of a GPU), you'll be able to at least benefit from the faster RAM. If you have a LOT of joins (a recursive join or some other kind of deeply nested computationally complex query), you probably benefit even more from the greater compute-power offered by GPUs. These operations (joins really) are nominally over the entire set of data, and cleanly break down into obvious parallelism.
  - withinboredom 16d ago
    The GPU isn’t connected to the disk though. Usually. So you’d still have to load from disk, to ram, then from ram to the GPU.
    - yfontana 16d ago
      [dead]

With a 4B parameter model that probably ran through 8GBs of RAM multiple times to run. At a certain point we should seriously talk about CUDA accelerating Postgres instead.

soerxpso 16d ago

I would think it's possible to make it so that the 4B model only needs to be called during an initial phase, and then the same queries it constructed can just be re-used with values replaced, unless you're generating a lot of unique on-the-fly query shapes.

setr 16d ago
With query hints finally being added it’d probably be doable as an extension
williamdclt 16d ago
Postgres takes the actual values into account when generating a query plan. The same query with different params can (and should) result in different query plans. It looks at statistics on the actual data stored.

bt1a 16d ago

pardon but aren't disks usually the bottleneck? im all for CUDA acceleration and CUDA accelerating culture

voganmother42 16d ago
I remember projects like PG-Strom back in the day, very cool stuff
tomnipotent 16d ago
Which is why a good query plan is so important, so that as much disk I/O can be avoided as possible (predicate push down, index elimination, join ordering, partition/scan pruning). Like the old CTE optimization fence problem.
dragontamer 16d ago
Parent post was talking about an 8GB dataset. 8GB isn't even CPU RAM these days. That's GPU super-mega-awesome ram. Ordinary Server CPUs are regularly pushing 2TB capacities. GPUs are in the 8GB to 32GB typically, at least for smaller and more regular GPUs. This GPU RAM is also well known to be at least 10x the bandwidth of CPU RAM.
- saghm 16d ago
  Yeah, I have a GPU from almost six years ago in my desktop that has twice that much VRAM. Less than a year ago my wife got a 5070 Ti with the same for around $750 without needing to wait for it to be in stock or anything. I'm inclined to think that for a server that needs a GPU, even 32 GB would probably be considered small.
fuy 16d ago
not necessarily, no. With SSDs you get much better IOPS for cold data, and many datasets fit in RAM. So a lot of (OLTP/HTAP) workloads can become CPU-bound due to sorting/hashing - bread and butter of joins.
- CodesInChaos 16d ago
  Though many use network block storage (e.g. EBS), which is significantly slower than an SSD.

eloisius 16d ago

What would you accelerate? Is there a lot of linear algebra you could throw cuda at in Postgres?

dragontamer 16d ago
You know that GPUs are more flexible than just linear algebra, right? GPUs are simply faster at fundamental algorithms like sorting (which has huge parallelism), and hashing. This is because both sorting and hashing benefit from endless growth of parallelism, offering enough "work" for these 10,000 SIMD-core systems to crunch work upon. And because of modern algorithms/libraries with 'Mergepath sort' (a GPU-SIMD parallel sorting algorithm), its not even that difficult to implement anymore. Naturally, this then leads to parallel Sort Merge Join, as well as parallel Hash-Join (two ways to implement left or right joins in a GPU that benefit from significant parallelism). So yeah, Joins. https://www.kenchoi.dev/papers/gpu-joins.pdf https://www.kenchoi.dev/papers/gpu-joins.pdf (This paper also has a description of "Mergepath sort", a GPU parallel way of sorting) --------- Even if GPUs weren't fundamentally faster at these kinds of operations... the RAM is simply 10x higher bandwidth and we all know its a RAM-constrained problem. Your typical SQL query is going to need multiple joins, probably a sort and possibly some "group" operations. As long as you have more than 10,000 elements or so (IE: can saturate all 10,000+ SIMD-units of a GPU), you'll be able to at least benefit from the faster RAM. If you have a LOT of joins (a recursive join or some other kind of deeply nested computationally complex query), you probably benefit even more from the greater compute-power offered by GPUs. These operations (joins really) are nominally over the entire set of data, and cleanly break down into obvious parallelism.
- withinboredom 16d ago
  The GPU isn’t connected to the disk though. Usually. So you’d still have to load from disk, to ram, then from ram to the GPU.
  - yfontana 16d ago
    [dead]