5 ms·

Past comments: https://news.ycombinator.com/item?id=19028860 https://news.ycombinator.com/item?id=19028860

ramoz 6y ago
> GPU databases are brilliant for cases where the working set can live entirely within the GPU's memory. For most applications with much larger (or more dynamic) working sets, the PCIe bus becomes a significant performance bottleneck. This is their traditional niche. > That said, I've heard anecdotes from people I trust that heavily optimized use of CPU vector instructions is competitive with GPUs for database use cases. This comment is important imo. Also related to applied ML inference in applications... the memory needs can grow quite a bit and this data transfer cost, including the memory size limitations vs RAM, becomes very real very fast. Not sure I understand the scale of the use case or where it's mentioned as well as in comparison to the big data tools mentioned.
- boulos 6y ago
  Disclosure: I work on Google Cloud (but you don’t need to rent GPUs from us). Absolutely, but as last year’s discussion highlights, a bunch of GPUs connected via NVLINK kind of gives you the aggregate memory of the set for some of these database applications (large-scale ML training has also gone this way). That’s why our A100 system design is 16 A100s in a single host. 16x40 GB gives you 640 GB of aggregate memory, which is pretty attractive for many applications. The question as always is cost vs benefit. If there’s something that a GPU backed < noun > can do that you “couldn’t” with a large Intel/AMD cpu box, or is actually a large integer multiple cheaper, it’s probably worth the development effort.
  - ramoz 6y ago
    Interesting. Much of my work here is on production GCP workloads. We've landed on C2s and cpu optimizations for our inference engine but hadn't really considered NVLINK. Now wondering if I can distribute batch inference across multi-gpus. We'll be on premium support soon ... hoping we can get access to folks like yourself for some of this.
    - boulos 6y ago
      I should clarify: A100s are likely a bad price/performance trade off for inference. The T4 part is designed for that, but doesn’t have NVLINK. Do you have models >16 GB that you’re trying to do real-time inference against? Feel free to send me an email regardless! (In my profile) Edit: https://news.ycombinator.com/item?id=23800049 https://news.ycombinator.com/item?id=23800049 was my writeup for a recent Ask HN about cost efficient inference.
      - zetazzed 6y ago
        A100s support MIG (multi instance GPUs), so you can carve each A100 up into 7 separate GPUs for inference of small models. If your inference workload is too small to feed this beast all at once, it can be pretty handy: https://docs.nvidia.com/datacenter/tesla/mig-user-guide/index.html https://docs.nvidia.com/datacenter/tesla/mig-user-guide/inde...
- kanwisher 6y ago
  Next gen Nvidia 30x0 series, can have direct access to SSD without hitting the CPU. In that case would they be any worse than cpus on any workloads? I guess you could still have larger ram amounts on the cpu, albeit slower ram usually
  - WanderPanda 6y ago
    Wow nvidia is selling SBCs now?
    - ReactiveJelly 6y ago
      No, there's a new thing about giving GPUs some kind of DMA to storage. And it's pointless on HDDs, so it's only discussed in terms of SSDs. Microsoft is bringing the DirectStorage API from XBox to Windows, Nvidia calls theirs RTX IO. I think they're the same class of idea, like Vulken vs. Metal. They do have SBCs, I think, but other than being the basis for the Nintendo Switch I haven't heard much about them.
    - manigandham 6y ago
      Like this? https://developer.nvidia.com/embedded/jetson-nano-developer-kit https://developer.nvidia.com/embedded/jetson-nano-developer-...
  - ramoz 6y ago
    Hmm - in our case all of the embeddings we process exist on SSD already. Idk enough here honestly but will see what I can learn.
  - t0mas88 6y ago
    Interesting, then the only difference is that a CPU easily as 256 GB ram and a high end GPU typically 16 GB or so? May Nvidia will start to create ML type cards with memory expansion options at some point.
  - dogma1138 6y ago
    That's for gaming (new DirectX feature), GPUDirect Storage is supported on older GPUs as well for compute. https://developer.nvidia.com/blog/gpudirect-storage/ https://developer.nvidia.com/blog/gpudirect-storage/
- lmeyerov 6y ago
  Yeah the sweet spots is landing as: * CPUs: medium data, and queries that are small / slow / irregular * GPUs: general analytics over data that is small (in-memory) or large / streaming data (replace Spark): Data perspective: -- small data (100MB - 512GB): all in GPU memory, so question if boring queries ("select username from django_table" better in psql) or compute ones ("select price where ..." better in GPU SQL) -- medium data: data sits in CPU RAM / SSD with compute nodes, and in a preorganized / static fashion, e.g., time series DB. too much data for GPU RAM, yet enough for for local SSD, so PCI bus is the bottleneck (8-32 GB/s) -- large data (ex: 10TB spread through S3 buckets) + streaming (ex: 10GB/s netflow): you'll be waiting on network bandwidth anyways, so network link of 10GB/s -> PCI of 10GB/s -> GPU wins out over CPU equiv anyways. Good chance, instead of the pricey multi-GPU V100/A100s, you'll want a fleet of wimpy T4 GPUs. As network/disk<>GPU high-bandwidth hw rolls out and libs automate their use, the current medium data sweet spot of CPU analytics systems goes away. Compute perspective: -- The category of 'irregular' (non-vectorizable) compute has been steadily shrinking for the last ~30 years as it's an important + fun topic for CS people. Even CPU systems now try to generally optimize for bulk-fetches (cacheline, ...) & SIMD-compute (e.g., SIMD over columns), and that inherently can only go so far until it's effectively a GPU alg on worse hw. I see other areas in practice like crazy-RAM CPU boxes and FPGA/ASIC systems that I'm intentionally skipping as these end up pretty tailored, while my breakdown above is increasingly common for 'commodity HPC'.
  - jandrewrogers 6y ago
    I don't think the "large data" case holds true, and I would not expect it to be economical to use GPUs for that. First, this is essentially limiting the scope of "analytics" to selection/aggregation centric operations which are memory bandwidth bound. Many types of high-value analytic workloads and data models don't look like that. Even when 90% of your workload is optimal for GPUs, I've often seen the pattern that the last 10% is poor enough that it largely offsets the benefit. Also, GPUs have better memory bandwidth than CPUs but people overlook that CPUs can use their limited memory bandwidth more efficiently for the same abstract workload, so the performance gap is smaller than memory-bandwidth numbers alone would suggest. Second, 10TB is tiny; this is around the top-end of what we consider "small data" at most companies where I work. For example, in the very broad domain of sensor and spatiotemporal analytics, we tend to use 10 petabytes as the point where data becomes "large" currently, and data sets this size are ubiquitous. This data is stored with the compute when at all possible for obvious reasons -- it ends up looking more like your "medium" case in practice, albeit across a small-ish number of machines. The cost of processing tens of petabytes of data on GPUs would be prohibitive. Lastly, a growing percentage of analytics at every scale is operational real-time, so new data needs to be integrated into the analytical data model approximately instantly. GPUs are not good at this type of architecture. GPUs have their use cases but their Achille's Heel is that their performance sweet spot is too narrow for many (most?) real-world analytic workloads, and for some workload patterns the performance can be much worse than CPUs. CPUs provide much more consistent and predictable performance across diverse and changing workload requirements, which is a valuable property even it has worse performance for some workloads. Databases give considerable priority to minimizing performance variability because users do.

> GPU databases are brilliant for cases where the working set can live entirely within the GPU's memory. For most applications with much larger (or more dynamic) working sets, the PCIe bus becomes a significant performance bottleneck. This is their traditional niche. > That said, I've heard anecdotes from people I trust that heavily optimized use of CPU vector instructions is competitive with GPUs for database use cases. This comment is important imo. Also related to applied ML inference in applications... the memory needs can grow quite a bit and this data transfer cost, including the memory size limitations vs RAM, becomes very real very fast. Not sure I understand the scale of the use case or where it's mentioned as well as in comparison to the big data tools mentioned.

boulos 6y ago
Disclosure: I work on Google Cloud (but you don’t need to rent GPUs from us). Absolutely, but as last year’s discussion highlights, a bunch of GPUs connected via NVLINK kind of gives you the aggregate memory of the set for some of these database applications (large-scale ML training has also gone this way). That’s why our A100 system design is 16 A100s in a single host. 16x40 GB gives you 640 GB of aggregate memory, which is pretty attractive for many applications. The question as always is cost vs benefit. If there’s something that a GPU backed < noun > can do that you “couldn’t” with a large Intel/AMD cpu box, or is actually a large integer multiple cheaper, it’s probably worth the development effort.
- ramoz 6y ago
  Interesting. Much of my work here is on production GCP workloads. We've landed on C2s and cpu optimizations for our inference engine but hadn't really considered NVLINK. Now wondering if I can distribute batch inference across multi-gpus. We'll be on premium support soon ... hoping we can get access to folks like yourself for some of this.
  - boulos 6y ago
    I should clarify: A100s are likely a bad price/performance trade off for inference. The T4 part is designed for that, but doesn’t have NVLINK. Do you have models >16 GB that you’re trying to do real-time inference against? Feel free to send me an email regardless! (In my profile) Edit: https://news.ycombinator.com/item?id=23800049 https://news.ycombinator.com/item?id=23800049 was my writeup for a recent Ask HN about cost efficient inference.
    - zetazzed 6y ago
      A100s support MIG (multi instance GPUs), so you can carve each A100 up into 7 separate GPUs for inference of small models. If your inference workload is too small to feed this beast all at once, it can be pretty handy: https://docs.nvidia.com/datacenter/tesla/mig-user-guide/index.html https://docs.nvidia.com/datacenter/tesla/mig-user-guide/inde...
kanwisher 6y ago
Next gen Nvidia 30x0 series, can have direct access to SSD without hitting the CPU. In that case would they be any worse than cpus on any workloads? I guess you could still have larger ram amounts on the cpu, albeit slower ram usually
- WanderPanda 6y ago
  Wow nvidia is selling SBCs now?
  - ReactiveJelly 6y ago
    No, there's a new thing about giving GPUs some kind of DMA to storage. And it's pointless on HDDs, so it's only discussed in terms of SSDs. Microsoft is bringing the DirectStorage API from XBox to Windows, Nvidia calls theirs RTX IO. I think they're the same class of idea, like Vulken vs. Metal. They do have SBCs, I think, but other than being the basis for the Nintendo Switch I haven't heard much about them.
  - manigandham 6y ago
    Like this? https://developer.nvidia.com/embedded/jetson-nano-developer-kit https://developer.nvidia.com/embedded/jetson-nano-developer-...
- ramoz 6y ago
  Hmm - in our case all of the embeddings we process exist on SSD already. Idk enough here honestly but will see what I can learn.
- t0mas88 6y ago
  Interesting, then the only difference is that a CPU easily as 256 GB ram and a high end GPU typically 16 GB or so? May Nvidia will start to create ML type cards with memory expansion options at some point.
- dogma1138 6y ago
  That's for gaming (new DirectX feature), GPUDirect Storage is supported on older GPUs as well for compute. https://developer.nvidia.com/blog/gpudirect-storage/ https://developer.nvidia.com/blog/gpudirect-storage/
lmeyerov 6y ago
Yeah the sweet spots is landing as: * CPUs: medium data, and queries that are small / slow / irregular * GPUs: general analytics over data that is small (in-memory) or large / streaming data (replace Spark): Data perspective: -- small data (100MB - 512GB): all in GPU memory, so question if boring queries ("select username from django_table" better in psql) or compute ones ("select price where ..." better in GPU SQL) -- medium data: data sits in CPU RAM / SSD with compute nodes, and in a preorganized / static fashion, e.g., time series DB. too much data for GPU RAM, yet enough for for local SSD, so PCI bus is the bottleneck (8-32 GB/s) -- large data (ex: 10TB spread through S3 buckets) + streaming (ex: 10GB/s netflow): you'll be waiting on network bandwidth anyways, so network link of 10GB/s -> PCI of 10GB/s -> GPU wins out over CPU equiv anyways. Good chance, instead of the pricey multi-GPU V100/A100s, you'll want a fleet of wimpy T4 GPUs. As network/disk<>GPU high-bandwidth hw rolls out and libs automate their use, the current medium data sweet spot of CPU analytics systems goes away. Compute perspective: -- The category of 'irregular' (non-vectorizable) compute has been steadily shrinking for the last ~30 years as it's an important + fun topic for CS people. Even CPU systems now try to generally optimize for bulk-fetches (cacheline, ...) & SIMD-compute (e.g., SIMD over columns), and that inherently can only go so far until it's effectively a GPU alg on worse hw. I see other areas in practice like crazy-RAM CPU boxes and FPGA/ASIC systems that I'm intentionally skipping as these end up pretty tailored, while my breakdown above is increasingly common for 'commodity HPC'.
- jandrewrogers 6y ago
  I don't think the "large data" case holds true, and I would not expect it to be economical to use GPUs for that. First, this is essentially limiting the scope of "analytics" to selection/aggregation centric operations which are memory bandwidth bound. Many types of high-value analytic workloads and data models don't look like that. Even when 90% of your workload is optimal for GPUs, I've often seen the pattern that the last 10% is poor enough that it largely offsets the benefit. Also, GPUs have better memory bandwidth than CPUs but people overlook that CPUs can use their limited memory bandwidth more efficiently for the same abstract workload, so the performance gap is smaller than memory-bandwidth numbers alone would suggest. Second, 10TB is tiny; this is around the top-end of what we consider "small data" at most companies where I work. For example, in the very broad domain of sensor and spatiotemporal analytics, we tend to use 10 petabytes as the point where data becomes "large" currently, and data sets this size are ubiquitous. This data is stored with the compute when at all possible for obvious reasons -- it ends up looking more like your "medium" case in practice, albeit across a small-ish number of machines. The cost of processing tens of petabytes of data on GPUs would be prohibitive. Lastly, a growing percentage of analytics at every scale is operational real-time, so new data needs to be integrated into the analytical data model approximately instantly. GPUs are not good at this type of architecture. GPUs have their use cases but their Achille's Heel is that their performance sweet spot is too narrow for many (most?) real-world analytic workloads, and for some workload patterns the performance can be much worse than CPUs. CPUs provide much more consistent and predictable performance across diverse and changing workload requirements, which is a valuable property even it has worse performance for some workloads. Databases give considerable priority to minimizing performance variability because users do.