4 ms·

Can't they make a GPU instead? Please save us!

by esafak 1y ago

AlotOfReading 1y ago
A GPU is a very different beast that relies much more heavily on having a gigantic team of software developers supporting it. A CPU is (comparatively) straightforward. You fab and validate a world class design, make sure compiler support is good enough, upstream some drivers and kernel support, and make sure the standard documentation/debugging/optimization tools are all functional. This is incredibly difficult, but achievable because these are all standardized and well understood interface points. With GPUs you have all these challenges while also building a massively complicated set of custom compilers and interfaces on the software side, while at the same time trying to keep broken user software written against some other company's interface not only functional, but performant.
- esafak 1y ago
  It's not the GPU I want per se but its ability to run ML tasks. If you can do that with your CPU fine!
  - mort96 1y ago
    Well that's even more difficult because not only do you need drivers for the widespread graphics libraries Vulkan, OpenGL and Direct3D, but you also need to deal with the GPGPU mess. Most software won't ever support your compute-focused GPU because you won't support CUDA.
  - AlotOfReading 1y ago
    Echoing the other comment, this isn't easier. I was on a team that did it. The ML team was overheard by media complaining that we were preventing them from achieving their goals because we had taken 2 years to build something that didn't beat the latest hardware from Nvidia, let alone keep pace with how fast their demands had grown.
    - mdaniel 1y ago
      I don't need it to beat the latest from nvidia, just be affordable, available, and have user servicable ram slots so "48gb" isn't such an ooo-ahh amount of memory I couldn't find any buy it now links but 512gb sticks don't seem to be fantasies, either: https://news.samsung.com/global/samsung-develops-industrys-first-hkmg-based-ddr5-memory-ideal-for-bandwidth-intensive-advanced-computing-applications https://news.samsung.com/global/samsung-develops-industrys-f...
      - deleted 1y ago
        [deleted]
      - kvemkon 1y ago
        And now, 4 years later, I still can choose only among micron and hynix for consumer DDR5 DIMM. No samsung or nanya which I could order right now. While micron (crucial) 64GB DDR5 (SO-)DIMMs are available since few months.
      - mdaniel 1y ago
        Now that I'm back at my computer, can search harder and it seems one can legitimately buy 256GB sticks at approximately USD$2000 a pop <https://www.ebay.com/itm/267177294719 https://www.ebay.com/itm/267177294719> or 128GB for $790 <https://www.ebay.com/itm/205354052535 https://www.ebay.com/itm/205354052535> Since it seems A100s top out at 80GB, and appear to start at $10,000 I'd say it's a steal Yes, I'm acutely aware that bandwidth matters, but my mental model is the rest of that sentence is "up to a point," since those "self hosted LLM" threads are filled to the brim with people measuring tokens-per-minute or even running inference on CPU I'm not hardware adjacent enough to try such a stunt, but there was also recently a submission of a BSD-3-Clause implementation of Google's TPU <https://news.ycombinator.com/item?id=44111452 https://news.ycombinator.com/item?id=44111452>
      - AlotOfReading 1y ago
        What would you want from on-gpu slots that you don't get from existing mechanisms to give your GPU access to system memory?
  - Bolwin 1y ago
    I mean you most certainly can. Pretty much every ml library has cpu support
    - esafak 1y ago
      Not theoretically, but practically, viably.
- Asraelite 1y ago
  > make sure compiler support is good enough Do compilers optimize for specific RISC-V CPUs, not just profiles/extensions? Same for drivers and kernel support. My understanding was that if it's RISC-V compliant, no extra work is needed for existing software to run on it.
  - AlotOfReading 1y ago
    The major compilers optimize for microarchitecture, yes. Here's the tablegen scheduling definition behind LLVM's -mtune=sifive-670 flag as an example: https://github.com/llvm/llvm-project/blob/main/llvm/lib/Target/RISCV/RISCVSchedSiFiveP600.td https://github.com/llvm/llvm-project/blob/main/llvm/lib/Targ... It's not that things won't run, but this is necessary for compilers to generate well optimized code.
  - Arnavion 1y ago
    You want to optimize for specific chips because different chips have different capabilities that are not captured by just what extensions they support. A simple example is that the CPU might support running two specific instructions better if they were adjacent than if they were separated by other instructions ( https://en.wikichip.org/wiki/macro-operation_fusion https://en.wikichip.org/wiki/macro-operation_fusion ). So the optimizer can try to put those instructions next to each other. LLVM has target features for this, like "lui-addi-fusion" for CPUs that will fuse a `lui; addi` sequence into a single immediate load. A more complex example is keeping track of the CPU's internal state. The optimizer models the state of the CPU's functional units (integer, address generation, etc) so that it has an idea of which units will be in use at what time. If the optimizer has to allocate multiple instructions that will use some combination of those units, it can try to lay them out in an order that will minimize stalling on busy units while leaving other units unused. That information also tells the optimizer about the latency of each instruction, so when it has a choice between multiple ways to compute the same operation it can choose the one that works better on this CPU. See also: https://myhsu.xyz/llvm-sched-model-1/ https://myhsu.xyz/llvm-sched-model-1/ https://myhsu.xyz/llvm-sched-model-1.5/ https://myhsu.xyz/llvm-sched-model-1.5/ If you don't do this your code will still run on your CPU. It just won't necessarily be as optimal as it could be.
speedgoose 1y ago
I hope to see dedicated GPU coprocessors disappear sooner rather than later, just like arithmetic coprocessors did.
- wtallis 1y ago
  Arithmetic co-processors didn't disappear so much as they moved onto the main CPU die. There were performance advantages to having the FPU on the CPU, and there were no longer significant cost advantages to having the FPU be separate and optional. For GPUs today and in the foreseeable future, there are still good reasons for them to remain discrete, in some market segments. Low-power laptops have already moved entirely to integrated GPUs, and entry-level gaming laptops are moving in that direction. Desktops have widely varying GPU needs ranging from the minimal iGPUs that all desktop CPUs now already have, up to GPUs that dwarf the CPU in die and package size and power budget. Servers have needs ranging from one to several GPUs per CPU. There's no one right answer for how much GPU to integrate with the CPU.
  - otabdeveloper4 1y ago
    By "GPU" they probably mean "matrix multiplication coprocessor for AI tasks", not actually a graphics processor.
    - wtallis 1y ago
      That doesn't really change anything. The use cases for a GPU in any given market segment don't change depending on whether you call it a GPU. And for low-power consumer devices like laptops, "matrix multiplication coprocessor for AI tasks" is at least as likely to mean NPU as GPU, and NPUs are always integrated rather than discrete.
      - touisteur 1y ago
        Wondering how you'd classify Gaudi, tenstorrent-stuff, groq, or lightmatter's photonic thing. Calling something a GPU tends to make people ask for (good, performant) support for opengl, Vulkan, direct3d... which seem like a huge waste of effort if you want to be an "AI-coprocessor".

Can't they make a GPU instead? Please save us!

by esafak 1y ago

Can't they make a GPU instead? Please save us!

A GPU is a very different beast that relies much more heavily on having a gigantic team of software developers supporting it. A CPU is (comparatively) straightforward. You fab and validate a world class design, make sure compiler support is good enough, upstream some drivers and kernel support, and make sure the standard documentation/debugging/optimization tools are all functional. This is incredibly difficult, but achievable because these are all standardized and well understood interface points. With GPUs you have all these challenges while also building a massively complicated set of custom compilers and interfaces on the software side, while at the same time trying to keep broken user software written against some other company's interface not only functional, but performant.

esafak 1y ago
It's not the GPU I want per se but its ability to run ML tasks. If you can do that with your CPU fine!
- mort96 1y ago
  Well that's even more difficult because not only do you need drivers for the widespread graphics libraries Vulkan, OpenGL and Direct3D, but you also need to deal with the GPGPU mess. Most software won't ever support your compute-focused GPU because you won't support CUDA.
- AlotOfReading 1y ago
  Echoing the other comment, this isn't easier. I was on a team that did it. The ML team was overheard by media complaining that we were preventing them from achieving their goals because we had taken 2 years to build something that didn't beat the latest hardware from Nvidia, let alone keep pace with how fast their demands had grown.
  - mdaniel 1y ago
    I don't need it to beat the latest from nvidia, just be affordable, available, and have user servicable ram slots so "48gb" isn't such an ooo-ahh amount of memory I couldn't find any buy it now links but 512gb sticks don't seem to be fantasies, either: https://news.samsung.com/global/samsung-develops-industrys-first-hkmg-based-ddr5-memory-ideal-for-bandwidth-intensive-advanced-computing-applications https://news.samsung.com/global/samsung-develops-industrys-f...
    - deleted 1y ago
      [deleted]
    - kvemkon 1y ago
      And now, 4 years later, I still can choose only among micron and hynix for consumer DDR5 DIMM. No samsung or nanya which I could order right now. While micron (crucial) 64GB DDR5 (SO-)DIMMs are available since few months.
    - mdaniel 1y ago
      Now that I'm back at my computer, can search harder and it seems one can legitimately buy 256GB sticks at approximately USD$2000 a pop <https://www.ebay.com/itm/267177294719 https://www.ebay.com/itm/267177294719> or 128GB for $790 <https://www.ebay.com/itm/205354052535 https://www.ebay.com/itm/205354052535> Since it seems A100s top out at 80GB, and appear to start at $10,000 I'd say it's a steal Yes, I'm acutely aware that bandwidth matters, but my mental model is the rest of that sentence is "up to a point," since those "self hosted LLM" threads are filled to the brim with people measuring tokens-per-minute or even running inference on CPU I'm not hardware adjacent enough to try such a stunt, but there was also recently a submission of a BSD-3-Clause implementation of Google's TPU <https://news.ycombinator.com/item?id=44111452 https://news.ycombinator.com/item?id=44111452>
    - AlotOfReading 1y ago
      What would you want from on-gpu slots that you don't get from existing mechanisms to give your GPU access to system memory?
- Bolwin 1y ago
  I mean you most certainly can. Pretty much every ml library has cpu support
  - esafak 1y ago
    Not theoretically, but practically, viably.
Asraelite 1y ago
> make sure compiler support is good enough Do compilers optimize for specific RISC-V CPUs, not just profiles/extensions? Same for drivers and kernel support. My understanding was that if it's RISC-V compliant, no extra work is needed for existing software to run on it.
- AlotOfReading 1y ago
  The major compilers optimize for microarchitecture, yes. Here's the tablegen scheduling definition behind LLVM's -mtune=sifive-670 flag as an example: https://github.com/llvm/llvm-project/blob/main/llvm/lib/Target/RISCV/RISCVSchedSiFiveP600.td https://github.com/llvm/llvm-project/blob/main/llvm/lib/Targ... It's not that things won't run, but this is necessary for compilers to generate well optimized code.
- Arnavion 1y ago
  You want to optimize for specific chips because different chips have different capabilities that are not captured by just what extensions they support. A simple example is that the CPU might support running two specific instructions better if they were adjacent than if they were separated by other instructions ( https://en.wikichip.org/wiki/macro-operation_fusion https://en.wikichip.org/wiki/macro-operation_fusion ). So the optimizer can try to put those instructions next to each other. LLVM has target features for this, like "lui-addi-fusion" for CPUs that will fuse a `lui; addi` sequence into a single immediate load. A more complex example is keeping track of the CPU's internal state. The optimizer models the state of the CPU's functional units (integer, address generation, etc) so that it has an idea of which units will be in use at what time. If the optimizer has to allocate multiple instructions that will use some combination of those units, it can try to lay them out in an order that will minimize stalling on busy units while leaving other units unused. That information also tells the optimizer about the latency of each instruction, so when it has a choice between multiple ways to compute the same operation it can choose the one that works better on this CPU. See also: https://myhsu.xyz/llvm-sched-model-1/ https://myhsu.xyz/llvm-sched-model-1/ https://myhsu.xyz/llvm-sched-model-1.5/ https://myhsu.xyz/llvm-sched-model-1.5/ If you don't do this your code will still run on your CPU. It just won't necessarily be as optimal as it could be.

speedgoose 1y ago

I hope to see dedicated GPU coprocessors disappear sooner rather than later, just like arithmetic coprocessors did.

wtallis 1y ago
Arithmetic co-processors didn't disappear so much as they moved onto the main CPU die. There were performance advantages to having the FPU on the CPU, and there were no longer significant cost advantages to having the FPU be separate and optional. For GPUs today and in the foreseeable future, there are still good reasons for them to remain discrete, in some market segments. Low-power laptops have already moved entirely to integrated GPUs, and entry-level gaming laptops are moving in that direction. Desktops have widely varying GPU needs ranging from the minimal iGPUs that all desktop CPUs now already have, up to GPUs that dwarf the CPU in die and package size and power budget. Servers have needs ranging from one to several GPUs per CPU. There's no one right answer for how much GPU to integrate with the CPU.
- otabdeveloper4 1y ago
  By "GPU" they probably mean "matrix multiplication coprocessor for AI tasks", not actually a graphics processor.
  - wtallis 1y ago
    That doesn't really change anything. The use cases for a GPU in any given market segment don't change depending on whether you call it a GPU. And for low-power consumer devices like laptops, "matrix multiplication coprocessor for AI tasks" is at least as likely to mean NPU as GPU, and NPUs are always integrated rather than discrete.
    - touisteur 1y ago
      Wondering how you'd classify Gaudi, tenstorrent-stuff, groq, or lightmatter's photonic thing. Calling something a GPU tends to make people ask for (good, performant) support for opengl, Vulkan, direct3d... which seem like a huge waste of effort if you want to be an "AI-coprocessor".