4 ms·

All of that is highly relevant for training but what the poster was asking for is a desktop inference card.

bayindirh 2y ago
You use at least half of this stack for desktop setups. You need copying daemons, the ecosystem support (docker-nvidia, etc.), some of the libraries, etc. even when you're on a single system. If you're doing inference on a server; MIG comes into play. If you're doing inference on a larger cloud, GPU-direct storage comes into play. It's all modular.
- WanderPanda 2y ago
  No you don‘t need much bandwidth between cards for inference
  - bayindirh 2y ago
    Copying daemons (gdrcopy) is about pumping data in and out of a single card. docker-nvidia and rest of the stack is enablement for using cards. GPU-Direct is about pumping data from storage devices to cards, esp. from high speed storage systems across networks. MIG actually shares a single card to multiple instances, so many processes or VMs can use a single card for smaller tasks. Nothing I have written in my previous comment is related to inter-card, inter-server communication, but all are related to disk-GPU, CPU-GPU or RAM-CPU communication. Edit: I mean, it's not OK to talk about downvoting, and downvote as you like but, I install and enable these cards for researchers. I know what I'm installing and what it does. C'mon now. :D
    - mikhael 2y ago
      Mostly, I think, we don’t really understand your argument that Intel couldn’t easily replicate the parts needed only for inference.
      - ac29 2y ago
        Yeah, for example llama.cpp runs on Intel GPUs via Vulkan or SYCL. The latter is actively being maintained by Intel developers. Obviously that is only one piece of software, but its a certainly a useful one if you are using one of the many LLMs it supports.
    - genewitch 2y ago
      i've run inference on Intel Arc and it works just fine so i am not sure what you're talking about. I certainly didn't need docker! I've never tried to do anything on AMD yet. I had the 16GB arc, and it was able to run inference at the speed i expected, but twice as many per batch as my 8GB card, which i think is about what you'd expect. once the model is on the card, there's no "disk" anymore, so having more vram to load the model and the tokenizer and whatever else on means there's no disk, and realistically when i am running loads on my 24GB 3090 the CPU is maybe 4% over idle usage. My bottleneck, as it stands, to running large models is vram, not anything else. If i needed to train (from scratch or whatever) i'd just rent time somewhere, even with a 128GB card locally, because obviously more tensors is better. and you're getting downvoted because there's literally lm studio and llama.cpp and sd-webui that run just fine for inference on our non-dc, non-nvlink, 1/15th the cost GPUs.
- landryraccoon 2y ago
  It's possible you're underestimating the open source community. If there's a competing platform that hobbyists can tinker with, the ecosystem can improve quite rapidly, especially when the competing platform is completely closed and hobbyists basically are locked out and have no alternative.
  - throwaway48476 2y ago
    Innovation is a bottom up process. If they sell the hardware the community will spring up to take advantage.
  - bayindirh 2y ago
    > It's possible you're underestimating the open source community. On the contrary. You really don't know how I love and prefer open source and love a more leveling playing field. > If there's a competing platform that hobbyists can tinker with... AMD's cards are better from hardware and software architecture standpoint, but the performance is not there yet. Plus, ROCm libraries are not that mature, but they're getting there. Developing high performance, high quality code is deceivingly expensive, because it's very heavy in theory, and you fly very close to the metal. I did that in my Ph.D., so I know what it entails. So it requires more than a couple (hundred) hobbyists to pull off (see the development of Eigen linear algebra library, or any high end math library). Some big guns are pouring money into AMD to implement good ROCm libraries, and it started paying off (Debian has a ton of ROCm packages now, too). However, you need to be able to pull it off in the datacenter to be able to pull it off on the desktop. AMD also needs to be able to enable ROCm on desktop properly, so people can start hacking it at home. > especially when the competing platform is completely closed... NVIDIA gives a lot of support to universities, researchers and institutions who play with their cards. Big cards may not be free, but know-how, support and first steps are always within reach. Plus, their researchers dogfood their own cards, and write papers with them. So, as long as papers got published, researchers do their research, and something got invented, many people don't care about how open source the ecosystem is. This upsets me a ton, but when closed source AI companies and researchers who forget to add crucial details to their papers so what they did can't be reproduced don't care about open source, because they think like NVIDIA. "My research, my secrets, my fame, my money". It's not about sharing. It's about winning, and it's ugly in some aspects.
- phkahler 2y ago
  No. I've been reading up. I'm planning to run Flux 12b on my AMD 5700G with 64GB RAM. CPU will take 5-10minutes per image which will be fine for me tinkering while writing code. Maybe I'll be able to get the GPU going on it too. Point of the OP is this is entirely possible with even an iGPU if only we have the RAM. nVidia should be irrelevant for local inference.
  - slavik81 2y ago
    The Ryzen 5700G is one of the APUs tested on the Debian ROCm CI [1]. It works quite well with the Debian / Ubuntu system packages. [1]: http://ci.rocm.debian.net/ http://ci.rocm.debian.net/
- ryao 2y ago
  Inferencing is much more simple than you think: See the precompute_input_logits() and forward() functions here: https://github.com/ryao/llama3.c/blob/master/run.c#L520 https://github.com/ryao/llama3.c/blob/master/run.c#L520 As a preface, precompute_input_logits() is really just a generalized version of the forward() function that can operate on multiple input tokens at a time to do faster input processing, although it can be used in place of the forward() function for output generation just by passing only a single token at a time. Also, my apologies for the code being a bit messy. matrix_multiply() and batched_matrix_multiply() are wrappers for GEMM, which I ended up having to use directly anyway when I needed to do strided access. Then matmul() is a wrapper for GEMV, which is really just a special case of GEMM. This is a work in progress personal R&D project that is based on prior work others did (as it spared me from having to do the legwork to implement the less interesting parts of inferencing), so it is not meant to be pretty. Anyway, my purpose in providing that link is to show what is needed to do inferencing (on llama 3). You have a bunch of matrix weights, plus a lookup table for vectors that represent tokens, in memory. Then your operations are: * memcpy() * memset() * GEMM (GEMV is a special case of GEMM) * sinf() * cosf() * expf() * sqrtf() * rmsnorm (see the C function for the definition) * softmax (see the C function for the definition) * Addition, subtraction, multiplication and division. I specify rmsnorm and softmax for completeness, but they can be implemented in terms of the other operations. If you can do those, you can do inferencing. You don’t really need very specialized things. Over 95% of time will be spent in GEMM too. My next steps likely will be to figure out how to implement fast GEMM kernels on my CPU. While my own SGEMV code outperforms the Intel MKL SGEMV code on my CPU (Ryzen 7 5800X where 1 core can use all memory bandwidth), my initial attempts at implementing SGEMM have not fared quite so well, but I will likely figure it out eventually. After I do, I can try adapting this to FP16 and then memory usage will finally be low enough that I can port it to a GPU with 24GB of VRAM. That would enable me to do what I say is possible rather than just saying it as I do here. By the way, the llama.cpp project has already figured all of this out and has things running on both GPUs and CPUs using just about every major quantization. I am rolling my own to teach myself how things work. By happy coincidence, I am somehow outperforming llama.cpp in prompt processing on my CPU but sadly, the secrets of how I am doing it are in Intel’s proprietary cblas_sgemm_batch() function. However, since I know it is possible for the hardware to perform like that, I can keep trying ideas for my own implementation until I get something that performs at the same level or better.

You use at least half of this stack for desktop setups. You need copying daemons, the ecosystem support (docker-nvidia, etc.), some of the libraries, etc. even when you're on a single system. If you're doing inference on a server; MIG comes into play. If you're doing inference on a larger cloud, GPU-direct storage comes into play. It's all modular.

WanderPanda 2y ago
No you don‘t need much bandwidth between cards for inference
- bayindirh 2y ago
  Copying daemons (gdrcopy) is about pumping data in and out of a single card. docker-nvidia and rest of the stack is enablement for using cards. GPU-Direct is about pumping data from storage devices to cards, esp. from high speed storage systems across networks. MIG actually shares a single card to multiple instances, so many processes or VMs can use a single card for smaller tasks. Nothing I have written in my previous comment is related to inter-card, inter-server communication, but all are related to disk-GPU, CPU-GPU or RAM-CPU communication. Edit: I mean, it's not OK to talk about downvoting, and downvote as you like but, I install and enable these cards for researchers. I know what I'm installing and what it does. C'mon now. :D
  - mikhael 2y ago
    Mostly, I think, we don’t really understand your argument that Intel couldn’t easily replicate the parts needed only for inference.
    - ac29 2y ago
      Yeah, for example llama.cpp runs on Intel GPUs via Vulkan or SYCL. The latter is actively being maintained by Intel developers. Obviously that is only one piece of software, but its a certainly a useful one if you are using one of the many LLMs it supports.
  - genewitch 2y ago
    i've run inference on Intel Arc and it works just fine so i am not sure what you're talking about. I certainly didn't need docker! I've never tried to do anything on AMD yet. I had the 16GB arc, and it was able to run inference at the speed i expected, but twice as many per batch as my 8GB card, which i think is about what you'd expect. once the model is on the card, there's no "disk" anymore, so having more vram to load the model and the tokenizer and whatever else on means there's no disk, and realistically when i am running loads on my 24GB 3090 the CPU is maybe 4% over idle usage. My bottleneck, as it stands, to running large models is vram, not anything else. If i needed to train (from scratch or whatever) i'd just rent time somewhere, even with a 128GB card locally, because obviously more tensors is better. and you're getting downvoted because there's literally lm studio and llama.cpp and sd-webui that run just fine for inference on our non-dc, non-nvlink, 1/15th the cost GPUs.
landryraccoon 2y ago
It's possible you're underestimating the open source community. If there's a competing platform that hobbyists can tinker with, the ecosystem can improve quite rapidly, especially when the competing platform is completely closed and hobbyists basically are locked out and have no alternative.
- throwaway48476 2y ago
  Innovation is a bottom up process. If they sell the hardware the community will spring up to take advantage.
- bayindirh 2y ago
  > It's possible you're underestimating the open source community. On the contrary. You really don't know how I love and prefer open source and love a more leveling playing field. > If there's a competing platform that hobbyists can tinker with... AMD's cards are better from hardware and software architecture standpoint, but the performance is not there yet. Plus, ROCm libraries are not that mature, but they're getting there. Developing high performance, high quality code is deceivingly expensive, because it's very heavy in theory, and you fly very close to the metal. I did that in my Ph.D., so I know what it entails. So it requires more than a couple (hundred) hobbyists to pull off (see the development of Eigen linear algebra library, or any high end math library). Some big guns are pouring money into AMD to implement good ROCm libraries, and it started paying off (Debian has a ton of ROCm packages now, too). However, you need to be able to pull it off in the datacenter to be able to pull it off on the desktop. AMD also needs to be able to enable ROCm on desktop properly, so people can start hacking it at home. > especially when the competing platform is completely closed... NVIDIA gives a lot of support to universities, researchers and institutions who play with their cards. Big cards may not be free, but know-how, support and first steps are always within reach. Plus, their researchers dogfood their own cards, and write papers with them. So, as long as papers got published, researchers do their research, and something got invented, many people don't care about how open source the ecosystem is. This upsets me a ton, but when closed source AI companies and researchers who forget to add crucial details to their papers so what they did can't be reproduced don't care about open source, because they think like NVIDIA. "My research, my secrets, my fame, my money". It's not about sharing. It's about winning, and it's ugly in some aspects.
phkahler 2y ago
No. I've been reading up. I'm planning to run Flux 12b on my AMD 5700G with 64GB RAM. CPU will take 5-10minutes per image which will be fine for me tinkering while writing code. Maybe I'll be able to get the GPU going on it too. Point of the OP is this is entirely possible with even an iGPU if only we have the RAM. nVidia should be irrelevant for local inference.
- slavik81 2y ago
  The Ryzen 5700G is one of the APUs tested on the Debian ROCm CI [1]. It works quite well with the Debian / Ubuntu system packages. [1]: http://ci.rocm.debian.net/ http://ci.rocm.debian.net/
ryao 2y ago
Inferencing is much more simple than you think: See the precompute_input_logits() and forward() functions here: https://github.com/ryao/llama3.c/blob/master/run.c#L520 https://github.com/ryao/llama3.c/blob/master/run.c#L520 As a preface, precompute_input_logits() is really just a generalized version of the forward() function that can operate on multiple input tokens at a time to do faster input processing, although it can be used in place of the forward() function for output generation just by passing only a single token at a time. Also, my apologies for the code being a bit messy. matrix_multiply() and batched_matrix_multiply() are wrappers for GEMM, which I ended up having to use directly anyway when I needed to do strided access. Then matmul() is a wrapper for GEMV, which is really just a special case of GEMM. This is a work in progress personal R&D project that is based on prior work others did (as it spared me from having to do the legwork to implement the less interesting parts of inferencing), so it is not meant to be pretty. Anyway, my purpose in providing that link is to show what is needed to do inferencing (on llama 3). You have a bunch of matrix weights, plus a lookup table for vectors that represent tokens, in memory. Then your operations are: * memcpy() * memset() * GEMM (GEMV is a special case of GEMM) * sinf() * cosf() * expf() * sqrtf() * rmsnorm (see the C function for the definition) * softmax (see the C function for the definition) * Addition, subtraction, multiplication and division. I specify rmsnorm and softmax for completeness, but they can be implemented in terms of the other operations. If you can do those, you can do inferencing. You don’t really need very specialized things. Over 95% of time will be spent in GEMM too. My next steps likely will be to figure out how to implement fast GEMM kernels on my CPU. While my own SGEMV code outperforms the Intel MKL SGEMV code on my CPU (Ryzen 7 5800X where 1 core can use all memory bandwidth), my initial attempts at implementing SGEMM have not fared quite so well, but I will likely figure it out eventually. After I do, I can try adapting this to FP16 and then memory usage will finally be low enough that I can port it to a GPU with 24GB of VRAM. That would enable me to do what I say is possible rather than just saying it as I do here. By the way, the llama.cpp project has already figured all of this out and has things running on both GPUs and CPUs using just about every major quantization. I am rolling my own to teach myself how things work. By happy coincidence, I am somehow outperforming llama.cpp in prompt processing on my CPU but sadly, the secrets of how I am doing it are in Intel’s proprietary cblas_sgemm_batch() function. However, since I know it is possible for the hardware to perform like that, I can keep trying ideas for my own implementation until I get something that performs at the same level or better.