4 ms·

very interesting. Do you know of anything that would take advantage of the 128 cores of my Ryzen Threadripper even though I have a 2080 and a 3080 as well? (Or

very interesting. Do you know of anything that would take advantage of the 128 cores of my Ryzen Threadripper even though I have a 2080 and a 3080 as well? (Or all three... lol)

vorticalbox 4y ago
GPU is massively faster than CPUs for this sort of work load. Even if you could use all 128 cores I would still get the 2080 would be faster
- pmarreck 4y ago
  ah ok, that's fair.
- zackmorris 4y ago
  Keep in mind that many of the GPU cores go unused, since they are dedicated to geometry or ray tracing or whatnot: https://www.pcworld.com/article/402592/nvidia-turing-gpu-geforce-rtx-2080-ti.html https://www.pcworld.com/article/402592/nvidia-turing-gpu-gef... IMHO these chips designed for specialized workloads are looking more and more ridiculous. I expect GPU manufacturers to keep dragging them out for another decade or so, as well as Apple as it explores offshoots of M1. It all makes me feel very tired (the ultimate code smell). A better design would be something like a 256+ core RISC-V with local memories in/by each core for data-locality and a content-addressable caching scheme for deduplication. Copy-on-write languages like Clojure and orchestrating processes under something like Docker would make it a breeze to program, although it would still support manually managed mutability like with Rust for innermost loops in games or whatever. It's fairly obvious how it would all work, but IP law and gatekeeping ensure that it will not happen anytime soon. Then stuff like CUDA would be just another framework run on a symmetric multiprocessor and we could get back to exploring alternatives like genetic algorithms like we did in the 1990s. Thankfully nobody cares what I think, heck even I'm sick of reading my own complaints, so it's easy enough to just unsee this.
  - dragonwriter 4y ago
    > Keep in mind that many of the GPU cores go unused, since they are dedicated to geometry or ray tracing or whatnot I mean, “many” is not usually the case; an RTX 3090 has 10,496 CUDA cores, 328 Tensor cores, 82 RT (raytracing) cores, and 96 render output pipelines. ML apps will uses the first and, depending on software, second set. The vast majority of the cores being CUDA cores is the norm.
- zamalek 4y ago
  This is not strictly true. GPUs fare better at these tasks for a few reasons: * The largest contributor is the sheer number of cores. * The latency between cores and memory. * FP16 instructions. 128 cores isn't an insignificant fraction of the number of cores on a 1050 (about 600), but CPU cores are individually more powerful. That advantage is potentially difficult to call. The top-end of Genoa has 96/192 cores, and you can slot many of them onto a single board. AMD is throwing more and more memory into the CPU cache. That's very different to a direct path to GBs of HBM, but at some point the difference in performance might not matter to a novice/dabbler. I would say give it a go, see what happens.
pumanoir 4y ago
Look into sparsification of ML models: https://neuralmagic.com/ https://neuralmagic.com/ If your cpu has avx512 it can make it even faster.
qwertox 4y ago
I'm using a Ryzen 9 5950X to run some tests with Whisper (ASR), and since I have no GPU with more than 4 GB VRAM, I'm running it on the CPU. It is slow. It takes between 20 seconds and 2 minutes to transcribe 1 minute of audio, I'm using 8 cores. Adding more cores doesn't seem to improve the inferencing time. ML is really something which should be left to a GPU.
amrb 4y ago
There is an example to use multi GPU in the link, Outside of this I have read about offloading to cpu/nvme in the case of 100GB+ models that don't fit in VRAM tho this will be at the expense of performance. https://pypi.org/project/rwkvstic/ https://pypi.org/project/rwkvstic/

very interesting. Do you know of anything that would take advantage of the 128 cores of my Ryzen Threadripper even though I have a 2080 and a 3080 as well? (Or all three... lol)

vorticalbox 4y ago

GPU is massively faster than CPUs for this sort of work load. Even if you could use all 128 cores I would still get the 2080 would be faster

pmarreck 4y ago
ah ok, that's fair.
zackmorris 4y ago
Keep in mind that many of the GPU cores go unused, since they are dedicated to geometry or ray tracing or whatnot: https://www.pcworld.com/article/402592/nvidia-turing-gpu-geforce-rtx-2080-ti.html https://www.pcworld.com/article/402592/nvidia-turing-gpu-gef... IMHO these chips designed for specialized workloads are looking more and more ridiculous. I expect GPU manufacturers to keep dragging them out for another decade or so, as well as Apple as it explores offshoots of M1. It all makes me feel very tired (the ultimate code smell). A better design would be something like a 256+ core RISC-V with local memories in/by each core for data-locality and a content-addressable caching scheme for deduplication. Copy-on-write languages like Clojure and orchestrating processes under something like Docker would make it a breeze to program, although it would still support manually managed mutability like with Rust for innermost loops in games or whatever. It's fairly obvious how it would all work, but IP law and gatekeeping ensure that it will not happen anytime soon. Then stuff like CUDA would be just another framework run on a symmetric multiprocessor and we could get back to exploring alternatives like genetic algorithms like we did in the 1990s. Thankfully nobody cares what I think, heck even I'm sick of reading my own complaints, so it's easy enough to just unsee this.
- dragonwriter 4y ago
  > Keep in mind that many of the GPU cores go unused, since they are dedicated to geometry or ray tracing or whatnot I mean, “many” is not usually the case; an RTX 3090 has 10,496 CUDA cores, 328 Tensor cores, 82 RT (raytracing) cores, and 96 render output pipelines. ML apps will uses the first and, depending on software, second set. The vast majority of the cores being CUDA cores is the norm.
zamalek 4y ago
This is not strictly true. GPUs fare better at these tasks for a few reasons: * The largest contributor is the sheer number of cores. * The latency between cores and memory. * FP16 instructions. 128 cores isn't an insignificant fraction of the number of cores on a 1050 (about 600), but CPU cores are individually more powerful. That advantage is potentially difficult to call. The top-end of Genoa has 96/192 cores, and you can slot many of them onto a single board. AMD is throwing more and more memory into the CPU cache. That's very different to a direct path to GBs of HBM, but at some point the difference in performance might not matter to a novice/dabbler. I would say give it a go, see what happens.

pumanoir 4y ago

Look into sparsification of ML models: https://neuralmagic.com/ https://neuralmagic.com/ If your cpu has avx512 it can make it even faster.

qwertox 4y ago

I'm using a Ryzen 9 5950X to run some tests with Whisper (ASR), and since I have no GPU with more than 4 GB VRAM, I'm running it on the CPU. It is slow. It takes between 20 seconds and 2 minutes to transcribe 1 minute of audio, I'm using 8 cores. Adding more cores doesn't seem to improve the inferencing time. ML is really something which should be left to a GPU.

amrb 4y ago

There is an example to use multi GPU in the link, Outside of this I have read about offloading to cpu/nvme in the case of 100GB+ models that don't fit in VRAM tho this will be at the expense of performance. https://pypi.org/project/rwkvstic/ https://pypi.org/project/rwkvstic/