4 ms·

What I couldn't find is inference benchmarks for consumer hardware. Just pick a reasonable workload with llama.cpp or ollama and show us some numbers. I'm part

What I couldn't find is inference benchmarks for consumer hardware. Just pick a reasonable workload with llama.cpp or ollama and show us some numbers. I'm particularly interested in building a Home Assistant machine that can run the voice assistant locally (STT/TTS/LLM) while using the least amount of power / generating the least amount of heat and noise.

GaggiX 2y ago
It would be cool to see these benchmarks on the newly released Jetson Orin Nano Super, like faster-whisper.
cherryteastain 2y ago
Based on a machine we had bought at my university with 4 AMD W6800s (which are just RX 6800s with double the VRAM), it's bad _even if it works at all_.
curt15 2y ago
AMD's software for consumer GPUs demonstrates a lack of seriousness. ROCm only officially supports RDNA2 and RDNA3 GPUs (their last two generations of hardware), and for some reason most of them are supported on only Windows (https://rocm.docs.amd.com/projects/install-on-windows/en/latest/reference/system-requirements.html https://rocm.docs.amd.com/projects/install-on-windows/en/lat...) and not Linux (https://rocm.docs.amd.com/projects/install-on-linux/en/latest/reference/system-requirements.html https://rocm.docs.amd.com/projects/install-on-linux/en/lates...), where most AI training and inference occurs. In particular, Linux users can only start playing with ROCm with a top-of-the-line, power-guzzling unit whereas they can get started with CUDA using basically any Nvidia GPU on desktops or laptops.
- cherryteastain 2y ago
  In practice, consumer Navi 21 based cards (RX 6900XT etc) and Navi 31 cards (RX 7900 XTX etc) are compatible with Pytorch on Linux. What they write about ROCm and Windows is equivocation. They target only one app: Blender. Pytorch+ROCm+Windows does not work. I had bought a 6900XT myself around launch time (the RTX3080 I ordered was not coming, it was the chip shortage times...) and it took around 2 years for Pytorch to become actually usable on it.
  - thousand_nights 2y ago
    in practice everyone who wants to do ML at home buys nvidia and pays the premium
    - tonetegeatinst 2y ago
      Sad but true. Years ago, pre2018 nvidia was the goto hardware supplier if you were doing anything with neural networks. I remember CUDA being much more buggy back then but it still worked pretty good. Back then AMD wasn't considered a real competition for ML/AI hardware. Glad as always to see more competition in the market to drive innovations. AMD seems to be letting larger VRAM onto consumer cards, which is nice to see, just hope the AI/ML experience can get better for their software ecosystem.
  - ryao 2y ago
    The obligatory link: https://xkcd.com/644/ https://xkcd.com/644/ That said, I would not expect it to stay working for long as long as ROCm is a dependency since AMD drops support for its older GPUs quickly while Nvidia continues to support older GPUs with less frequent legacy driver updates.
  - sliken 2y ago
    Maybe, but it's pretty lame when you buy a new $500 CPU (7800 XT) and the docs say it's unsupported, and makes you feel like even if you reported a bug they would just say "Sorry, not supported". Did make me wish I bought a Nvidia.
sroussey 2y ago
You might just check out the Home Assistant Voice: https://ameridroid.com/products/home-assistant-voice-preview-edition https://ameridroid.com/products/home-assistant-voice-preview...
- gigel82 2y ago
  Yes, that's exactly what I was checking out. You need fast enough hardware to run the speech to text, text to speech and (most importantly) LLM locally: https://www.youtube.com/watch?v=XvbVePuP7NY https://www.youtube.com/watch?v=XvbVePuP7NY (he has dual 3090 GPUs but that's not a practical setup for most people - budget / power / noise).

GaggiX 2y ago

It would be cool to see these benchmarks on the newly released Jetson Orin Nano Super, like faster-whisper.

cherryteastain 2y ago

Based on a machine we had bought at my university with 4 AMD W6800s (which are just RX 6800s with double the VRAM), it's bad _even if it works at all_.

curt15 2y ago

AMD's software for consumer GPUs demonstrates a lack of seriousness. ROCm only officially supports RDNA2 and RDNA3 GPUs (their last two generations of hardware), and for some reason most of them are supported on only Windows (https://rocm.docs.amd.com/projects/install-on-windows/en/latest/reference/system-requirements.html https://rocm.docs.amd.com/projects/install-on-windows/en/lat...) and not Linux (https://rocm.docs.amd.com/projects/install-on-linux/en/latest/reference/system-requirements.html https://rocm.docs.amd.com/projects/install-on-linux/en/lates...), where most AI training and inference occurs. In particular, Linux users can only start playing with ROCm with a top-of-the-line, power-guzzling unit whereas they can get started with CUDA using basically any Nvidia GPU on desktops or laptops.

cherryteastain 2y ago
In practice, consumer Navi 21 based cards (RX 6900XT etc) and Navi 31 cards (RX 7900 XTX etc) are compatible with Pytorch on Linux. What they write about ROCm and Windows is equivocation. They target only one app: Blender. Pytorch+ROCm+Windows does not work. I had bought a 6900XT myself around launch time (the RTX3080 I ordered was not coming, it was the chip shortage times...) and it took around 2 years for Pytorch to become actually usable on it.
- thousand_nights 2y ago
  in practice everyone who wants to do ML at home buys nvidia and pays the premium
  - tonetegeatinst 2y ago
    Sad but true. Years ago, pre2018 nvidia was the goto hardware supplier if you were doing anything with neural networks. I remember CUDA being much more buggy back then but it still worked pretty good. Back then AMD wasn't considered a real competition for ML/AI hardware. Glad as always to see more competition in the market to drive innovations. AMD seems to be letting larger VRAM onto consumer cards, which is nice to see, just hope the AI/ML experience can get better for their software ecosystem.
- ryao 2y ago
  The obligatory link: https://xkcd.com/644/ https://xkcd.com/644/ That said, I would not expect it to stay working for long as long as ROCm is a dependency since AMD drops support for its older GPUs quickly while Nvidia continues to support older GPUs with less frequent legacy driver updates.
- sliken 2y ago
  Maybe, but it's pretty lame when you buy a new $500 CPU (7800 XT) and the docs say it's unsupported, and makes you feel like even if you reported a bug they would just say "Sorry, not supported". Did make me wish I bought a Nvidia.

sroussey 2y ago

You might just check out the Home Assistant Voice: https://ameridroid.com/products/home-assistant-voice-preview-edition https://ameridroid.com/products/home-assistant-voice-preview...

gigel82 2y ago
Yes, that's exactly what I was checking out. You need fast enough hardware to run the speech to text, text to speech and (most importantly) LLM locally: https://www.youtube.com/watch?v=XvbVePuP7NY https://www.youtube.com/watch?v=XvbVePuP7NY (he has dual 3090 GPUs but that's not a practical setup for most people - budget / power / noise).