3 ms·
OK but I asked "Is this now the best hardware (or at least pareto frontier) for local LLMs?" and it sounds like you're mostly talking about software issues.
by ftxbro 3y ago
OK but I asked "Is this now the best hardware (or at least pareto frontier) for local LLMs?" and it sounds like you're mostly talking about software issues.
- brucethemoose2 3y agoHmmm... theoretically maybe? You can't argue with 192GB hooked up to a single GPU, even if the die is split. I'm not sure if it behaves more like a single GPU or 2 NVLink GPUs when running or training llms. But again, at that price point and with theoretical software support, you are starting to look at AMD MI300s, Intel GPU max/gaudi boxes, 4x 4090 boxes and such, so I am not sure.