4 ms·

I love LM studio but I’d never waste 12k like that. The memory bandwidth is too low trust me. Get the RTX Pro 6000 for 8.5k with double the bandwidth. It will

I love LM studio but I’d never waste 12k like that. The memory bandwidth is too low trust me. Get the RTX Pro 6000 for 8.5k with double the bandwidth. It will be way better

marci 1y ago
You can't run deepseek-v3/r1 on the RTX Pro 6000, not to mention the upcomming 1 million context qwen models, or the current qwen3-235b.
- 112233 1y ago
  I can run full deepseek r1 on m1 max with 64GB of ram. Around 0.5 t/s with small quant. Q4 quant of Maverick (253 GB) runs at 2.3 t/s on it (no GPU offload). Practically, last gen or even ES/QS EPYC or Xeon (with AMX), enough RAM to fill all 8 or 12 channels plus fast storage (4 Gen5 NVMEs are almost 60 GB/s) on paper at least look like cheapest way to run these huge MoE models at hobbyist speeds.
  - marci 1y ago
    If you're talking about Deepseek r1 with llama.cpp and mmap, then at this point you can run deepseek r1 on a raspberry zero with a 256GB micro sdcard and a phone charger. The only metric left to know is one's patience.
tymscar 1y ago
Why would they pay 2/3 of the price for something with 1/5 of ram? The whole point of spending that much money for them is to run massive models, like the full R1, which the Pro 6000 cant
- zackify 1y ago
  Because waiting forever for initial prompt processing with realistic number of MCP tools enabled on a prompt is going to suck without the most bandwidth possible And you are never going to sit around waiting for anything larger than the 96+gb of ram that the RTX pro has. If you’re using it for background tasks and not coding it’s a different story
  - johndough 1y ago
    If the MPC tools come first in the conversation, it should be technically possible to cache the activations, so you do not have to recompute them each time.
  - pests 1y ago
    Initial prompt processing with a large static context (system prompt + tools + whatever) could technically be improved by checkpointing the model state and reusing for future prompts. Not sure if any tools support this.
    - 112233 1y ago
      Dropping in late into this discussion, but is there any way to "comfortably" use multiple precomputed kv-caches with current models, in the style of this work: https://arxiv.org/abs/2212.10947 https://arxiv.org/abs/2212.10947 ? Meaning, I pre-parse multiple documents, and the prompt and completion attention sees all of them, but there is no attention between the documents (they are all encoded in the same overlapping positions). This way you can include basically unlimited amount of data in the prompt, paying for it with the perfomance.
  - tucnak 1y ago
    https://docs.vllm.ai/projects/production-stack/en/latest/tutorials/kv_cache.html https://docs.vllm.ai/projects/production-stack/en/latest/tut...
  - storus 1y ago
    M3 Ultra GPU is around 3070-3080 for the initial token processing. Not great, not terrible.
  - MangoToupe 1y ago
    > And you are never going to sit around waiting for anything larger than the 96+gb of ram that the RTX pro has. Am I the only person that gives aider instructions and leaves it alone for a few hours? This doesn't seem that difficult to integrate into my workflow.
t1amat 1y ago
(Replying to both siblings questioning this) If the primary use case is input heavy, which is true of agentic tools, there’s a world where partial GPU offload with many channels of DDR5 system RAM leads to an overall better experience. A good GPU will process input many times faster, and with good RAM you might end up with decent output speed still. Seems like that would come in close to $12k? And there would be no competition for models that do fit entirely inside that VRAM, for example Qwen3 32B.
storus 1y ago
RTX Pro 6000 can't do DeepSeek R1 671B Q4, you'd need 5-6 of them, which makes it way more expensive. Moreover, MacStudio will do it at 150W whereas Pro 6000 would start at 1500W.
- diggan 1y ago
  > Moreover, MacStudio will do it at 150W whereas Pro 6000 would start at 1500W. No, Pro 6000 pulls max 600W, not sure where you get 1500W from, that's more than double the specification. Besides, what is the token/second or second/token, and prompt processing speed for running DeepSeek R1 671B on a Mac Studio with Q4? Curious about those numbers, because I have a feeling they're very far off each other.
  - storus 1y ago
    You need at least 5x Pro 6000 (for smaller contexts), let's say Max-Q edition running at 300W, so overall you get a minimum of 1500W. You get around 6 tokens/second which is not great but not terrible. If you use very long prompts, things get bad.
smcleod 1y ago
RTX is nice, but it's memory limited and requires to have a full desktop machine to run it in. I'd take slower inference (as long as it's not less than 15tk/s) for more memory any day!
- diggan 1y ago
  I'd love to see more Very-Large-Memory Mac Studio benchmarks for prompt processing and inference. The few benchmarks I've seem either missed to take prompt processing into account, didn't share exact weights+setup that were used or showed really abysmal performance.
  - chisleu 1y ago
    Oh I plan to produce a ton of that. I'll post a blog on it to HN and /r/localllama when I'm done.
chisleu 1y ago
Only on HN can buying a $12k badass computer be a waste of money

I love LM studio but I’d never waste 12k like that. The memory bandwidth is too low trust me. Get the RTX Pro 6000 for 8.5k with double the bandwidth. It will be way better

marci 1y ago

You can't run deepseek-v3/r1 on the RTX Pro 6000, not to mention the upcomming 1 million context qwen models, or the current qwen3-235b.

112233 1y ago
I can run full deepseek r1 on m1 max with 64GB of ram. Around 0.5 t/s with small quant. Q4 quant of Maverick (253 GB) runs at 2.3 t/s on it (no GPU offload). Practically, last gen or even ES/QS EPYC or Xeon (with AMX), enough RAM to fill all 8 or 12 channels plus fast storage (4 Gen5 NVMEs are almost 60 GB/s) on paper at least look like cheapest way to run these huge MoE models at hobbyist speeds.
- marci 1y ago
  If you're talking about Deepseek r1 with llama.cpp and mmap, then at this point you can run deepseek r1 on a raspberry zero with a 256GB micro sdcard and a phone charger. The only metric left to know is one's patience.

tymscar 1y ago

Why would they pay 2/3 of the price for something with 1/5 of ram? The whole point of spending that much money for them is to run massive models, like the full R1, which the Pro 6000 cant

zackify 1y ago
Because waiting forever for initial prompt processing with realistic number of MCP tools enabled on a prompt is going to suck without the most bandwidth possible And you are never going to sit around waiting for anything larger than the 96+gb of ram that the RTX pro has. If you’re using it for background tasks and not coding it’s a different story
- johndough 1y ago
  If the MPC tools come first in the conversation, it should be technically possible to cache the activations, so you do not have to recompute them each time.
- pests 1y ago
  Initial prompt processing with a large static context (system prompt + tools + whatever) could technically be improved by checkpointing the model state and reusing for future prompts. Not sure if any tools support this.
  - 112233 1y ago
    Dropping in late into this discussion, but is there any way to "comfortably" use multiple precomputed kv-caches with current models, in the style of this work: https://arxiv.org/abs/2212.10947 https://arxiv.org/abs/2212.10947 ? Meaning, I pre-parse multiple documents, and the prompt and completion attention sees all of them, but there is no attention between the documents (they are all encoded in the same overlapping positions). This way you can include basically unlimited amount of data in the prompt, paying for it with the perfomance.
- tucnak 1y ago
  https://docs.vllm.ai/projects/production-stack/en/latest/tutorials/kv_cache.html https://docs.vllm.ai/projects/production-stack/en/latest/tut...
- storus 1y ago
  M3 Ultra GPU is around 3070-3080 for the initial token processing. Not great, not terrible.
- MangoToupe 1y ago
  > And you are never going to sit around waiting for anything larger than the 96+gb of ram that the RTX pro has. Am I the only person that gives aider instructions and leaves it alone for a few hours? This doesn't seem that difficult to integrate into my workflow.

t1amat 1y ago

(Replying to both siblings questioning this) If the primary use case is input heavy, which is true of agentic tools, there’s a world where partial GPU offload with many channels of DDR5 system RAM leads to an overall better experience. A good GPU will process input many times faster, and with good RAM you might end up with decent output speed still. Seems like that would come in close to $12k? And there would be no competition for models that do fit entirely inside that VRAM, for example Qwen3 32B.

storus 1y ago

RTX Pro 6000 can't do DeepSeek R1 671B Q4, you'd need 5-6 of them, which makes it way more expensive. Moreover, MacStudio will do it at 150W whereas Pro 6000 would start at 1500W.

diggan 1y ago
> Moreover, MacStudio will do it at 150W whereas Pro 6000 would start at 1500W. No, Pro 6000 pulls max 600W, not sure where you get 1500W from, that's more than double the specification. Besides, what is the token/second or second/token, and prompt processing speed for running DeepSeek R1 671B on a Mac Studio with Q4? Curious about those numbers, because I have a feeling they're very far off each other.
- storus 1y ago
  You need at least 5x Pro 6000 (for smaller contexts), let's say Max-Q edition running at 300W, so overall you get a minimum of 1500W. You get around 6 tokens/second which is not great but not terrible. If you use very long prompts, things get bad.

smcleod 1y ago

RTX is nice, but it's memory limited and requires to have a full desktop machine to run it in. I'd take slower inference (as long as it's not less than 15tk/s) for more memory any day!

diggan 1y ago
I'd love to see more Very-Large-Memory Mac Studio benchmarks for prompt processing and inference. The few benchmarks I've seem either missed to take prompt processing into account, didn't share exact weights+setup that were used or showed really abysmal performance.
- chisleu 1y ago
  Oh I plan to produce a ton of that. I'll post a blog on it to HN and /r/localllama when I'm done.

chisleu 1y ago

Only on HN can buying a $12k badass computer be a waste of money