3 ms·

> If someone put in frontier AI models from like .... last june I guess? in a box and let me run it with "decent" token throughput I would be happy. You can ha

> If someone put in frontier AI models from like .... last june I guess? in a box and let me run it with "decent" token throughput I would be happy. You can have that! Qwen 3.8 Flash-Next is ~Opus 4.6 and runs nicely on a DGX Spark. And that’s just an architecture preview. The Qwen 4 family is expected to arrive this fall.

rtpg 13d ago
DGX Spark is a biiiiit costly but neat to hear! Do you know what kinda throughput you’re getting on that kinda setup? (I have a secondary problem of being “locked into” Claude Code by it being good enough for me, I’d probably need to investigate the other harnesses… my impression is other harnesses are a bit more aggressively OK with nuking your setup from orbit)
- anon373839 13d ago
  It is costly, especially right now. I don’t think you can make a case for it on cost savings! The throughput in a single stream is about 50 tokens/sec (a bit less for prose, a bit more for code due to speculative draft acceptance rates) and about 2,000 tokens/sec for prefill. Both numbers are flat and stable as context accumulates. That’s what finally tilted me away from the Mac Studio despite its much superior memory bandwidth. I think these numbers may improve because the model is pretty new and optimizations aren’t done.
  - int_19h 13d ago
    I don't think you can ever make a case for it on cost savings in general. Inference is very obviously the kind of problem where things are cheaper at scale, and this is still true for smaller models. The only reason to run locally is privacy.
    - anon373839 13d ago
      Privacy is a great reason, but independence is another. It’s very nice knowing that you’re going to get the same reliable product every time you call the model. Nothing is going to change unless you decide to change it.
      - int_19h 13d ago
        That is not a counterargument to cloud inference though. You can also run open weight models in the cloud, and it's still cheaper. So privacy really is the only motivation to run on local hardware.
        anon373839 12d ago
        Ah, no, that’s not cheaper. Renting GPUs adds up quickly and leaves you with nothing in the end. Renting tokens from open model providers is cheaper but it incurs the same issues: unexpected changes in model quality, inconsistent speeds, service outages.
        int_19h 10d ago
        Renting tokens is exactly what I had in mind, and I don't see how it has the same issues. Sure, if you go for absolute bottom of the barrel than you can expect "changes in model quality", but in general that is not so, and you can pin specific model releases etc. Inconsistent speeds and service outages apply to your own hardware as well (say, power goes down). This all isn't even unique to AI, it's generally why cloud compute quickly became dominant.
    - rtpg 13d ago
      Right now the "subsidies" etc I think make the calculus really tough, but for general compute.... for example running CI just on a Mac Mini can get you real cost effective throughput compared to running CI on GH runners and whatnot. Things get cheaper at scale but that's where the provider's margins come in! I do think there's also an interesting idea: you buy a box like this and run it at a fixed-ish cost (well, electricity). Your demand goes up but your supply is fixed... and that back pressure means that you still have good cost control. With cloud providers it's a _biiiiit_ too easy to just increase spend. Sometimes it's OK for things to just be slow.

rtpg 13d ago

DGX Spark is a biiiiit costly but neat to hear! Do you know what kinda throughput you’re getting on that kinda setup? (I have a secondary problem of being “locked into” Claude Code by it being good enough for me, I’d probably need to investigate the other harnesses… my impression is other harnesses are a bit more aggressively OK with nuking your setup from orbit)

anon373839 13d ago
It is costly, especially right now. I don’t think you can make a case for it on cost savings! The throughput in a single stream is about 50 tokens/sec (a bit less for prose, a bit more for code due to speculative draft acceptance rates) and about 2,000 tokens/sec for prefill. Both numbers are flat and stable as context accumulates. That’s what finally tilted me away from the Mac Studio despite its much superior memory bandwidth. I think these numbers may improve because the model is pretty new and optimizations aren’t done.
- int_19h 13d ago
  I don't think you can ever make a case for it on cost savings in general. Inference is very obviously the kind of problem where things are cheaper at scale, and this is still true for smaller models. The only reason to run locally is privacy.
  - anon373839 13d ago
    Privacy is a great reason, but independence is another. It’s very nice knowing that you’re going to get the same reliable product every time you call the model. Nothing is going to change unless you decide to change it.
    - int_19h 13d ago
      That is not a counterargument to cloud inference though. You can also run open weight models in the cloud, and it's still cheaper. So privacy really is the only motivation to run on local hardware.
      - anon373839 12d ago
        Ah, no, that’s not cheaper. Renting GPUs adds up quickly and leaves you with nothing in the end. Renting tokens from open model providers is cheaper but it incurs the same issues: unexpected changes in model quality, inconsistent speeds, service outages.
        int_19h 10d ago
        Renting tokens is exactly what I had in mind, and I don't see how it has the same issues. Sure, if you go for absolute bottom of the barrel than you can expect "changes in model quality", but in general that is not so, and you can pin specific model releases etc. Inconsistent speeds and service outages apply to your own hardware as well (say, power goes down). This all isn't even unique to AI, it's generally why cloud compute quickly became dominant.
  - rtpg 13d ago
    Right now the "subsidies" etc I think make the calculus really tough, but for general compute.... for example running CI just on a Mac Mini can get you real cost effective throughput compared to running CI on GH runners and whatnot. Things get cheaper at scale but that's where the provider's margins come in! I do think there's also an interesting idea: you buy a box like this and run it at a fixed-ish cost (well, electricity). Your demand goes up but your supply is fixed... and that back pressure means that you still have good cost control. With cloud providers it's a _biiiiit_ too easy to just increase spend. Sometimes it's OK for things to just be slow.