4 ms·

RTX 6000 Pro retails for $10k so an 8x is $80k before anything else in the computer, and long-context will have... pretty bad performance (20+ seconds of waitin

RTX 6000 Pro retails for $10k so an 8x is $80k before anything else in the computer, and long-context will have... pretty bad performance (20+ seconds of waiting before any tokens come out), but it's true it technically works. I don't think cloud models are going away; the hardware for good perf is expensive and higher param count models will remain smarter for a looong time. Even if the hardware cost for kind-of-usable perf fell to only $10k, cloud ones will be way faster and you'd need a lot of tokens to break even.

zozbot234 5mo ago
> I don't think cloud models are going away; the hardware for good perf is expensive I think local AI will win in its niche by repurposing users' existing hardware, especially as cloud hardware itself gets increasingly bottlenecked in all sorts of ways and the price of cloud tokens rises. You don't have to care about "bad" performance when you've got dedicated hardware that runs your workloads 24/7. Time-critical work that also requires the latest and greatest model can stay on the cloud, but a vast amount of AI work just isn't that critical.
- ai_fry_ur_brain 5mo ago
  [flagged]
  - CamperBob2 5mo ago
    Other sites beckon.
- reissbaker 5mo ago
  Users do not have an existing $80k of hardware, are not going to buy $80k of hardware for worse performance than paying $100/month, and models are continuing to grow in size while memory grows in price.
  - otabdeveloper4 5mo ago
    > paying $100/month There will not ever be a monthly subscription for LLM tokens. The economics isn't there. Local tokens will always be cheaper.
    - entrope 5mo ago
      What's the basis for saying local tokens will always be cheaper? As others have outlined, LLMs serving one user at a time are pretty expensive, but concurrent users become much more cost-effective (assuming there's enough RAM for the contexts). If "local" to you means ~10 hours daily use by a team of employees, the company still has to balance against cloud services that can amortize non-recurring costs over 24 hours of service per day.
      - zozbot234 5mo ago
        Why would a team of employees not be able to run AI workloads 24/7? Not all workloads are time sensitive.
        entrope 5mo ago
        Both my experience, and Anthropic's off-peak promotion, indicate that there are very uneven levels of demand for peak hours versus off-peak hours. How close do you think they are?
        zozbot234 5mo ago
        But that's demand for cloud inference that's priced on a flat-rate basis with some adjustments (like "off-peak hours"). Not a local rig where inference is effectively free aside from the cost of power whenever the system isn't congested.
        reissbaker 5mo ago
        The local rig is not free and requires very large capital expenditures while producing very low token throughput for large models. Within any time budget, you can get many orders of magnitude more large-model tokens off an 8xB200 than off a local rig. Therefore cloud tokens have a huge capital efficiency advantage over local rigs. That will continue basically forever, since there will always be large cloud companies willing to spend millions of dollars for more capital-efficient hardware, so Nvidia and friends will continue to spare no expense producing it, meaning the cloud hardware will be way too expensive if you're not a large inference company. You can also buy local rigs, but they will be less capital efficient per token, not more. (This is a generous argument: it also ignores the massive software stack optimization the cloud companies do that doesn't trickle down to local-rig-sized deployments; for example, prefill/decode disaggregation, which would double the VRAM requirements for a local rig — if you could even do it on a local rig, which you can't, because local rigs don't have Infiniband. But at scale, prefill/decode disaggregation improves capital efficiency, since you can tune the compute-bound prefill node differently than the memory-bound decode node.) The advantage of local rigs is not capital-efficient tokens. It's privacy. But then again, you can get zero-data-retention options from many inference companies, so for many use cases it may not matter unless you need strict guarantees the data never leaves the building...
otabdeveloper4 5mo ago
> higher param count models will remain smarter for a looong time They're not smarter, they just know more stuff. You probably don't need knowledge about Pokemon or the Diamond Sutra in your enterprise coding LLM. The "smarts" comes from post-training, especially around tool use.
- anon7725 5mo ago
  If the smarts came from post-training, we could show significant gains by doing that post-training again for previous generations of models. But we know that isn’t happening - effective post training is necessary but not sufficient for model performance.
  - otabdeveloper4 5mo ago
    > we could show significant gains by doing that post-training again for previous generations of models That's what Chinese models are doing, and beating Opus et al.
- CamperBob2 5mo ago
  You probably don't need knowledge about Pokemon or the Diamond Sutra in your enterprise coding LLM. That's one of the biggest remaining head-scratchers in this whole business. You do need all that unrelated stuff to make a good coding model. Nobody knows why you can't build a coding model by training on nothing but code, CS texts, specifications, and case studies, but so far it appears that you can't.
  - otabdeveloper4 5mo ago
    This one is kind of obvious - because people prompt coding LLMs with natural language. That's unrelated to stuffing the pre-train set with trivia factoids. An LLM that knows English very well isn't actually very large and certainly not hundreds of billions of parameters.
alfiedotwtf 5mo ago
If 8 x RTX 6000 is getting you 20s before initial token, how are cloud vendors doing this?
- CamperBob2 5mo ago
  RTX6000s are great but they are several times slower than a real datacenter-grade GPU. They still use DDR memory rather than HBM, for example.

zozbot234 5mo ago

> I don't think cloud models are going away; the hardware for good perf is expensive I think local AI will win in its niche by repurposing users' existing hardware, especially as cloud hardware itself gets increasingly bottlenecked in all sorts of ways and the price of cloud tokens rises. You don't have to care about "bad" performance when you've got dedicated hardware that runs your workloads 24/7. Time-critical work that also requires the latest and greatest model can stay on the cloud, but a vast amount of AI work just isn't that critical.

ai_fry_ur_brain 5mo ago
[flagged]
- CamperBob2 5mo ago
  Other sites beckon.
reissbaker 5mo ago
Users do not have an existing $80k of hardware, are not going to buy $80k of hardware for worse performance than paying $100/month, and models are continuing to grow in size while memory grows in price.
- otabdeveloper4 5mo ago
  > paying $100/month There will not ever be a monthly subscription for LLM tokens. The economics isn't there. Local tokens will always be cheaper.
  - entrope 5mo ago
    What's the basis for saying local tokens will always be cheaper? As others have outlined, LLMs serving one user at a time are pretty expensive, but concurrent users become much more cost-effective (assuming there's enough RAM for the contexts). If "local" to you means ~10 hours daily use by a team of employees, the company still has to balance against cloud services that can amortize non-recurring costs over 24 hours of service per day.
    - zozbot234 5mo ago
      Why would a team of employees not be able to run AI workloads 24/7? Not all workloads are time sensitive.
      - entrope 5mo ago
        Both my experience, and Anthropic's off-peak promotion, indicate that there are very uneven levels of demand for peak hours versus off-peak hours. How close do you think they are?
        zozbot234 5mo ago
        But that's demand for cloud inference that's priced on a flat-rate basis with some adjustments (like "off-peak hours"). Not a local rig where inference is effectively free aside from the cost of power whenever the system isn't congested.
        reissbaker 5mo ago
        The local rig is not free and requires very large capital expenditures while producing very low token throughput for large models. Within any time budget, you can get many orders of magnitude more large-model tokens off an 8xB200 than off a local rig. Therefore cloud tokens have a huge capital efficiency advantage over local rigs. That will continue basically forever, since there will always be large cloud companies willing to spend millions of dollars for more capital-efficient hardware, so Nvidia and friends will continue to spare no expense producing it, meaning the cloud hardware will be way too expensive if you're not a large inference company. You can also buy local rigs, but they will be less capital efficient per token, not more. (This is a generous argument: it also ignores the massive software stack optimization the cloud companies do that doesn't trickle down to local-rig-sized deployments; for example, prefill/decode disaggregation, which would double the VRAM requirements for a local rig — if you could even do it on a local rig, which you can't, because local rigs don't have Infiniband. But at scale, prefill/decode disaggregation improves capital efficiency, since you can tune the compute-bound prefill node differently than the memory-bound decode node.) The advantage of local rigs is not capital-efficient tokens. It's privacy. But then again, you can get zero-data-retention options from many inference companies, so for many use cases it may not matter unless you need strict guarantees the data never leaves the building...

otabdeveloper4 5mo ago

> higher param count models will remain smarter for a looong time They're not smarter, they just know more stuff. You probably don't need knowledge about Pokemon or the Diamond Sutra in your enterprise coding LLM. The "smarts" comes from post-training, especially around tool use.

anon7725 5mo ago
If the smarts came from post-training, we could show significant gains by doing that post-training again for previous generations of models. But we know that isn’t happening - effective post training is necessary but not sufficient for model performance.
- otabdeveloper4 5mo ago
  > we could show significant gains by doing that post-training again for previous generations of models That's what Chinese models are doing, and beating Opus et al.
CamperBob2 5mo ago
You probably don't need knowledge about Pokemon or the Diamond Sutra in your enterprise coding LLM. That's one of the biggest remaining head-scratchers in this whole business. You do need all that unrelated stuff to make a good coding model. Nobody knows why you can't build a coding model by training on nothing but code, CS texts, specifications, and case studies, but so far it appears that you can't.
- otabdeveloper4 5mo ago
  This one is kind of obvious - because people prompt coding LLMs with natural language. That's unrelated to stuffing the pre-train set with trivia factoids. An LLM that knows English very well isn't actually very large and certainly not hundreds of billions of parameters.

alfiedotwtf 5mo ago

If 8 x RTX 6000 is getting you 20s before initial token, how are cloud vendors doing this?

CamperBob2 5mo ago
RTX6000s are great but they are several times slower than a real datacenter-grade GPU. They still use DDR memory rather than HBM, for example.