3 ms·

While this might be true I’m worried about the hardware side of things. What if you have a good enough model but the cloud model providers are better in procur

by baxtr 5mo ago

While this might be true I’m worried about the hardware side of things. What if you have a good enough model but the cloud model providers are better in procuring hardware for interference?

gleenn 5mo ago
Local inference is definitely going to make more and more sense. Modern CPUs have all this amazing hardware well-optimized for inference purposes. I use a lot of web tools and see AI baked in and it feels weird. I want the smartness localized for speed and data security. I think and hope the industry points towards smart ai agents operating as locally as possible.
zozbot234 5mo ago
The cloud providers are probably better at procuring hardware for inference, but on prem users are better at repurposing hardware that they'd need anyway for their existing uses. In a world where AI compute is likely inherently scarce, it makes sense to rely on both.
Gigachad 5mo ago
You’ll be able to run the open models on any cloud at the cost of the hardware rental. While the closed models will try to mark up beyond the base cost.
pheggs 5mo ago
I personally believe that eventually manufacturers will want to sell more of their hardware and look for ways to sell hardware to consumers. isnt that situation quite similar to the days of early computers? I am for sure biased in hoping that will be the case
- fireant 5mo ago
  Perhaps for some very specific capabilities such as TTS, translation, voice recognition and so on. But for general intelligence models, better hardware just directly allows better models and that doesn't seem to be changing any time soon.
  - pheggs 5mo ago
    I'm pretty sure that's not linear, so I personally expect the benefits of larger models to diminish. The question is at what point that's the case. I guess a lot of variables play into it, but it is possible that the benefits of running larger models will be too expensive for the little benefit they provide

by baxtr 5mo ago

While this might be true I’m worried about the hardware side of things. What if you have a good enough model but the cloud model providers are better in procuring hardware for interference?

gleenn 5mo ago

Local inference is definitely going to make more and more sense. Modern CPUs have all this amazing hardware well-optimized for inference purposes. I use a lot of web tools and see AI baked in and it feels weird. I want the smartness localized for speed and data security. I think and hope the industry points towards smart ai agents operating as locally as possible.

zozbot234 5mo ago

The cloud providers are probably better at procuring hardware for inference, but on prem users are better at repurposing hardware that they'd need anyway for their existing uses. In a world where AI compute is likely inherently scarce, it makes sense to rely on both.

Gigachad 5mo ago

You’ll be able to run the open models on any cloud at the cost of the hardware rental. While the closed models will try to mark up beyond the base cost.

pheggs 5mo ago

I personally believe that eventually manufacturers will want to sell more of their hardware and look for ways to sell hardware to consumers. isnt that situation quite similar to the days of early computers? I am for sure biased in hoping that will be the case

fireant 5mo ago
Perhaps for some very specific capabilities such as TTS, translation, voice recognition and so on. But for general intelligence models, better hardware just directly allows better models and that doesn't seem to be changing any time soon.
- pheggs 5mo ago
  I'm pretty sure that's not linear, so I personally expect the benefits of larger models to diminish. The question is at what point that's the case. I guess a lot of variables play into it, but it is possible that the benefits of running larger models will be too expensive for the little benefit they provide