4 ms·

What use are weights without the hardware to run them? That's the gate. Local AI right now is a toy in comparison. Nukes are actually a great example of someth

What use are weights without the hardware to run them? That's the gate. Local AI right now is a toy in comparison. Nukes are actually a great example of something also gated by resources. Just having the knowledge/plans isn't good enough.

reactordev 8mo ago
I run local models on Mac studios and they are more than capable. Don’t spread fud.
- bottlepalm 8mo ago
  You're spreading fud. There's nothing you can run locally that's on par with the speed/intelligence of a SOTA model.
  - 3836293648 8mo ago
    You may be correct about the level of models you can actually run on consumer hardware, but it's not fud and you're being needlessly aggressive here.
  - CamperBob2 8mo ago
    Incorrect as of a couple of days ago, when Qwen 3.5 came out. It's a GPT 5-class model that you can run at full strength on a small DGX Spark or Mac cluster, and it still works pretty well after quantization.
- deleted 8mo ago
  [deleted]
- xpe 8mo ago
  My take on the parent (^) and grandparent (^^): >> Local AI right now is a toy in comparison. Charitable interpretation: Local AI (unclear; maybe gpt-oss-120b) isn't nearly as good as SoTA (unstated; perhaps Claude Opus 4.6). Unstated use case(s). > I run local models on Mac studios and they are more than capable. Don't spread fud. Charitable interpretation: On their Mac studio (could be a cluster or single machine: unclear), local models (unclear; maybe gpt-oss-120b, maybe not) are capable for their needs. Unstated use case(s). / The "Don't spread fud." advocates for accurate information, which is a useful goal in general. However, it was uncharitable and brusque. An alternative approach would have been to ask a clarification question. > Please respond to the strongest plausible interpretation of what someone says, not a weaker one that's easier to criticize. Assume good faith. - HN Guidelines I promise I wrote this by hand. If you confidently thought otherwise, then I would kindly ask you to read my about page.
fooker 8mo ago
> hardware to run them Costs a few hundred thousand per server, it's a huge expense if you want it at your home but a rounding error for most organizations.
- bottlepalm 8mo ago
  You're buying what exactly for a few hundred thousand? and running what model on it? to support how many users? at what tps?
  - fooker 8mo ago
    Not every use case is a cloud provider or tech giant. Newer Blackwell does 200+ tokens per second on the largest models and tens of thousands on the smaller models. Most military applications require fast smaller models, I'd imagine. Also, custom chips are reportedly approaching an order of magnitude more for the price. It's a matter of availability right now, but that will be solved at some point.
txrx0000 8mo ago
Scaling has hit a wall and will not get us to AGI. Open-source models are only a couple of months behind closed models, and the same level of capability will require smaller and smaller models in the future. This is where open research can help: make the models smaller ASAP. I think it's likely that we'll be able to get something human-level to run on a single 16GB GPU before the end of the decade.
- tbrownaw 8mo ago
  > human-level to run on a single 16GB GPU before the end of the decade. That's apparently about 6k books' worth of data.
  - txrx0000 8mo ago
    For the weights and temporary state, yes. It doesn't sound like a lot until you remember that your DNA is about 600 books worth of data by the same metric.
  - octoberfranklin 8mo ago
    How many humans do you know who can recite 6000 books, word for word, exactly?
- drdaeman 8mo ago
  > Open-source models are only a couple of months behind closed models Oh, come on, surely not just a couple months. Benchmarks may boast some fancy numbers, but I just tried to save some money by trying out Qwen3-Next 80B and Qwen3.5 35B-A3B (since I've recently got a machine that can run those at a tolerable speed) to generate some documentation from a messy legacy codebase. It was nowhere close neither in the output quality nor in performance to any current models that the SaaS LLM behemoth corps offer. Just an anecdote, of course, but that's all I have.
- Tade0 8mo ago
  > Scaling has hit a wall and will not get us to AGI. That was never the aim. LLMs are not designed to be generally intelligent, just to be really good at producing believable text.

reactordev 8mo ago

I run local models on Mac studios and they are more than capable. Don’t spread fud.

bottlepalm 8mo ago
You're spreading fud. There's nothing you can run locally that's on par with the speed/intelligence of a SOTA model.
- 3836293648 8mo ago
  You may be correct about the level of models you can actually run on consumer hardware, but it's not fud and you're being needlessly aggressive here.
- CamperBob2 8mo ago
  Incorrect as of a couple of days ago, when Qwen 3.5 came out. It's a GPT 5-class model that you can run at full strength on a small DGX Spark or Mac cluster, and it still works pretty well after quantization.
deleted 8mo ago
[deleted]
xpe 8mo ago
My take on the parent (^) and grandparent (^^): >> Local AI right now is a toy in comparison. Charitable interpretation: Local AI (unclear; maybe gpt-oss-120b) isn't nearly as good as SoTA (unstated; perhaps Claude Opus 4.6). Unstated use case(s). > I run local models on Mac studios and they are more than capable. Don't spread fud. Charitable interpretation: On their Mac studio (could be a cluster or single machine: unclear), local models (unclear; maybe gpt-oss-120b, maybe not) are capable for their needs. Unstated use case(s). / The "Don't spread fud." advocates for accurate information, which is a useful goal in general. However, it was uncharitable and brusque. An alternative approach would have been to ask a clarification question. > Please respond to the strongest plausible interpretation of what someone says, not a weaker one that's easier to criticize. Assume good faith. - HN Guidelines I promise I wrote this by hand. If you confidently thought otherwise, then I would kindly ask you to read my about page.

fooker 8mo ago

> hardware to run them Costs a few hundred thousand per server, it's a huge expense if you want it at your home but a rounding error for most organizations.

bottlepalm 8mo ago
You're buying what exactly for a few hundred thousand? and running what model on it? to support how many users? at what tps?
- fooker 8mo ago
  Not every use case is a cloud provider or tech giant. Newer Blackwell does 200+ tokens per second on the largest models and tens of thousands on the smaller models. Most military applications require fast smaller models, I'd imagine. Also, custom chips are reportedly approaching an order of magnitude more for the price. It's a matter of availability right now, but that will be solved at some point.

txrx0000 8mo ago

Scaling has hit a wall and will not get us to AGI. Open-source models are only a couple of months behind closed models, and the same level of capability will require smaller and smaller models in the future. This is where open research can help: make the models smaller ASAP. I think it's likely that we'll be able to get something human-level to run on a single 16GB GPU before the end of the decade.

tbrownaw 8mo ago
> human-level to run on a single 16GB GPU before the end of the decade. That's apparently about 6k books' worth of data.
- txrx0000 8mo ago
  For the weights and temporary state, yes. It doesn't sound like a lot until you remember that your DNA is about 600 books worth of data by the same metric.
- octoberfranklin 8mo ago
  How many humans do you know who can recite 6000 books, word for word, exactly?
drdaeman 8mo ago
> Open-source models are only a couple of months behind closed models Oh, come on, surely not just a couple months. Benchmarks may boast some fancy numbers, but I just tried to save some money by trying out Qwen3-Next 80B and Qwen3.5 35B-A3B (since I've recently got a machine that can run those at a tolerable speed) to generate some documentation from a messy legacy codebase. It was nowhere close neither in the output quality nor in performance to any current models that the SaaS LLM behemoth corps offer. Just an anecdote, of course, but that's all I have.
Tade0 8mo ago
> Scaling has hit a wall and will not get us to AGI. That was never the aim. LLMs are not designed to be generally intelligent, just to be really good at producing believable text.