4 ms·

I think this would make sense for consumer hardware, not for AI companies. AI companies constantly update/change stuff, new models come out, new requirements,

by XCSme 2mo ago

I think this would make sense for consumer hardware, not for AI companies. AI companies constantly update/change stuff, new models come out, new requirements, etc. But if you ship an "ai-powered" dishwasher, it can come with the chip built-in to do computer vision and precisely target each spot, and will be sold as-is with no updates.

m463 2mo ago
subscription "ai-powered" dishwasher with personalized user ads, most of the chip dedicated to "personalized" not spots.
- XCSme 2mo ago
  So local personalized ads? Not sure if that's better or worse than online personalizaed ads...
throwaway173738 2mo ago
You don’t need this chip to do that. Computer vision has used machine learning for decades. The task you’re describing is pretty rudimentary and an off the shelf model with a control system would do it way cheaper.
- XCSme 2mo ago
  It was just a random example, you could think of it as being a lot more complex (detect which type of food it is, what detergent to use, how much water, remember patterns, learn over time, adapt, etc.)
- tyre 2mo ago
  Think of a HomePod. 99% (and likely much more) of what people are asking is super simple.
yumraj 2mo ago
Makes sense. Actually to expand, I believe this can make a lot of sense for industrial robots and such which have a more or less fixed job and latency matters more, so a well tested model may be more valuable than need to keep updating them
thunderbird120 2mo ago
It makes fairly little sense for consumer hardware, unless you're embedding an absolutely tiny model. Since you're embedding the entire model in the chip itself, required chip size scales directly with model size. The reason they have llama3.1-8B as their model in https://chatjimmy.ai/ https://chatjimmy.ai/ (their demo) is because it's very hard to do anything bigger than that, you literally start running out of physical space on the chip, and bigger chips cost money. You can't have a dishwasher using a chip the size of a B200 die to run a model you could run on a Raspberry Pi. What's worse is that this is when it's already quantized to ~3 bits per parameter (which is fairly lobotomized). Yes, the chip will run it 1000X faster than the Raspberry Pi, but it will only be stupid faster. Their press release explicitly lists that their HC1 puts the entire Llama 3.1 8B model on one 815 mm² TSMC N6 die, with about 53 billion transistors. 815 mm² is fucking huge. An RTX 5090 is 750 mm². A mid-to-high end consumer CPU die is something like 300 mm², with a lot of budget parts being significantly smaller than that, down to ~70 mm². Every square mm costs money. If they upgraded from N6 to N3P they MIGHT be able to get as much as a ~35B class model on a a chip which fits in the reticle limit, maybe, probably not, because model weights aren't the only thing that needs to fit on the chip. There are very serious issues with agentic performance in this setup, which is exactly where you would want something really fast. Their Llama 3.1 demo lists a context of 6,144, which is dramatically lower than the 131,072 Llama-3.1 supports. Reasoning models are barely usable with contexts that short. The reason for this is that to actually get those speeds, the KV cache needs to live in SRAM. You can't bake the KV cache into the circuitry since it... changes. They clearly don't have enough SRAM, and the problem gets worse the bigger you make the model since KV cache grows (sort of) with model dim. The longer you want to make your context, the more of your chip needs to be SRAM. Frankly, I don't see the use-case for this tech. It's too expensive and too inflexible. Just doing what Cerebras did and making a wafer-sized chip which is mostly SRAM is a much better solution to serving LLMs at extreme speeds and you don't need to make a new chip every time a new model comes out.
- hypfer 2mo ago
  And yield. Good luck getting good yield out of such ginormous chips. That said, I've been wondering if they could go with multiple smaller ones instead. Like one per layer maybe even? What are your thoughts on that? You seem to be more qualified than me on that matter.

by XCSme 2mo ago

m463 2mo ago

subscription "ai-powered" dishwasher with personalized user ads, most of the chip dedicated to "personalized" not spots.

XCSme 2mo ago
So local personalized ads? Not sure if that's better or worse than online personalizaed ads...

throwaway173738 2mo ago

You don’t need this chip to do that. Computer vision has used machine learning for decades. The task you’re describing is pretty rudimentary and an off the shelf model with a control system would do it way cheaper.

XCSme 2mo ago
It was just a random example, you could think of it as being a lot more complex (detect which type of food it is, what detergent to use, how much water, remember patterns, learn over time, adapt, etc.)
tyre 2mo ago
Think of a HomePod. 99% (and likely much more) of what people are asking is super simple.

yumraj 2mo ago

Makes sense. Actually to expand, I believe this can make a lot of sense for industrial robots and such which have a more or less fixed job and latency matters more, so a well tested model may be more valuable than need to keep updating them

thunderbird120 2mo ago

It makes fairly little sense for consumer hardware, unless you're embedding an absolutely tiny model. Since you're embedding the entire model in the chip itself, required chip size scales directly with model size. The reason they have llama3.1-8B as their model in https://chatjimmy.ai/ https://chatjimmy.ai/ (their demo) is because it's very hard to do anything bigger than that, you literally start running out of physical space on the chip, and bigger chips cost money. You can't have a dishwasher using a chip the size of a B200 die to run a model you could run on a Raspberry Pi. What's worse is that this is when it's already quantized to ~3 bits per parameter (which is fairly lobotomized). Yes, the chip will run it 1000X faster than the Raspberry Pi, but it will only be stupid faster. Their press release explicitly lists that their HC1 puts the entire Llama 3.1 8B model on one 815 mm² TSMC N6 die, with about 53 billion transistors. 815 mm² is fucking huge. An RTX 5090 is 750 mm². A mid-to-high end consumer CPU die is something like 300 mm², with a lot of budget parts being significantly smaller than that, down to ~70 mm². Every square mm costs money. If they upgraded from N6 to N3P they MIGHT be able to get as much as a ~35B class model on a a chip which fits in the reticle limit, maybe, probably not, because model weights aren't the only thing that needs to fit on the chip. There are very serious issues with agentic performance in this setup, which is exactly where you would want something really fast. Their Llama 3.1 demo lists a context of 6,144, which is dramatically lower than the 131,072 Llama-3.1 supports. Reasoning models are barely usable with contexts that short. The reason for this is that to actually get those speeds, the KV cache needs to live in SRAM. You can't bake the KV cache into the circuitry since it... changes. They clearly don't have enough SRAM, and the problem gets worse the bigger you make the model since KV cache grows (sort of) with model dim. The longer you want to make your context, the more of your chip needs to be SRAM. Frankly, I don't see the use-case for this tech. It's too expensive and too inflexible. Just doing what Cerebras did and making a wafer-sized chip which is mostly SRAM is a much better solution to serving LLMs at extreme speeds and you don't need to make a new chip every time a new model comes out.

hypfer 2mo ago
And yield. Good luck getting good yield out of such ginormous chips. That said, I've been wondering if they could go with multiple smaller ones instead. Like one per layer maybe even? What are your thoughts on that? You seem to be more qualified than me on that matter.