3 ms·

This would be a very interesting future. I can imagine Gemma 5 Mini running locally on hardware, or a hard-coded "AI core" like an ALU or media processor that s

by byran 7mo ago

This would be a very interesting future. I can imagine Gemma 5 Mini running locally on hardware, or a hard-coded "AI core" like an ALU or media processor that supports particular encoding mechanisms like H.264, AV1, etc. Other than the obvious costs (but Taalas seems to be bringing back the structured ASIC era so costs shouldn't be that low [1]), I'm curious why this isn't getting much attention from larger companies. Of course, this wouldn't be useful for training models but as the models further improve, I can totally see this inside fully local + ultrafast + ultra efficient processors. [1] https://en.wikipedia.org/wiki/Structured_ASIC_platform https://en.wikipedia.org/wiki/Structured_ASIC_platform

roncesvalles 7mo ago
Well even programmable ASICs like Cerebras and Groq give many-multiples speedup over GPUs and the market has hardly reacted at all.
- mips_avatar 7mo ago
  The problem with groq was they only allowed LORA on llama 8b and 70b, and you had to have an enterprise contract it wasn't self service.
- fooker 7mo ago
  > market has hardly reacted at all Guess who acqui-hired Groq to push this into GPUs? The name GPU has been an anachronism for a couple of years now.
  - fuck_google 7mo ago
    [dead]
- brainless 7mo ago
  Seems both Nvidia (Groq) and OpenAI (Codex Spark) are now invested in the ASIC route one way or another.
- IshKebab 7mo ago
  Cerebras gives a many multiple speedup but it's also many multiples more expensive.
JKCalhoun 7mo ago
Apple should have done this yesterday. A local AI on my phone/Macbook is all I really want from this tech. The cloud-based AI (OpenAI, etc.) are todays AOL.
- post-it 7mo ago
  The hardware isn't there yet. Apple's neural engine is neat and has some uses but it just isn't in the same league as Claude right now. We'll get there.
- Aurornis 7mo ago
  The die size is huge. This isn’t the kind of chip that would go into your MacBook, let alone an iPhone. It’s for cloud based servers.
  - adeelk93 7mo ago
    And computers used to be the size of a room. I think they can get it to iPhone size in the future, this is an early prototype.
    - MarsIronPI 7mo ago
      Well, there's a limit to how small we can make transistors with our current technology. As I understand it, Intel is already running into those limits with their new CPUs (they had to redesign the fins IIRC). I can imagine that without an actual breakthrough in chip manufacturing the size could stay large. That's not to say that a breakthrough won't happen, though.
      - layla5alive 7mo ago
        Yes, in 2D, but NAND has been using layers for a while. We call HBM interposers 2.5D. 3D breakthrough would be pretty easy but for those pesky problems like power delivery and cooling. (/s) But give that time (e.g. microfluidics) - something interesting is that it would be extra hard to use all layers at once, but NN might be a good fit, imagining that computation will be sparse (subsets activating simultaneously)...
RobotToaster 7mo ago
> I'm curious why this isn't getting much attention from larger companies. I can see two potential reasons: 1) Most of the big players seem convinced that AI is going to continue to improve at the rate it did in 2025, if their assumption is somehow correct by the time any chip entered mass production it would be obsolete. 2) The business model of the big players is to sell expensive subscriptions, and train on and sell the data you give it. Chips that allow for relatively inexpensive offline AI aren't conducive to that.
- deleted 7mo ago
  [deleted]
deleted 7mo ago
[deleted]
hrn_frs 7mo ago
> I'm curious why this isn't getting much attention from larger companies. Time is money and when you're competing with multiple companies with little margin for error you'll focus all your effort into releasing things quickly. This chip is "only" a performance boost. It will unlock a lot of potential, but startups can't divide their attention like this. Big companies like google are surely already investigating this venue, but they might lack hardware expertise.
theptip 7mo ago
> I'm curious why this isn't getting much attention from larger companies I would be shocked if Google isn’t working on this right now. They build their own TPUs, this is an extremely obvious direction from there. (And there are plenty of interesting co-design questions that only the frontier labs can dabble with; Taalas is stuck working around architectural quirks like “top-8 MoE”, Google can just rework the architecture hyperparameters to whatever gets best results in silico.)

by byran 7mo ago

roncesvalles 7mo ago

Well even programmable ASICs like Cerebras and Groq give many-multiples speedup over GPUs and the market has hardly reacted at all.

mips_avatar 7mo ago
The problem with groq was they only allowed LORA on llama 8b and 70b, and you had to have an enterprise contract it wasn't self service.
fooker 7mo ago
> market has hardly reacted at all Guess who acqui-hired Groq to push this into GPUs? The name GPU has been an anachronism for a couple of years now.
- fuck_google 7mo ago
  [dead]
brainless 7mo ago
Seems both Nvidia (Groq) and OpenAI (Codex Spark) are now invested in the ASIC route one way or another.
IshKebab 7mo ago
Cerebras gives a many multiple speedup but it's also many multiples more expensive.

JKCalhoun 7mo ago

Apple should have done this yesterday. A local AI on my phone/Macbook is all I really want from this tech. The cloud-based AI (OpenAI, etc.) are todays AOL.

post-it 7mo ago
The hardware isn't there yet. Apple's neural engine is neat and has some uses but it just isn't in the same league as Claude right now. We'll get there.
Aurornis 7mo ago
The die size is huge. This isn’t the kind of chip that would go into your MacBook, let alone an iPhone. It’s for cloud based servers.
- adeelk93 7mo ago
  And computers used to be the size of a room. I think they can get it to iPhone size in the future, this is an early prototype.
  - MarsIronPI 7mo ago
    Well, there's a limit to how small we can make transistors with our current technology. As I understand it, Intel is already running into those limits with their new CPUs (they had to redesign the fins IIRC). I can imagine that without an actual breakthrough in chip manufacturing the size could stay large. That's not to say that a breakthrough won't happen, though.
    - layla5alive 7mo ago
      Yes, in 2D, but NAND has been using layers for a while. We call HBM interposers 2.5D. 3D breakthrough would be pretty easy but for those pesky problems like power delivery and cooling. (/s) But give that time (e.g. microfluidics) - something interesting is that it would be extra hard to use all layers at once, but NN might be a good fit, imagining that computation will be sparse (subsets activating simultaneously)...

RobotToaster 7mo ago

> I'm curious why this isn't getting much attention from larger companies. I can see two potential reasons: 1) Most of the big players seem convinced that AI is going to continue to improve at the rate it did in 2025, if their assumption is somehow correct by the time any chip entered mass production it would be obsolete. 2) The business model of the big players is to sell expensive subscriptions, and train on and sell the data you give it. Chips that allow for relatively inexpensive offline AI aren't conducive to that.

deleted 7mo ago
[deleted]

deleted 7mo ago

[deleted]

hrn_frs 7mo ago

> I'm curious why this isn't getting much attention from larger companies. Time is money and when you're competing with multiple companies with little margin for error you'll focus all your effort into releasing things quickly. This chip is "only" a performance boost. It will unlock a lot of potential, but startups can't divide their attention like this. Big companies like google are surely already investigating this venue, but they might lack hardware expertise.

theptip 7mo ago

> I'm curious why this isn't getting much attention from larger companies I would be shocked if Google isn’t working on this right now. They build their own TPUs, this is an extremely obvious direction from there. (And there are plenty of interesting co-design questions that only the frontier labs can dabble with; Taalas is stuck working around architectural quirks like “top-8 MoE”, Google can just rework the architecture hyperparameters to whatever gets best results in silico.)