4 ms·

The Taalas chips are not physically small. And part of their secret (if you look at the design) is just locating a bunch of memory soldered on the edges ( I bel

The Taalas chips are not physically small. And part of their secret (if you look at the design) is just locating a bunch of memory soldered on the edges ( I belive higher amounts of SRAM ? )

chorizo 2mo ago
Baking the base models on to ROM makes a lot of economic sense. SRAM for the KV cache & fine-tunes, not so much. Sure you’d get incredible speeds but it’s not scalable from a die-size or cost perspective. Rather base model on ROM + KV cache on DRAM is much more scalable. Also this would work great for edge devices that have a 2-5 year lifecycle.
- adrianN 2mo ago
  It is my understanding that just baking the model itself into silicon only gives moderate gains because memory bandwidth remains a bottleneck.
  - chorizo 2mo ago
    The big benefit is ROM cells require fewer components than DRAM. So the chips would be tiny, dense, cheap and consume far less power.
    - klodolph 2mo ago
      I thought DRAM was pretty dense already. Is mask ROM that much denser?
      - chorizo 2mo ago
        Yes, each rom bit can be a transistor or even a diode with a decoder circuit. Simplest Dram cell is capacitor+transistor - and you need a clock, refresh circuit etc. Someday, I imagine model weights could even be encoded as analog resistors (memristors or similar) for even greater density
        ReactiveJelly 2mo ago
        Hm. I wonder how many relays I'd need to make a physical MNIST classifier. That'd be dope
    - Tuna-Fish 2mo ago
      It's not even just that. If you just built the rom chips separately and swapped them for the RAM of a normal accelerator, it would not help at all. The trick is that every compute element in their system has it's own small pool of ROM, instead of putting all the ram behind a common pipe. ROM is just used because it's the densest kind of memory that can be fabricated on the same process as their logic.
  - lelanthran 2mo ago
    Did you try using the the talaas chat? Something stupid like 18k tokens/second. Think it's called Askjimmy or similar.
    - jgilias 2mo ago
      Oh boy, thanks for sharing this, truly mind blowing. It was chatjimmy.ai
    - TeMPOraL 2mo ago
      What is the model they're using there though? Interrogated, it claims it's a BERT variant and has capabilities around GPT-3 and below GPT-4. (Not that I believe it, it writes too well for GPT-3.) Hosted frontier models from two years ago would be much faster today, too.
      - egorfine 2mo ago
        They run Llama 3.1 8B.
- onion2k 2mo ago
  Baking the base models on to ROM makes a lot of economic sense. Less so for consumers though, because it'd mean the phone is out of date in 3 months when a better model comes along.
  - ronsor 2mo ago
    It's a perfect reason to get consumers to buy a new phone every year again! They got bored of the camera.
    - miohtama 2mo ago
      This stopped working some years ago. Now you sell the same phone with higher price tag.
    - ben_w 2mo ago
      Right now the models are doubling in performance (by the METR time horizon metric at least) every 4 months, so 3 doublings in a year; conversely, I hear (not my field) it takes around a year to make a prototype IC and another year to turn that into mass production, i.e. if the next (late-2026 model) iPhone has a chip like this, it will likely be with, at best, a late-2024 set of weights. I think you can get open-weights models today that have performance equivalent to the SOTA-late-2024 while fitting in the RAM of a (high end) 2025-26 phone. At some point the music will stop on training bigger models, and when that happens it will make sense to have ROM weights (or 100% analog circuits given how noise-resistant LLMs are), but we'll know when that is because the investment bubble funding the training of new models will have burst.
      - dgrabla 2mo ago
        wait, analog circuits? can you elaborate this?
        inigyou 2mo ago
        Circuits that aren't restricted to two particular voltage levels.
- tesnorindian 2mo ago
  We also have ReRAM (Analog Computing), which also holds a promising future given its efficiency and low power. Though ReRAM of larger size is still a research area.
- petra 2mo ago
  AMD can let the SRAM be on a different chip. Maybe even something similar to their 3D cache. that could increase density to 20B[1]. They could also move from 6nm to 2nm. that would probably increase density by another 3x to 60B. Add a bunch of chips together, and you get to a server that can run a 800B model, very fast and probably significantly cheaper than others. [1]https://www.eetimes.com/taalas-specializes-to-extremes-for-extraordinary-token-speed/ https://www.eetimes.com/taalas-specializes-to-extremes-for-e...
- coderbants 2mo ago
  I think the lifecycle for these chips could stretch far longer. If you're offering these models on a two year lifecycle, then you'd be able to stand up your top tier (wouldn't need to be frontier) at high speed. Run (for example) Kimi K3 on it and give it a brand name: AcmeAI Carbon Market it as your premier (only) model at high throughput. Two years later you stand up MSICs for the new state of the art with entirely new hardware, your lineup becomes: AcmeAI Nitrogen (top tier) AcmeAI Carbon (mid tier) If you just kept pushing the same model down your pricing tier over time you could still extract a lot of value from an old model, even years after it's been set in stone. Working on brand new code/frameworks? Pay to use the newest model. Working on legacy code? Use the lower tier models that will already know your legacy frameworks, pay far less and still get massive throughput. I've worked on a lot of government projects that this would be absolutely brilliant for. The other side of this is that agent harnesses are NOT set in stone, so even a legacy model with a knowledge cut-off that's years out of date can likely still be helped quite a bit by harness and fetch behaviours that are still developing rapidly. Especially at this kind of throughput.
  - hypfer 2mo ago
    I'm somewhat doubtful that we will be seeing something as large as Kimi K3 in silicon any time soon. This tech can definitely scale up from the current 8B prototype, but - at least as far as my limited understanding of the tech involved goes - you cannot just ASIC a trillion weights model due to physical size constraints. ___ Specification HC1 Model Llama 3.1 8B (hardwired) Process TSMC 6nm Die size 815mm² ___ So the current prototype already pushes the limits of what we can fit on a single die, and that is already likely going to limit your yield.
    - A_D_E_P_T 2mo ago
      Kimi K3 is huge, though. Deepseek V4 Flash is a much more moderate model (284B total), and it works extremely well. Models of that size, and smaller, are just going to keep getting better and better. Presumably there's a threshold below which models are not generally useful or competitive, but if models-on-silicon can scale up to just 256B, that would be really remarkable.
- to11mtm 2mo ago
  There's in-betweens; '1T SRAM' or eDRAM. Of course, 1T SRAM isn't really SRAM, but my understanding is it doesn't require external refresh like eDRAM, is a bit easier to fab on-die than eDRAM, and is half the mm2 per Megabit compared to real SRAM (15% more die size than eDRAM)...
selcuka 2mo ago
Their PoC chips are big, but then it's ridiculously fast (have you seen chatjimmy.ai?). Also they must be holding a bunch of patents.
- aenis 2mo ago
  Its a cool demo, but its gpt-3.5 level stupid, or worse. edit: Ok, I will self-apologize. Its apparently a 3B model. Mighty impressive for what it does.
  - selcuka 2mo ago
    It's a quantised 8B model (Llama 3.1 8B to be exact). [1] https://taalas.com/the-path-to-ubiquitous-ai/ https://taalas.com/the-path-to-ubiquitous-ai/
  - teiferer 2mo ago
    How time flies. Just 3.5 years ago, gpt-3.5 was touted as almost AGI, we're all going to be replaced by machines and worst case they will kill us all. And here we are, not much later, and it serves as the benchmark for "stupid"..
    - aenis 2mo ago
      Yup. Where I work we invited a bunch of people circa May 2023: a few top tier academics, a few government and NGO officials in charge of our industry, and a few startup founders. We are a big company, so people were kind enough to come and give speeches - and discuss. This was a roundtable on what's going to happen. You could play some of the speeches verbatim today and they would not feel out of place. And that is telling. We had a Stanford professor saying that gpt-3.5 can do everything he can - just better, and he feels his profession is on borrowed time. We had a government guy saying that entire swaths of jobs will be displaced before the year ends. And so on. The interesting thing is, a lot of people believed it then - and a lot of people believe it now. I wonder if we will have the same deja vu in, say, 2029. No, for sure not. It's going to be done and dusted for human thinking by end of this calendar year.
    - 2mo ago
deaton 2mo ago
Sure but the first step to having something that is physically small, small enough to cram into an iPhone, is to have something that, at first, isn't.

The Taalas chips are not physically small. And part of their secret (if you look at the design) is just locating a bunch of memory soldered on the edges ( I belive higher amounts of SRAM ? )

chorizo 2mo ago

Baking the base models on to ROM makes a lot of economic sense. SRAM for the KV cache & fine-tunes, not so much. Sure you’d get incredible speeds but it’s not scalable from a die-size or cost perspective. Rather base model on ROM + KV cache on DRAM is much more scalable. Also this would work great for edge devices that have a 2-5 year lifecycle.

adrianN 2mo ago
It is my understanding that just baking the model itself into silicon only gives moderate gains because memory bandwidth remains a bottleneck.
- chorizo 2mo ago
  The big benefit is ROM cells require fewer components than DRAM. So the chips would be tiny, dense, cheap and consume far less power.
  - klodolph 2mo ago
    I thought DRAM was pretty dense already. Is mask ROM that much denser?
    - chorizo 2mo ago
      Yes, each rom bit can be a transistor or even a diode with a decoder circuit. Simplest Dram cell is capacitor+transistor - and you need a clock, refresh circuit etc. Someday, I imagine model weights could even be encoded as analog resistors (memristors or similar) for even greater density
      - ReactiveJelly 2mo ago
        Hm. I wonder how many relays I'd need to make a physical MNIST classifier. That'd be dope
  - Tuna-Fish 2mo ago
    It's not even just that. If you just built the rom chips separately and swapped them for the RAM of a normal accelerator, it would not help at all. The trick is that every compute element in their system has it's own small pool of ROM, instead of putting all the ram behind a common pipe. ROM is just used because it's the densest kind of memory that can be fabricated on the same process as their logic.
- lelanthran 2mo ago
  Did you try using the the talaas chat? Something stupid like 18k tokens/second. Think it's called Askjimmy or similar.
  - jgilias 2mo ago
    Oh boy, thanks for sharing this, truly mind blowing. It was chatjimmy.ai
  - TeMPOraL 2mo ago
    What is the model they're using there though? Interrogated, it claims it's a BERT variant and has capabilities around GPT-3 and below GPT-4. (Not that I believe it, it writes too well for GPT-3.) Hosted frontier models from two years ago would be much faster today, too.
    - egorfine 2mo ago
      They run Llama 3.1 8B.
onion2k 2mo ago
Baking the base models on to ROM makes a lot of economic sense. Less so for consumers though, because it'd mean the phone is out of date in 3 months when a better model comes along.
- ronsor 2mo ago
  It's a perfect reason to get consumers to buy a new phone every year again! They got bored of the camera.
  - miohtama 2mo ago
    This stopped working some years ago. Now you sell the same phone with higher price tag.
  - ben_w 2mo ago
    Right now the models are doubling in performance (by the METR time horizon metric at least) every 4 months, so 3 doublings in a year; conversely, I hear (not my field) it takes around a year to make a prototype IC and another year to turn that into mass production, i.e. if the next (late-2026 model) iPhone has a chip like this, it will likely be with, at best, a late-2024 set of weights. I think you can get open-weights models today that have performance equivalent to the SOTA-late-2024 while fitting in the RAM of a (high end) 2025-26 phone. At some point the music will stop on training bigger models, and when that happens it will make sense to have ROM weights (or 100% analog circuits given how noise-resistant LLMs are), but we'll know when that is because the investment bubble funding the training of new models will have burst.
    - dgrabla 2mo ago
      wait, analog circuits? can you elaborate this?
      - inigyou 2mo ago
        Circuits that aren't restricted to two particular voltage levels.
tesnorindian 2mo ago
We also have ReRAM (Analog Computing), which also holds a promising future given its efficiency and low power. Though ReRAM of larger size is still a research area.
petra 2mo ago
AMD can let the SRAM be on a different chip. Maybe even something similar to their 3D cache. that could increase density to 20B[1]. They could also move from 6nm to 2nm. that would probably increase density by another 3x to 60B. Add a bunch of chips together, and you get to a server that can run a 800B model, very fast and probably significantly cheaper than others. [1]https://www.eetimes.com/taalas-specializes-to-extremes-for-extraordinary-token-speed/ https://www.eetimes.com/taalas-specializes-to-extremes-for-e...
coderbants 2mo ago
I think the lifecycle for these chips could stretch far longer. If you're offering these models on a two year lifecycle, then you'd be able to stand up your top tier (wouldn't need to be frontier) at high speed. Run (for example) Kimi K3 on it and give it a brand name: AcmeAI Carbon Market it as your premier (only) model at high throughput. Two years later you stand up MSICs for the new state of the art with entirely new hardware, your lineup becomes: AcmeAI Nitrogen (top tier) AcmeAI Carbon (mid tier) If you just kept pushing the same model down your pricing tier over time you could still extract a lot of value from an old model, even years after it's been set in stone. Working on brand new code/frameworks? Pay to use the newest model. Working on legacy code? Use the lower tier models that will already know your legacy frameworks, pay far less and still get massive throughput. I've worked on a lot of government projects that this would be absolutely brilliant for. The other side of this is that agent harnesses are NOT set in stone, so even a legacy model with a knowledge cut-off that's years out of date can likely still be helped quite a bit by harness and fetch behaviours that are still developing rapidly. Especially at this kind of throughput.
- hypfer 2mo ago
  I'm somewhat doubtful that we will be seeing something as large as Kimi K3 in silicon any time soon. This tech can definitely scale up from the current 8B prototype, but - at least as far as my limited understanding of the tech involved goes - you cannot just ASIC a trillion weights model due to physical size constraints. ___ Specification HC1 Model Llama 3.1 8B (hardwired) Process TSMC 6nm Die size 815mm² ___ So the current prototype already pushes the limits of what we can fit on a single die, and that is already likely going to limit your yield.
  - A_D_E_P_T 2mo ago
    Kimi K3 is huge, though. Deepseek V4 Flash is a much more moderate model (284B total), and it works extremely well. Models of that size, and smaller, are just going to keep getting better and better. Presumably there's a threshold below which models are not generally useful or competitive, but if models-on-silicon can scale up to just 256B, that would be really remarkable.
to11mtm 2mo ago
There's in-betweens; '1T SRAM' or eDRAM. Of course, 1T SRAM isn't really SRAM, but my understanding is it doesn't require external refresh like eDRAM, is a bit easier to fab on-die than eDRAM, and is half the mm2 per Megabit compared to real SRAM (15% more die size than eDRAM)...

selcuka 2mo ago

Their PoC chips are big, but then it's ridiculously fast (have you seen chatjimmy.ai?). Also they must be holding a bunch of patents.

aenis 2mo ago
Its a cool demo, but its gpt-3.5 level stupid, or worse. edit: Ok, I will self-apologize. Its apparently a 3B model. Mighty impressive for what it does.
- selcuka 2mo ago
  It's a quantised 8B model (Llama 3.1 8B to be exact). [1] https://taalas.com/the-path-to-ubiquitous-ai/ https://taalas.com/the-path-to-ubiquitous-ai/
- teiferer 2mo ago
  How time flies. Just 3.5 years ago, gpt-3.5 was touted as almost AGI, we're all going to be replaced by machines and worst case they will kill us all. And here we are, not much later, and it serves as the benchmark for "stupid"..
  - aenis 2mo ago
    Yup. Where I work we invited a bunch of people circa May 2023: a few top tier academics, a few government and NGO officials in charge of our industry, and a few startup founders. We are a big company, so people were kind enough to come and give speeches - and discuss. This was a roundtable on what's going to happen. You could play some of the speeches verbatim today and they would not feel out of place. And that is telling. We had a Stanford professor saying that gpt-3.5 can do everything he can - just better, and he feels his profession is on borrowed time. We had a government guy saying that entire swaths of jobs will be displaced before the year ends. And so on. The interesting thing is, a lot of people believed it then - and a lot of people believe it now. I wonder if we will have the same deja vu in, say, 2029. No, for sure not. It's going to be done and dusted for human thinking by end of this calendar year.
  - 2mo ago

deaton 2mo ago

Sure but the first step to having something that is physically small, small enough to cram into an iPhone, is to have something that, at first, isn't.