4 ms·

> 17k tps taalas chip It's odd to me that I haven't heard anything about this approach (baking LLMs/weights into silicon directly) since. It seems almost commo

by ipdashc 3mo ago

> 17k tps taalas chip It's odd to me that I haven't heard anything about this approach (baking LLMs/weights into silicon directly) since. It seems almost common-sense that we're going to end up there eventually. And it feels like that point is drawing ever closer now that model capabilities, if not quite plateauing out, are at least getting to a "good enough" point for a LOT of use cases. I wonder if it's being worked on in secret, if there's something about it that makes it infeasible, or if companies are really too nervous to lock in one model like that because the next one down the line could be a huge improvement. Re. infeasability, I have heard that the Taalas demonstration chip ran Llama 3.1 8B (a pretty horrible model) and that even that took a massive amount of transistors / die area. So it might just be the case that the good models are too big to fit on silicon?

wmf 3mo ago
Good models will require multiple Taalas chips but Groq and Cerebras also require a lot of chips and that hasn't stopped them.
- ipdashc 3mo ago
  > Good models will require multiple Taalas chips I guess that makes sense. Is this feasible, or does the added latency between chips kill any of the performance gains?
  - wmf 3mo ago
    Using multiple chips seems to work fine for Cerebras and Groq so it should also work for Taalas. It does sounds challenging to reach >10K tok/s but latency could be below 1 us which is a small part of the token budget.
topspin 3mo ago
I have also been thinking about this a lot, and share your belief that this is inevitable. Taalas has a running demo here: https://chatjimmy.ai/ https://chatjimmy.ai/ It's eye opening: generated an AVX-512 optimized Mersenne Twister in C in 0.076s, 13,706 tok/s. Too fast for the tok/s to be terribly accurate.
mdp2021 3mo ago
> It's odd to me that I haven't heard anything about this approach ... I wonder if it's being worked on in secret, if there's something about it that makes it infeasible The studies and efforts are ongoing and public, and there are technical hurdles to be faced - but the relevant works go back in time quite a lot and there is heightened interest in it now. It seems that you simply took the "hyped headlines" for the whole of the work.
- ipdashc 3mo ago
  > It seems that you simply took the "hyped headlines" for the whole of the work. Well, yeah, that's what I'm saying. It's odd that there haven't been any major headlines (customer interest, competitors' announcements, etc) other than their initial demo. Good to hear it's being worked on though!
  - mdp2021 3mo ago
    Did we not play with MNIST and placed some calculated bet on NNs well before Yann LeCun started the fire with the explosive success of the Convolutional NNs? I'd say it pretty consistently starts in the underground. The real revolution in the context is that it /could/ be done practically - overcoming the hurdles. But for what the interest in the matter is concerned, I'd say there almost cannot be a greater interest at this stage: making NNs efficient. This must be absolutely evident, as evident it is that the separation of memory and processor is against the idea of NNs, as evident as it is that multiplication is achievable just physically. Of course many have seen that and got on studying it. As soon as it will be optimally practical...
coder543 3mo ago
> It's odd to me that I haven't heard anything about this approach since. It has only been four months since they unveiled their first prototype. I don't understand your confusion. Chip development does not happen overnight...? Their initial blog post laid out a roadmap, so theoretically they should have another thing to demonstrate this summer.
- mdp2021 3mo ago
  You are focusing on Taalas, but (specific) analogue computing, electronic NNs, compute-in-memory etc. - the field including the contextual approach - backdate to Rosenblatt.
  - coder543 3mo ago
    Yes, I’m focused on the topic at hand that the person I replied to was also talking about. The person I replied to was acting as if Taalas was ancient history. I was pointing out it has only been a few months.
    - mdp2021 3mo ago
      I'd say the original remark was more general («this approach (baking LLMs/weights into silicon directly) [... as if] worked on in secret») - which is salient, because when I investigated weeks ago, I found a large number of attempts to CIM and to general branching from Von Neumann architecture for the purpose of optimizing NNs implementations in HW. Universities are studying, startups are proposing - the «approach» is under the big headlines level but quite lively. Not just Taalas, not just their way - which remains remarkable in the scene as the HW is achieved, working, online, available... and amazing.
      - coder543 3mo ago
        CIM does not bake the weights into silicon. The level of optimization that you can do down to the last transistor when the weights are fixed is on an entirely different level than CIM where you still need general purpose ALUs all over the place.

by ipdashc 3mo ago

wmf 3mo ago

Good models will require multiple Taalas chips but Groq and Cerebras also require a lot of chips and that hasn't stopped them.

ipdashc 3mo ago
> Good models will require multiple Taalas chips I guess that makes sense. Is this feasible, or does the added latency between chips kill any of the performance gains?
- wmf 3mo ago
  Using multiple chips seems to work fine for Cerebras and Groq so it should also work for Taalas. It does sounds challenging to reach >10K tok/s but latency could be below 1 us which is a small part of the token budget.

topspin 3mo ago

I have also been thinking about this a lot, and share your belief that this is inevitable. Taalas has a running demo here: https://chatjimmy.ai/ https://chatjimmy.ai/ It's eye opening: generated an AVX-512 optimized Mersenne Twister in C in 0.076s, 13,706 tok/s. Too fast for the tok/s to be terribly accurate.

mdp2021 3mo ago

> It's odd to me that I haven't heard anything about this approach ... I wonder if it's being worked on in secret, if there's something about it that makes it infeasible The studies and efforts are ongoing and public, and there are technical hurdles to be faced - but the relevant works go back in time quite a lot and there is heightened interest in it now. It seems that you simply took the "hyped headlines" for the whole of the work.

ipdashc 3mo ago
> It seems that you simply took the "hyped headlines" for the whole of the work. Well, yeah, that's what I'm saying. It's odd that there haven't been any major headlines (customer interest, competitors' announcements, etc) other than their initial demo. Good to hear it's being worked on though!
- mdp2021 3mo ago
  Did we not play with MNIST and placed some calculated bet on NNs well before Yann LeCun started the fire with the explosive success of the Convolutional NNs? I'd say it pretty consistently starts in the underground. The real revolution in the context is that it /could/ be done practically - overcoming the hurdles. But for what the interest in the matter is concerned, I'd say there almost cannot be a greater interest at this stage: making NNs efficient. This must be absolutely evident, as evident it is that the separation of memory and processor is against the idea of NNs, as evident as it is that multiplication is achievable just physically. Of course many have seen that and got on studying it. As soon as it will be optimally practical...

coder543 3mo ago

> It's odd to me that I haven't heard anything about this approach since. It has only been four months since they unveiled their first prototype. I don't understand your confusion. Chip development does not happen overnight...? Their initial blog post laid out a roadmap, so theoretically they should have another thing to demonstrate this summer.

mdp2021 3mo ago
You are focusing on Taalas, but (specific) analogue computing, electronic NNs, compute-in-memory etc. - the field including the contextual approach - backdate to Rosenblatt.
- coder543 3mo ago
  Yes, I’m focused on the topic at hand that the person I replied to was also talking about. The person I replied to was acting as if Taalas was ancient history. I was pointing out it has only been a few months.
  - mdp2021 3mo ago
    I'd say the original remark was more general («this approach (baking LLMs/weights into silicon directly) [... as if] worked on in secret») - which is salient, because when I investigated weeks ago, I found a large number of attempts to CIM and to general branching from Von Neumann architecture for the purpose of optimizing NNs implementations in HW. Universities are studying, startups are proposing - the «approach» is under the big headlines level but quite lively. Not just Taalas, not just their way - which remains remarkable in the scene as the HW is achieved, working, online, available... and amazing.
    - coder543 3mo ago
      CIM does not bake the weights into silicon. The level of optimization that you can do down to the last transistor when the weights are fixed is on an entirely different level than CIM where you still need general purpose ALUs all over the place.