4 ms·

The magic is in the fact that they essentially have an ASIC llm device. There is no other trickery. The problem they will face is that it is actually locked in

by Cort3z 2mo ago

The magic is in the fact that they essentially have an ASIC llm device. There is no other trickery. The problem they will face is that it is actually locked in silicon, so upgrading models will be difficult, and likely require new hardware each time.

ch4s3 2mo ago
At some point I imagine you’d add a software layer on top that holds more current training and can be called as needed trading off for slower responses. There’s already work out there splitting models across networks. You could have the base on silicon, some stuff in memory on the machine, and another frontier tool in the cloud.
- jaggederest 2mo ago
  Taalas actually already support LoRA, basically doing exactly what you say. The other thing that I think is really interesting about all of this, is that LLMs are already perforce behind the times with their knowledge cutoff, so adding an additional ~3 months for bake into silicon isn't such a huge deal, I think, for the ~10x more efficient and faster you get.
  - ch4s3 2mo ago
    Yeah, really a fascinating time in computing. Once these chips start to become more common I'll be curious how people find ways to use them. One can imagine a world where really simple inference is available on dirt cheap chips found in toys and other low cost consumer electronics.
- theshrike79 2mo ago
  There are already studies proving that a "stupid" model with a good harness + tool calling will outperform a "smart" model. Things like this give me hope for a system that can be fully local and private, but also with the ability to be almost infinitely extendable with tools.
  - ch4s3 2mo ago
    Can you point me to something to read about this?
antman 2mo ago
Is the size of the ASIC limiting factor or can they infinitely tensor parallelize? If thats possible then it would make it only a matter of economy of scale, there are good enough models already for people to invest in that kind of platform

by Cort3z 2mo ago

ch4s3 2mo ago

At some point I imagine you’d add a software layer on top that holds more current training and can be called as needed trading off for slower responses. There’s already work out there splitting models across networks. You could have the base on silicon, some stuff in memory on the machine, and another frontier tool in the cloud.

jaggederest 2mo ago
Taalas actually already support LoRA, basically doing exactly what you say. The other thing that I think is really interesting about all of this, is that LLMs are already perforce behind the times with their knowledge cutoff, so adding an additional ~3 months for bake into silicon isn't such a huge deal, I think, for the ~10x more efficient and faster you get.
- ch4s3 2mo ago
  Yeah, really a fascinating time in computing. Once these chips start to become more common I'll be curious how people find ways to use them. One can imagine a world where really simple inference is available on dirt cheap chips found in toys and other low cost consumer electronics.
theshrike79 2mo ago
There are already studies proving that a "stupid" model with a good harness + tool calling will outperform a "smart" model. Things like this give me hope for a system that can be fully local and private, but also with the ability to be almost infinitely extendable with tools.
- ch4s3 2mo ago
  Can you point me to something to read about this?

antman 2mo ago

Is the size of the ASIC limiting factor or can they infinitely tensor parallelize? If thats possible then it would make it only a matter of economy of scale, there are good enough models already for people to invest in that kind of platform