7 ms·

> Kinda like a CD-ROM/Game cartridge, or a printed book, it only holds one model and cannot be rewritten. Imagine a slot on your computer where you physically

> Kinda like a CD-ROM/Game cartridge, or a printed book, it only holds one model and cannot be rewritten. Imagine a slot on your computer where you physically pop out and replace the chip with different models, sort of like a Nintendo DS.

beAroundHere 8mo ago
That's the kind of hardware am rooting for. Since it'll encourage Open weighs models, and would be much more private. Infact, I was thinking, if robots of future could have such slots, where they can use different models, depending on the task they're given. Like a Hardware MoE.
- NitpickLawyer 8mo ago
  > Since it'll encourage Open weighs models Is this accurate? I don't know enough about hardware, but perhaps someone could clarify: how hard would it be to reverse engineer this to "leak" the model weights? Is it even possible? There are some labs that sell access to their models (mistral, cohere, etc) without having their models open. I could see a world where more companies can do this if this turns out to be a viable way. Even to end customers, if reverse engineering is deemed impossible. You could have a device that does most of the inference locally and only "call home" when stumped (think alexa with local processing for intent detection and cloud processing for the rest, but better).
  - yonatan8070 8mo ago
    It's likely possible to extract model weights from the chip's design, but you'd need tooling at the level of an Intel R&D lab, not something any hobbyist could afford. I doubt anyone would have the skills, wallet, and tools to RE one of these and extract model weights to run them on other hardware. Maybe state actors like the Chinese government or similar could pull that off.
    - tmzt 8mo ago
      Or a grinder and a camera. See CCC of years past.
8cvor6j844qw_d6 8mo ago
A cartridge slot for models is a fun idea. Instead of one chip running any model, you get one model or maybe a family of models per chip at (I assume) much better perf/watt. Curious whether the economics work out for consumer use or if this stays in the embedded/edge space.
- sixtyj 8mo ago
  Plug it into skull bone. Neuralink + slot for a model that you can buy in s grocery store instead of prepaid Netflix card.
  - pennomi 8mo ago
    We better solve the energy usage and cooling first otherwise that will be a very spicy body mod.
Onavo 8mo ago
Yeah maybe you can call it PCIe.
roncesvalles 8mo ago
That slot is called USB-C. I can fully imagine inference ASICs coming in powerbank form factor that you'd just plug and play.
- XorNot 8mo ago
  Pretty sure it'd just be a thumbdrive. Are the Taalas chips particularly large in surface area?
  - dmurray 8mo ago
    The only product they've announced at the moment [0] is a PCI-e card. It's more like a small power bank than a big thumb drive. But sure, the next generation could be much smaller. It doesn't require battery cells, (much) heat management, or ruggedization, all of which put hard limits on how much you can miniaturise power banks. [0] https://taalas.com/the-path-to-ubiquitous-ai/ https://taalas.com/the-path-to-ubiquitous-ai/
    - ChrisMarshallNY 8mo ago
      I’m old enough to remember your typical computer filling warehouse-sized buildings. Nowadays, your average cellphone has more computing power than those behemoths. I have a micro SD card with 256GB capacity, and I think they are up to 2TB. On a device the size of a fingernail.
      - slfnflctd 8mo ago
        That is all definitely amazing, but data storage is a fundamentally different process with far fewer constraints than continuous computation.
        ChrisMarshallNY 8mo ago
        It all uses the same miniaturization techniques, though.
    - 8mo ago
Someone 8mo ago
Would somewhat work except for the power usage. I doubt it would scale linearly, but for home use 170 tokens/s at 2.5W would be cool; 17 tokens/s at 0,25W would be awesome. On the other hand, this may be a step towards positronic brains (https://en.wikipedia.org/wiki/Positronic_brain https://en.wikipedia.org/wiki/Positronic_brain)
kilroy123 8mo ago
This is what I've been wanting! Just like those eGPUs you would plug into your Mac. You would have a big model or device capable of running a top-tier model under your desk. All local, completely private.

beAroundHere 8mo ago

That's the kind of hardware am rooting for. Since it'll encourage Open weighs models, and would be much more private. Infact, I was thinking, if robots of future could have such slots, where they can use different models, depending on the task they're given. Like a Hardware MoE.

NitpickLawyer 8mo ago
> Since it'll encourage Open weighs models Is this accurate? I don't know enough about hardware, but perhaps someone could clarify: how hard would it be to reverse engineer this to "leak" the model weights? Is it even possible? There are some labs that sell access to their models (mistral, cohere, etc) without having their models open. I could see a world where more companies can do this if this turns out to be a viable way. Even to end customers, if reverse engineering is deemed impossible. You could have a device that does most of the inference locally and only "call home" when stumped (think alexa with local processing for intent detection and cloud processing for the rest, but better).
- yonatan8070 8mo ago
  It's likely possible to extract model weights from the chip's design, but you'd need tooling at the level of an Intel R&D lab, not something any hobbyist could afford. I doubt anyone would have the skills, wallet, and tools to RE one of these and extract model weights to run them on other hardware. Maybe state actors like the Chinese government or similar could pull that off.
  - tmzt 8mo ago
    Or a grinder and a camera. See CCC of years past.

8cvor6j844qw_d6 8mo ago

A cartridge slot for models is a fun idea. Instead of one chip running any model, you get one model or maybe a family of models per chip at (I assume) much better perf/watt. Curious whether the economics work out for consumer use or if this stays in the embedded/edge space.

sixtyj 8mo ago
Plug it into skull bone. Neuralink + slot for a model that you can buy in s grocery store instead of prepaid Netflix card.
- pennomi 8mo ago
  We better solve the energy usage and cooling first otherwise that will be a very spicy body mod.

Onavo 8mo ago

Yeah maybe you can call it PCIe.

roncesvalles 8mo ago

That slot is called USB-C. I can fully imagine inference ASICs coming in powerbank form factor that you'd just plug and play.

XorNot 8mo ago
Pretty sure it'd just be a thumbdrive. Are the Taalas chips particularly large in surface area?
- dmurray 8mo ago
  The only product they've announced at the moment [0] is a PCI-e card. It's more like a small power bank than a big thumb drive. But sure, the next generation could be much smaller. It doesn't require battery cells, (much) heat management, or ruggedization, all of which put hard limits on how much you can miniaturise power banks. [0] https://taalas.com/the-path-to-ubiquitous-ai/ https://taalas.com/the-path-to-ubiquitous-ai/
  - ChrisMarshallNY 8mo ago
    I’m old enough to remember your typical computer filling warehouse-sized buildings. Nowadays, your average cellphone has more computing power than those behemoths. I have a micro SD card with 256GB capacity, and I think they are up to 2TB. On a device the size of a fingernail.
    - slfnflctd 8mo ago
      That is all definitely amazing, but data storage is a fundamentally different process with far fewer constraints than continuous computation.
      - ChrisMarshallNY 8mo ago
        It all uses the same miniaturization techniques, though.
  - 8mo ago

Someone 8mo ago

Would somewhat work except for the power usage. I doubt it would scale linearly, but for home use 170 tokens/s at 2.5W would be cool; 17 tokens/s at 0,25W would be awesome. On the other hand, this may be a step towards positronic brains (https://en.wikipedia.org/wiki/Positronic_brain https://en.wikipedia.org/wiki/Positronic_brain)

kilroy123 8mo ago

This is what I've been wanting! Just like those eGPUs you would plug into your Mac. You would have a big model or device capable of running a top-tier model under your desk. All local, completely private.