4 ms·

Wouldn't this mean someone with sufficient hardware could lift the SOTA model weights off the chip? Or are you saying that these chips would only be used intern

Wouldn't this mean someone with sufficient hardware could lift the SOTA model weights off the chip? Or are you saying that these chips would only be used internally by these companies and not sold to the public?

syntaxing 2mo ago
I don’t get why this is an issue? You can run Claude/OpenAI SOTA models through Amazon bedrock. These weights have to live somewhere to run on Bedrock.
- wmf 2mo ago
  somewhere = an AWS data center with multiple layers of security and NDAs They won't sell/rent/license the weights to an end user at any price because they don't trust your security.
  - syntaxing 2mo ago
    I work in embedded space. Just because it’s in hardware doesn’t mean you can’t “protect” it. Most modern software (regardless if it’s hardware or not) can be cryptophically signed.
    - bluezly 2mo ago
      Signing protects authenticity and integrity, but it doesn’t really solve confidentiality. If the weights are physically encoded in hardware and the attacker owns the device, the problem becomes hardware extraction: decapping, probing, imaging, side channels, etc. You can make that very expensive, but it’s still a very different security model from keeping the weights in a datacenter.
snek_case 2mo ago
The weights are very unlikely to be on the chip itself. That wouldn't work for SOTA models that are terabyte scale, even quantized. This is probably an accelerator for specific kernels in the model, but the weights are likely loaded from memory. The chip may have SRAM to store some of the weights temporarily during inference.
- foltik 2mo ago
  At least in the case of Taalas the weights are physically encoded directly on the chip. It’s composed of 4-bit multiplier cells that compute all 16 possible results in parallel. The top metal wiring layer physically selects the one that corresponds to a multiplication with that cell’s constant weight, and routes it to the next layer.
  - mdp2021 2mo ago
    Are you sure? Source? (does not seem to be https://taalas.com/the-path-to-ubiquitous-ai/ https://taalas.com/the-path-to-ubiquitous-ai/ , for example)
    - foltik 2mo ago
      It’s described in this patent application [0]. There’s a bit of hand waving so the HC1 might be slightly different, but the gist is the same. https://patents.justia.com/patent/20250123802 https://patents.justia.com/patent/20250123802
amazingamazing 2mo ago
One idea would be to use an open model.
dumberquestions 2mo ago
I wouldn't expect companies not sharing their weights today to be any more likely to share them if they're on hardware, this doesn't sufficiently hide weights from a local user.

syntaxing 2mo ago

I don’t get why this is an issue? You can run Claude/OpenAI SOTA models through Amazon bedrock. These weights have to live somewhere to run on Bedrock.

wmf 2mo ago
somewhere = an AWS data center with multiple layers of security and NDAs They won't sell/rent/license the weights to an end user at any price because they don't trust your security.
- syntaxing 2mo ago
  I work in embedded space. Just because it’s in hardware doesn’t mean you can’t “protect” it. Most modern software (regardless if it’s hardware or not) can be cryptophically signed.
  - bluezly 2mo ago
    Signing protects authenticity and integrity, but it doesn’t really solve confidentiality. If the weights are physically encoded in hardware and the attacker owns the device, the problem becomes hardware extraction: decapping, probing, imaging, side channels, etc. You can make that very expensive, but it’s still a very different security model from keeping the weights in a datacenter.

snek_case 2mo ago

The weights are very unlikely to be on the chip itself. That wouldn't work for SOTA models that are terabyte scale, even quantized. This is probably an accelerator for specific kernels in the model, but the weights are likely loaded from memory. The chip may have SRAM to store some of the weights temporarily during inference.

foltik 2mo ago
At least in the case of Taalas the weights are physically encoded directly on the chip. It’s composed of 4-bit multiplier cells that compute all 16 possible results in parallel. The top metal wiring layer physically selects the one that corresponds to a multiplication with that cell’s constant weight, and routes it to the next layer.
- mdp2021 2mo ago
  Are you sure? Source? (does not seem to be https://taalas.com/the-path-to-ubiquitous-ai/ https://taalas.com/the-path-to-ubiquitous-ai/ , for example)
  - foltik 2mo ago
    It’s described in this patent application [0]. There’s a bit of hand waving so the HC1 might be slightly different, but the gist is the same. https://patents.justia.com/patent/20250123802 https://patents.justia.com/patent/20250123802

amazingamazing 2mo ago

One idea would be to use an open model.

dumberquestions 2mo ago

I wouldn't expect companies not sharing their weights today to be any more likely to share them if they're on hardware, this doesn't sufficiently hide weights from a local user.