5 ms·
I'm curious: If someone wanted to serve models off hardware/silicon directly (like Cerebras or Taalas, and soon Google I think) rather than GPUs, would these pa
by herpderperator 2mo ago
I'm curious: If someone wanted to serve models off hardware/silicon directly (like Cerebras or Taalas, and soon Google I think) rather than GPUs, would these parameters still be adjustable at request time? Or would they have to decide that before the model can even start serving and it would be locked in until they reload it (which would make it briefly unavailable)?
- StrauXX 2mo agoI'd expect that sampling would happen in software. Probably the hardware system would output a vector of probabilities over the tokenspace, just as the nets do when run in software. I don't know though and am not aware of any docs going into detail here. That being said, sampling is really cheap. So implementing it in hardware wouldn't be worth it.