5 ms·

What quantization is used for it? I see you can run models that use up to approximately 640GB but the full non-quantized 405B uses ~810GB of VRAM (native weight

• reissbaker 2y ago
  * Models we run on our own infra (the ones with boot times) always run at whatever quantization the repo uses — typically unquantized, although if you paste in a link to a 4-bit AWQ repo of course that's what we'll run. * For always-on proxied models, we're dependent on the inference provider. Typically we use Together AI, but for Llama 3.1 we found some show-stopping context length related bugs that caused us to switch to Fireworks. Since we proxy 405B to Fireworks, that one runs at FP8 (that's what Fireworks uses). This may change in the future and in general we prefer to match the underlying repo quantization when possible; proxying is a bit of a hack in order to keep more models warmed up and reduce boot times (and to run 405B for now, although we're working on getting larger clusters and supporting distributed multi-node inference). As we grow, hopefully we'll be able to reduce our dependency on third party proxies. I'm also curious to try running models on AMD, since that would allow for much larger single-node models — for example, an MI300X has 192GB per card, so 8x of those would run 405B at fp16. Obviously there are some support issues with AMD so TBD how well it'll work but it's an interesting option.