4 ms·
Is there a way to run this in AWS? Seems like the biggest GPU node they have is the p5.48xlarge @ 640GB (8xH100s). Routing between multiple nodes would be too
by Atreiden 2y ago
Is there a way to run this in AWS?
Seems like the biggest GPU node they have is the p5.48xlarge @ 640GB (8xH100s). Routing between multiple nodes would be too slow unless there's an InfiniBand fabric you can leverage. Interested to know if anyone else is exploring this.
- Tiberium 2y agoAWS has a separate service for running LLMs called Amazon Bedrock, it shouldn't take long for them to add 3.1 since they have 3 and 2 already.
- tpm 2y agofp8 quantization should work if that's acceptable?
- woodson 2y agoYou can run multi-node with tensor parallel plus pipeline parallel inference, e.g. with vLLM (https://docs.vllm.ai/en/latest/serving/distributed_serving.html https://docs.vllm.ai/en/latest/serving/distributed_serving.h...).