3 ms·
> "more than a 100 AMD GPUs in production all year" and could be scaled up to "thousands of MI GPUs."
by htrp 3y ago
> "more than a 100 AMD GPUs in production all year" and could be scaled up to "thousands of MI GPUs."
- gdiamos 3y agoWe built a new horizontally scalable LLM inference server that is optimized for AMD. API docs: https://lamini-ai.github.io/API/completions/ https://lamini-ai.github.io/API/completions/ Put a load balancer in front of it and it will scale to as many GPUs as you can get. For training we build on SLURM. We containerized SLURM, so you just need to provision GPU servers, launch the SLURMd containers on training nodes. SLURM scales to 10,000s of servers. We typically launch SLURMd containers on bare metal, but some of our customers manage them with kubernetes/etc