3 ms·
I was thinking maybe RunPod or Modal could use it to handle multiple containers per CPU, possibly including some that are much lighter like stable diffusion.
by ilaksh 3y ago
I was thinking maybe RunPod or Modal could use it to handle multiple containers per CPU, possibly including some that are much lighter like stable diffusion.
- brucethemoose2 3y agoIf its multiple llm client requests, batched inference would be massively more efficient. And for multiple services... Probably just better to run multiple cheap instances and/or load dynamically? The MI300 is super expensive.
- ilaksh 3y agoHow expensive is it compared to A100?
- brucethemoose2 3y agoThat is an excellent question. In theory its more expensive to produce than an H100, but in practice... shrug. But I was thinking 3090 or lesser tesla/Quadro instances would be sufficient.