3 ms·
If its multiple llm client requests, batched inference would be massively more efficient. And for multiple services... Probably just better to run multiple che
by brucethemoose2 3y ago
If its multiple llm client requests, batched inference would be massively more efficient.
And for multiple services... Probably just better to run multiple cheap instances and/or load dynamically? The MI300 is super expensive.
- ilaksh 3y agoHow expensive is it compared to A100?
- brucethemoose2 3y agoThat is an excellent question. In theory its more expensive to produce than an H100, but in practice... shrug. But I was thinking 3090 or lesser tesla/Quadro instances would be sufficient.