4 ms·
so for fast responses we usea a rtx4090 with vllm, but yeah it depends on your use case
by K0IN 2y ago
so for fast responses we usea a rtx4090 with vllm, but yeah it depends on your use case
- samspenc 2y agoAgreed, this works great for a single or small set of users (2-3), but OP is asking about self-hosting an open-source / open-weight LLM locally to serve a larger set of users (100s per post title). I suspect they may need a larger server farm with a lot of GPUs to handle the load.