4 ms·

LLM-D: Kubernetes-Native Distributed Inference

120 points by smarterclayton 1y ago

anttiharju 1y ago
I wonder if this is preferable to kServe
- smarterclayton 1y ago
  llm-d would make sense if you are running a very large production LLM serving setup - say 5+ full H100 hosts. The aim is to be much more focused than kserve is on exactly the needs of serving LLMs. It would of course be possible to run alongside kserve, but the user we are targeting is not typically a kserve deployer today.
  - anttiharju 1y ago
    Do you think https://github.com/openai/CLIP https://github.com/openai/CLIP can be ran on it? LLM makes me think of chatbots but I suppose because it's inference-based it would work. Somewhat unclear on what's the difference between LLMs and inference, I think inference is the type of compute LLMs use. I wonder if inference-d would be a fitting name.
    - smarterclayton 1y ago
      Inference is the process of evaluating a model ("inferring" a response to the inputs). LLMs are uniquely difficult to serve because they push the limits on the hardware. The models we support come from the model server vLLM https://docs.vllm.ai/en/latest/models/supported_models.html https://docs.vllm.ai/en/latest/models/supported_models.html, which has a focus on large generative models. I don't see CLIP in the list.
dzr0001 1y ago
I did a quick scan of the repo and didn't see any reference to Ray. Would this indicate that llm-d lacks support for pipeline parallelism?
- qntty 1y ago
  I believe this is a question you should ask about vLLM, not llm-d. It looks like vLLM does support pipeline parallelism via Ray: https://docs.vllm.ai/en/latest/serving/distributed_serving.html#running-vllm-on-a-single-node https://docs.vllm.ai/en/latest/serving/distributed_serving.h... This project appears to make use of both vLLM and Inference Gateway (an official Kubernetes extension to the Gateway resource). The contributions of llm-d itself seems to mostly be a scheduling algorithm for load balancing across vLLM instances.
- smarterclayton 1y ago
  We inherit any multi-host support from vLLM, so https://docs.vllm.ai/en/latest/serving/distributed_serving.html#running-vllm-on-multiple-nodes https://docs.vllm.ai/en/latest/serving/distributed_serving.h... would be the expected path. We plan to publish examples of multi-host inference that leverages LeaderWorkerSets - https://github.com/kubernetes-sigs/lws https://github.com/kubernetes-sigs/lws - which helps run ranked serving workloads across hosts. LeaderWorkerSet is how Google supports both TPU and GPU multi-host deployments - see https://github.com/kubernetes-sigs/lws/blob/main/config/samples/leaderworkerset_tpu.yaml https://github.com/kubernetes-sigs/lws/blob/main/config/samp... for an example. Edit: Here is an example Kubernetes configuration running DeepSeek-R1 on vLLM multi-host using LeaderWorkerSet https://github.com/kubernetes-sigs/wg-serving/blob/main/serving-catalog/core/lws/vllm/deepseek-r1/gke/README.md https://github.com/kubernetes-sigs/wg-serving/blob/main/serv.... This work would be integrated into llm-d.
rdli 1y ago
This is really interesting. For SOTA inference systems, I've seen two general approaches: * The "stack-centric" approach such as vLLM production stack, AIBrix, etc. These set up an entire inference stack for you including KV cache, routing, etc. * The "pipeline-centric" approach such as NVidia Dynamo, Ray, BentoML. These give you more of an SDK so you can define inference pipelines that you can then deploy on your specific hardware. It seems like LLM-d is the former. Is that right? What prompted you to go down that direction, instead of the direction of Dynamo?
- qntty 1y ago
  It sounds like you might be confusing different parts of the stack. NVIDIA Dynamo for example supports vLLM as the inference engine. I think you should think of something like vLLM as more akin to GUnicorn, and llm-d as an application load balancer. And I guess something like NVIDIA Dynamo would be like Django.
  - smarterclayton 1y ago
    llm-d is intended to be three clean layers: 1. Balance / schedule incoming requests to the right backend 2. Model server replicas that can run on multiple hardware topologies 3. Prefix caching hierarchy with well-tested variants for different use cases So it's a 3-tier architecture. The biggest difference with Dynamo is that llm-d is using the inference gateway extension - https://github.com/kubernetes-sigs/gateway-api-inference-extension https://github.com/kubernetes-sigs/gateway-api-inference-ext... - which brings Kubernetes owned APIs for managing model routing, request priority and flow control, LoRA support etc.
    - rdli 1y ago
      I would think that that the NVidia Dynamo SDK (pipelines) is a big difference as well (https://github.com/ai-dynamo/dynamo/tree/main/deploy/sdk/docs/sdk https://github.com/ai-dynamo/dynamo/tree/main/deploy/sdk/doc...), or am I missing something?
      - 1y ago
gitroom 1y ago
[dead]
Kemschumam 1y ago
What would be the benefit of this project over hosting VLLM in Ray?

I wonder if this is preferable to kServe

smarterclayton 1y ago
llm-d would make sense if you are running a very large production LLM serving setup - say 5+ full H100 hosts. The aim is to be much more focused than kserve is on exactly the needs of serving LLMs. It would of course be possible to run alongside kserve, but the user we are targeting is not typically a kserve deployer today.
- anttiharju 1y ago
  Do you think https://github.com/openai/CLIP https://github.com/openai/CLIP can be ran on it? LLM makes me think of chatbots but I suppose because it's inference-based it would work. Somewhat unclear on what's the difference between LLMs and inference, I think inference is the type of compute LLMs use. I wonder if inference-d would be a fitting name.
  - smarterclayton 1y ago
    Inference is the process of evaluating a model ("inferring" a response to the inputs). LLMs are uniquely difficult to serve because they push the limits on the hardware. The models we support come from the model server vLLM https://docs.vllm.ai/en/latest/models/supported_models.html https://docs.vllm.ai/en/latest/models/supported_models.html, which has a focus on large generative models. I don't see CLIP in the list.

I did a quick scan of the repo and didn't see any reference to Ray. Would this indicate that llm-d lacks support for pipeline parallelism?

qntty 1y ago
I believe this is a question you should ask about vLLM, not llm-d. It looks like vLLM does support pipeline parallelism via Ray: https://docs.vllm.ai/en/latest/serving/distributed_serving.html#running-vllm-on-a-single-node https://docs.vllm.ai/en/latest/serving/distributed_serving.h... This project appears to make use of both vLLM and Inference Gateway (an official Kubernetes extension to the Gateway resource). The contributions of llm-d itself seems to mostly be a scheduling algorithm for load balancing across vLLM instances.
smarterclayton 1y ago
We inherit any multi-host support from vLLM, so https://docs.vllm.ai/en/latest/serving/distributed_serving.html#running-vllm-on-multiple-nodes https://docs.vllm.ai/en/latest/serving/distributed_serving.h... would be the expected path. We plan to publish examples of multi-host inference that leverages LeaderWorkerSets - https://github.com/kubernetes-sigs/lws https://github.com/kubernetes-sigs/lws - which helps run ranked serving workloads across hosts. LeaderWorkerSet is how Google supports both TPU and GPU multi-host deployments - see https://github.com/kubernetes-sigs/lws/blob/main/config/samples/leaderworkerset_tpu.yaml https://github.com/kubernetes-sigs/lws/blob/main/config/samp... for an example. Edit: Here is an example Kubernetes configuration running DeepSeek-R1 on vLLM multi-host using LeaderWorkerSet https://github.com/kubernetes-sigs/wg-serving/blob/main/serving-catalog/core/lws/vllm/deepseek-r1/gke/README.md https://github.com/kubernetes-sigs/wg-serving/blob/main/serv.... This work would be integrated into llm-d.

rdli 1y ago

This is really interesting. For SOTA inference systems, I've seen two general approaches: * The "stack-centric" approach such as vLLM production stack, AIBrix, etc. These set up an entire inference stack for you including KV cache, routing, etc. * The "pipeline-centric" approach such as NVidia Dynamo, Ray, BentoML. These give you more of an SDK so you can define inference pipelines that you can then deploy on your specific hardware. It seems like LLM-d is the former. Is that right? What prompted you to go down that direction, instead of the direction of Dynamo?

qntty 1y ago
It sounds like you might be confusing different parts of the stack. NVIDIA Dynamo for example supports vLLM as the inference engine. I think you should think of something like vLLM as more akin to GUnicorn, and llm-d as an application load balancer. And I guess something like NVIDIA Dynamo would be like Django.
- smarterclayton 1y ago
  llm-d is intended to be three clean layers: 1. Balance / schedule incoming requests to the right backend 2. Model server replicas that can run on multiple hardware topologies 3. Prefix caching hierarchy with well-tested variants for different use cases So it's a 3-tier architecture. The biggest difference with Dynamo is that llm-d is using the inference gateway extension - https://github.com/kubernetes-sigs/gateway-api-inference-extension https://github.com/kubernetes-sigs/gateway-api-inference-ext... - which brings Kubernetes owned APIs for managing model routing, request priority and flow control, LoRA support etc.
  - rdli 1y ago
    I would think that that the NVidia Dynamo SDK (pipelines) is a big difference as well (https://github.com/ai-dynamo/dynamo/tree/main/deploy/sdk/docs/sdk https://github.com/ai-dynamo/dynamo/tree/main/deploy/sdk/doc...), or am I missing something?
    - 1y ago

gitroom 1y ago

[dead]

Kemschumam 1y ago

What would be the benefit of this project over hosting VLLM in Ray?