5 ms·

> a stateful load balancer that is aware of each server's available slots Interesting. Curious to understand what a 'slot' is, in this context. Is that a llam

> a stateful load balancer that is aware of each server's available slots Interesting. Curious to understand what a 'slot' is, in this context. Is that a llama.cpp specific application-layer state that llama.cpp makes available? Or is this an application-layer state that is being inferred? If later, how?

tjfontaine 2y ago
Looks like it’s to only have one completion running at a time currently, https://github.com/distantmagic/paddler/blob/main/loadbalancer/LoadBalancerRequest.go#L10 https://github.com/distantmagic/paddler/blob/main/loadbalanc... I was curious if there was any other cache tracking happening
- mcharytoniuk 2y ago
  Just open an issue if you need anything. I want to make it as good and helpful as possible. Every kind of feedback is appreciated.
asne11 2y ago
"slot" is a processing unit. Either GPU or CPU. I believe `llama.c` is only CPU so I'm guessing 1 slot = 1 core (or thread)?
- new299 2y ago
  llama.cpp is not CPU only…
- kgeist 2y ago
  Llama.cpp can run on CPU, on GPU, or in mixed mode (some layers run on CPU and some on GPU if you don't have enough VRAM).
- mcharytoniuk 2y ago
  It divides the context into smaller "slots", so it can process requests concurrently with continuous batching. See also: https://github.com/ggerganov/llama.cpp/tree/master/examples/server https://github.com/ggerganov/llama.cpp/tree/master/examples/...
riedel 2y ago
What does stateful mean: I always wonder how loading states of users is done, it seems that one can call `llama_state_set_data` , does this load balancer create a central store for such states? What is the overhead of transfering state?
- mcharytoniuk 2y ago
  Currently, it is a single instance in memory, so it doesn't transfer state. HA is on the roadmap; only then will it need some kind of distributed state store. Local states are reported by the agents installed alongside llama.cpp to the load balancer. That means they can be dynamically added and removed; it doesn't need a central configuration.
kherud 2y ago
I think this comment explains it https://github.com/ggerganov/llama.cpp/discussions/4130#discussioncomment-8053636 https://github.com/ggerganov/llama.cpp/discussions/4130#disc... As far as I understand (and mcharytoniuk should better confirm this), llama.cpp allows to chunk the context window of an LLM into independent blocks, such that multiple requests can be processed in a single inference. I think due to the auto-regressive nature of LLMs, you also don't have to wait for all sequences to finish to output them. As soon as one sequence finishes, you can use its "slot" in the context window for other requests.
- mcharytoniuk 2y ago
  Yes, exactly. You can split the available context into "slots" (chunks) so it can handle multipe requests concurrently. The number of them is configurable.
BoredPositron 2y ago
Probably just free accelerators.

tjfontaine 2y ago

Looks like it’s to only have one completion running at a time currently, https://github.com/distantmagic/paddler/blob/main/loadbalancer/LoadBalancerRequest.go#L10 https://github.com/distantmagic/paddler/blob/main/loadbalanc... I was curious if there was any other cache tracking happening

mcharytoniuk 2y ago
Just open an issue if you need anything. I want to make it as good and helpful as possible. Every kind of feedback is appreciated.

asne11 2y ago

"slot" is a processing unit. Either GPU or CPU. I believe `llama.c` is only CPU so I'm guessing 1 slot = 1 core (or thread)?

new299 2y ago
llama.cpp is not CPU only…
kgeist 2y ago
Llama.cpp can run on CPU, on GPU, or in mixed mode (some layers run on CPU and some on GPU if you don't have enough VRAM).
mcharytoniuk 2y ago
It divides the context into smaller "slots", so it can process requests concurrently with continuous batching. See also: https://github.com/ggerganov/llama.cpp/tree/master/examples/server https://github.com/ggerganov/llama.cpp/tree/master/examples/...

riedel 2y ago

What does stateful mean: I always wonder how loading states of users is done, it seems that one can call `llama_state_set_data` , does this load balancer create a central store for such states? What is the overhead of transfering state?

mcharytoniuk 2y ago
Currently, it is a single instance in memory, so it doesn't transfer state. HA is on the roadmap; only then will it need some kind of distributed state store. Local states are reported by the agents installed alongside llama.cpp to the load balancer. That means they can be dynamically added and removed; it doesn't need a central configuration.

kherud 2y ago

I think this comment explains it https://github.com/ggerganov/llama.cpp/discussions/4130#discussioncomment-8053636 https://github.com/ggerganov/llama.cpp/discussions/4130#disc... As far as I understand (and mcharytoniuk should better confirm this), llama.cpp allows to chunk the context window of an LLM into independent blocks, such that multiple requests can be processed in a single inference. I think due to the auto-regressive nature of LLMs, you also don't have to wait for all sequences to finish to output them. As soon as one sequence finishes, you can use its "slot" in the context window for other requests.

mcharytoniuk 2y ago
Yes, exactly. You can split the available context into "slots" (chunks) so it can handle multipe requests concurrently. The number of them is configurable.

BoredPositron 2y ago

Probably just free accelerators.