6 ms·

I’m slightly confuse as to how all this works. Do the GPUs just sit there with the models on them when the models are not in use? I guess I’d assumed this sort

I’m slightly confuse as to how all this works. Do the GPUs just sit there with the models on them when the models are not in use? I guess I’d assumed this sort of thing would be allocated dynamically. Of course, there’s a benefit to minimizing the number of times you load a model. But surely if a GPU+model is idle for more than a couple minutes it could be freed? (I’m not an AI guy, though—actually I’m used to asking SLURM for new nodes with every run I do!)

make3 1y ago
the models are huge, so not a single (latest gen) one can fit on a single GPU. It's likely that these are small unpopular (non flagship) models, or that they only pack eg one layer of each model.
- svachalek 1y ago
  Per the very short article, the solution was to pack multiple models per GPU.
  - make3 1y ago
    yes but that could mean a layer per model
smallnix 1y ago
> I guess I’d assumed this sort of thing would be allocated dynamically At the scale of a hyperscaler I think Alibaba is the one that would be doing that. AWS, Azure and I assume Alibaba do lease/rent data centers, but someone has to own the servers / GPU racks. I know there are specialized companies like nscale (and more further down the chain) in the mix, but I always assumed they only lease out fixed capacity.
yorwba 1y ago
The paper is about techniques to do that dynamic allocation to maximize utilization without incurring unacceptable latencies. If you let a GPU sit idle for several minutes after serving a single request, you're setting money on fire. So they reuse it for a different model as soon as possible, starting even before the first request is finished, because: If you don't have a dedicated GPU for a model, are you going to wait for a multi-gigabyte transfer before each request? So they have a dedicated GPU (or two, one for prefill, one for decode) for a group of models that are processed in an interleaved fashion, scheduled such that they stay within the latency budget.
svachalek 1y ago
Models take a lot of VRAM which is tightly coupled to the GPU so yeah, it's basically sitting there with the model waiting for use. I'm sure they probably do idle out but a few minutes of idle time is a lot of waste--possibly the full 82% mentioned. In this case they optimized by letting the GPUs load multiple models and sharing the load out by token.
- andy_ppp 1y ago
  How does this work with anything but trivially small context sizes!?
  - jychang 1y ago
    Tensor parallelism, so you only need to store a fraction of kv cache per gpu.
- jychang 1y ago
  They definitely won't idle out- if they idle out, it'll take on the order of up to 60 seconds to load the model back into VRAM, depending on the model. That's an eternity for a request. I highly doubt they will timeout any model they serve.
  - all2 1y ago
    If I had to handle this problem, I'd do some kind of "split on existing loaded GPUs" for new sessions, and then when some cap is hit, spool an additional GPU in the background and the transfer the new session to that GPU as soon as the model is loaded. I'd have to play with the configuration and load calcs, but I'm sure there's a low param, neat solution to the request/service problem.
  - arthurcolle 1y ago
    That's why deepseek only serves two models
  - godelski 1y ago
    > That's an eternity for a request. I highly doubt they will timeout any model they serve. That's what easing functions are for. Let's say 10 GPUs are in use. You keep another 3 with the model loaded. If demand increases slowly you slowly increase your headroom. If demand increases rapidly, you also increase rapidly. The correct way to do this is more complicated and you should model based on your usage history, but if you have sufficient headroom then very few should be left idle. Remember that these models do requests in batches. If they don't timeout models, they're throwing money down the drain. Though that wouldn't be uncommon.
citizenpaul 1y ago
>Do the GPUs just sit there with the models on them when the models are not in use I've assumed that as well. It makes sense to me since loading up a model locally takes a while. I wonder if there is some sort of better way I'm not in the know about. That or too GPU poor to know about.
miki123211 1y ago
Loading a model takes at least a few seconds, usually more, depending on model size, disk / network speed and a bunch of other factors. If you're using an efficient inference engine like VLLM, you're adding compilation into the mix, and not all of that is fully cached yet. If that kind of latency isn't acceptable to you, you have to keep the models loaded. This (along with batching) is why large local models are a dumb and wasteful idea if you're not serving them at enterprise scale.
- cnr 1y ago
  Can you elaborate the last statement? Don't quite understand why loading local LLM to GPU RAM, using it for the job and then "ejecting" is "dumb and wasteful" idea?
  - carderne 1y ago
    Layman understanding: Because as a function of hardware and electricity costs, a “cloud” GPU will be many times more efficient per output token. You aren’t loading/offloading models and don’t have any parts of the GPU waiting for input. Everything is fully saturated always.
  - OJFord 1y ago
    I believe GP means it still to be connected to 'if this kind of latency is unacceptable to you' - i.e. you can't load/use/unload, you have to keep it in RAM all the time. In that case it's massively increasing your memory requirement not just to the peak the model needs, but to + whatever the other biggest use might be that'll be inherently concurrent with it.
- behnamoh 1y ago
  > This (along with batching) is why large local models are a dumb and wasteful idea if you're not serving them at enterprise scale. Local models are never a dumb idea. The only time it's dumb to use them in an enterprise is if the infra is Mac Studio with M3 Ultra because pp time is terrible.
- cnr 1y ago

make3 1y ago

the models are huge, so not a single (latest gen) one can fit on a single GPU. It's likely that these are small unpopular (non flagship) models, or that they only pack eg one layer of each model.

svachalek 1y ago
Per the very short article, the solution was to pack multiple models per GPU.
- make3 1y ago
  yes but that could mean a layer per model

smallnix 1y ago

> I guess I’d assumed this sort of thing would be allocated dynamically At the scale of a hyperscaler I think Alibaba is the one that would be doing that. AWS, Azure and I assume Alibaba do lease/rent data centers, but someone has to own the servers / GPU racks. I know there are specialized companies like nscale (and more further down the chain) in the mix, but I always assumed they only lease out fixed capacity.

yorwba 1y ago

The paper is about techniques to do that dynamic allocation to maximize utilization without incurring unacceptable latencies. If you let a GPU sit idle for several minutes after serving a single request, you're setting money on fire. So they reuse it for a different model as soon as possible, starting even before the first request is finished, because: If you don't have a dedicated GPU for a model, are you going to wait for a multi-gigabyte transfer before each request? So they have a dedicated GPU (or two, one for prefill, one for decode) for a group of models that are processed in an interleaved fashion, scheduled such that they stay within the latency budget.

svachalek 1y ago

Models take a lot of VRAM which is tightly coupled to the GPU so yeah, it's basically sitting there with the model waiting for use. I'm sure they probably do idle out but a few minutes of idle time is a lot of waste--possibly the full 82% mentioned. In this case they optimized by letting the GPUs load multiple models and sharing the load out by token.

andy_ppp 1y ago
How does this work with anything but trivially small context sizes!?
- jychang 1y ago
  Tensor parallelism, so you only need to store a fraction of kv cache per gpu.
jychang 1y ago
They definitely won't idle out- if they idle out, it'll take on the order of up to 60 seconds to load the model back into VRAM, depending on the model. That's an eternity for a request. I highly doubt they will timeout any model they serve.
- all2 1y ago
  If I had to handle this problem, I'd do some kind of "split on existing loaded GPUs" for new sessions, and then when some cap is hit, spool an additional GPU in the background and the transfer the new session to that GPU as soon as the model is loaded. I'd have to play with the configuration and load calcs, but I'm sure there's a low param, neat solution to the request/service problem.
- arthurcolle 1y ago
  That's why deepseek only serves two models
- godelski 1y ago
  > That's an eternity for a request. I highly doubt they will timeout any model they serve. That's what easing functions are for. Let's say 10 GPUs are in use. You keep another 3 with the model loaded. If demand increases slowly you slowly increase your headroom. If demand increases rapidly, you also increase rapidly. The correct way to do this is more complicated and you should model based on your usage history, but if you have sufficient headroom then very few should be left idle. Remember that these models do requests in batches. If they don't timeout models, they're throwing money down the drain. Though that wouldn't be uncommon.

citizenpaul 1y ago

>Do the GPUs just sit there with the models on them when the models are not in use I've assumed that as well. It makes sense to me since loading up a model locally takes a while. I wonder if there is some sort of better way I'm not in the know about. That or too GPU poor to know about.

miki123211 1y ago

Loading a model takes at least a few seconds, usually more, depending on model size, disk / network speed and a bunch of other factors. If you're using an efficient inference engine like VLLM, you're adding compilation into the mix, and not all of that is fully cached yet. If that kind of latency isn't acceptable to you, you have to keep the models loaded. This (along with batching) is why large local models are a dumb and wasteful idea if you're not serving them at enterprise scale.

cnr 1y ago
Can you elaborate the last statement? Don't quite understand why loading local LLM to GPU RAM, using it for the job and then "ejecting" is "dumb and wasteful" idea?
- carderne 1y ago
  Layman understanding: Because as a function of hardware and electricity costs, a “cloud” GPU will be many times more efficient per output token. You aren’t loading/offloading models and don’t have any parts of the GPU waiting for input. Everything is fully saturated always.
- OJFord 1y ago
  I believe GP means it still to be connected to 'if this kind of latency is unacceptable to you' - i.e. you can't load/use/unload, you have to keep it in RAM all the time. In that case it's massively increasing your memory requirement not just to the peak the model needs, but to + whatever the other biggest use might be that'll be inherently concurrent with it.
behnamoh 1y ago
> This (along with batching) is why large local models are a dumb and wasteful idea if you're not serving them at enterprise scale. Local models are never a dumb idea. The only time it's dumb to use them in an enterprise is if the infra is Mac Studio with M3 Ultra because pp time is terrible.
cnr 1y ago