5 ms·

For those looking to save time, the answer is batched inference. Pretty much running multiple people's "prompts" through a model instance at the same time inste

For those looking to save time, the answer is batched inference. Pretty much running multiple people's "prompts" through a model instance at the same time instead of just really tightly timesharing each model instance. This is also why you may experience a variance in replies when using these services, even when you set the temperature to 0 and the seed to a fixed value. It's cause you don't control the other prompts yours get batched with. Could this be a data exfiltration attack vector? Probably, I didn't "research" that far.

pcwelder 1y ago
> other prompts yours get batched with Why would batching lead to variance?
- Hendrikto 1y ago
  Because these models are context-sensitive. Every token can influence the output.
  - simianwords 1y ago
    I believe they are talking about latency variance. Batching can increase variance because you may have to wait for enough prompts to get to the batch size.
    - perching_aix 1y ago
      No, I meant that the responses will be different run-to-run. [0] [0] https://152334h.github.io/blog/non-determinism-in-gpt-4/ https://152334h.github.io/blog/non-determinism-in-gpt-4/
      - exe34 1y ago
        Variance based on actual randomness would be one thing, but to me variance based on what other people are running seems concerning, for reasons I can't quite articulate. I don't want the model to reply to a question in one domain based on what a large group of other people are thinking in a different domain (e.g. if they're discussing the news with chatgpt).
        zackangelo 1y ago
        This definitely happens, and I'm surprised it's not talked about more often. Some attention kernels are more susceptible to this than others (I've found that paged attention is better than just naive attention, for example).
        exe34 1y ago
        To be fair, I suppose people do it too - if you ask me a question about A, often as not the answer will be coloured by the fact that I just learnt about B.
  - immibis 1y ago
    But not the tokens that don't even feed into your output because they're feeding into someone else's output. Separate items in batches don't get mixed up with each other - they just run the model separately on each item at the same time, like SIMD.
- jerpint 1y ago
  Batching can lead to variance with things like batchnorm but most transformers use layer norm to avoid this problem
  - amelius 1y ago
    Batchnorm can only have an effect between batches during training, not inference.
- kouteiheika 1y ago
  > Why would batching lead to variance? Depending on the shape of the data a slightly different kernel implementation (for e.g. matrix multiplication, etc.) will be the most optimal, and those will give slightly different results. There could also be other sources of non-determinism depending on the implementation (e.g. some kernels are inherently not entirely deterministic as they use tricks to go faster).
  - zxexz 1y ago
    Yep, this. I see a lot of other worryingly confident answers in the thread that are wrong. SGLang finally has at least some notes[0], but I’m always surprised there isn’t more of a community wide effort to trace down the sources of indeterminism. [0] https://docs.sglang.ai/references/faq.html https://docs.sglang.ai/references/faq.html
  - bhickey 1y ago
    Some of the non-determinism mentioned above manifests as sensitivity to _where_ data falls within a batch.
    - tough 1y ago
      In my experience with other regular models, once the context starts to fill up, quality starts to degrade. wouldn't getting batched at the end of a batch, have a similar -effect- on the results, where your prompt might recieve overall less attention focused into it, if the context window is almost full? Idk just going by the vibes
  - delusional 1y ago
    > not entirely deterministic There's a Nobel prize waiting for you if that's the case. I'll assume you meant theoretically consistent or accurate.
- matusp 1y ago
  In some mixture-of-experts approaches, samples or tokens are being distributed among experts. The experts are selected by trying to predict what is a good expert-sample match. Depending on your neighbors in the batch, you might be assigned different experts.
- imtringued 1y ago
  Attention doesn't get batched and the runtime of attention for a given users token depends on the total context length. Hence even in the ideal scenario of you getting a dedicated attention calculating GPU, the MLP calculating GPU doing batching will have to wait for the slowest user. In the worst case scenario you are sharing a single attention calculating GPU with someone who has a super long context window, then that guy will be hogging most of the memory bandwidth of the GPU, even though you both are generating the same quantity of tokens. This means that in the distributed setting, you will not only need dedicated GPUs for the model and attention calculations, you will also need to duplicate the whole setup for a variety of context lengths, so that long contexts are batches alongside other long contexts and short contexts are batches alongside other short contexts.
yjftsjthsd-h 1y ago
> Pretty much running multiple people's "prompts" through a model instance at the same time instead of just really tightly timesharing each model instance. I naively assumed providers did that with all models. Or does it only work for this (family of?) model(s)?
- hansvm 1y ago
  It works for a lot of families but not all. You need a high enough degree of sharing of model weights between different queries for that to make sense (memory access being the usual bottleneck nowadays, though smaller models see something similar with matmul batch efficiencies for CPU related reasons). Fully connected transformers trivially work (every weight for every query). MoE works beyond a certain size or with certain types of mixing (still using every weight, or using a high enough fraction that there's some sharing with batches of 20+ queries). As you push further that direction though (lots of techniques, but the key point being accessing less of the model at once and bypassing some of it for each query), you need larger and larger batches for those efficiency gains to materialize. At some point it becomes untenable because of latency waiting for batches of data, and past that it becomes untenable because of the volume of query data.
VectorLock 1y ago
Sounds like an amazing attack vector if your prompts get mixed with other's.
taneq 1y ago
Wow, almost like Deepseek’s impressive performance is the result of optimisation by smart engineers.
- perching_aix 1y ago
  Not sure why the snarky tone, didn't say or imply otherwise, nor did anyone else in the thread so far that I could see.
  - taneq 1y ago
    It wasn't meant to come across that snarky. Sorry about that. :/
- draw_down 1y ago
  [dead]
energy123 1y ago
What's the average batch size?
larodi 1y ago
Batching. Yes. And one thing it can help you locally is when you rate certain content and want to make sure it didn’t hallucinate. So you toss 3 or 5 times or… batch_size times .) Curious that batch if has been there from day one, but it takes a while for people to see/grasp/grok it.
deleted 1y ago
[deleted]

pcwelder 1y ago

> other prompts yours get batched with Why would batching lead to variance?

Hendrikto 1y ago
Because these models are context-sensitive. Every token can influence the output.
- simianwords 1y ago
  I believe they are talking about latency variance. Batching can increase variance because you may have to wait for enough prompts to get to the batch size.
  - perching_aix 1y ago
    No, I meant that the responses will be different run-to-run. [0] [0] https://152334h.github.io/blog/non-determinism-in-gpt-4/ https://152334h.github.io/blog/non-determinism-in-gpt-4/
    - exe34 1y ago
      Variance based on actual randomness would be one thing, but to me variance based on what other people are running seems concerning, for reasons I can't quite articulate. I don't want the model to reply to a question in one domain based on what a large group of other people are thinking in a different domain (e.g. if they're discussing the news with chatgpt).
      - zackangelo 1y ago
        This definitely happens, and I'm surprised it's not talked about more often. Some attention kernels are more susceptible to this than others (I've found that paged attention is better than just naive attention, for example).
        exe34 1y ago
        To be fair, I suppose people do it too - if you ask me a question about A, often as not the answer will be coloured by the fact that I just learnt about B.
- immibis 1y ago
  But not the tokens that don't even feed into your output because they're feeding into someone else's output. Separate items in batches don't get mixed up with each other - they just run the model separately on each item at the same time, like SIMD.
jerpint 1y ago
Batching can lead to variance with things like batchnorm but most transformers use layer norm to avoid this problem
- amelius 1y ago
  Batchnorm can only have an effect between batches during training, not inference.
kouteiheika 1y ago
> Why would batching lead to variance? Depending on the shape of the data a slightly different kernel implementation (for e.g. matrix multiplication, etc.) will be the most optimal, and those will give slightly different results. There could also be other sources of non-determinism depending on the implementation (e.g. some kernels are inherently not entirely deterministic as they use tricks to go faster).
- zxexz 1y ago
  Yep, this. I see a lot of other worryingly confident answers in the thread that are wrong. SGLang finally has at least some notes[0], but I’m always surprised there isn’t more of a community wide effort to trace down the sources of indeterminism. [0] https://docs.sglang.ai/references/faq.html https://docs.sglang.ai/references/faq.html
- bhickey 1y ago
  Some of the non-determinism mentioned above manifests as sensitivity to _where_ data falls within a batch.
  - tough 1y ago
    In my experience with other regular models, once the context starts to fill up, quality starts to degrade. wouldn't getting batched at the end of a batch, have a similar -effect- on the results, where your prompt might recieve overall less attention focused into it, if the context window is almost full? Idk just going by the vibes
- delusional 1y ago
  > not entirely deterministic There's a Nobel prize waiting for you if that's the case. I'll assume you meant theoretically consistent or accurate.
matusp 1y ago
In some mixture-of-experts approaches, samples or tokens are being distributed among experts. The experts are selected by trying to predict what is a good expert-sample match. Depending on your neighbors in the batch, you might be assigned different experts.
imtringued 1y ago
Attention doesn't get batched and the runtime of attention for a given users token depends on the total context length. Hence even in the ideal scenario of you getting a dedicated attention calculating GPU, the MLP calculating GPU doing batching will have to wait for the slowest user. In the worst case scenario you are sharing a single attention calculating GPU with someone who has a super long context window, then that guy will be hogging most of the memory bandwidth of the GPU, even though you both are generating the same quantity of tokens. This means that in the distributed setting, you will not only need dedicated GPUs for the model and attention calculations, you will also need to duplicate the whole setup for a variety of context lengths, so that long contexts are batches alongside other long contexts and short contexts are batches alongside other short contexts.

yjftsjthsd-h 1y ago

> Pretty much running multiple people's "prompts" through a model instance at the same time instead of just really tightly timesharing each model instance. I naively assumed providers did that with all models. Or does it only work for this (family of?) model(s)?

hansvm 1y ago
It works for a lot of families but not all. You need a high enough degree of sharing of model weights between different queries for that to make sense (memory access being the usual bottleneck nowadays, though smaller models see something similar with matmul batch efficiencies for CPU related reasons). Fully connected transformers trivially work (every weight for every query). MoE works beyond a certain size or with certain types of mixing (still using every weight, or using a high enough fraction that there's some sharing with batches of 20+ queries). As you push further that direction though (lots of techniques, but the key point being accessing less of the model at once and bypassing some of it for each query), you need larger and larger batches for those efficiency gains to materialize. At some point it becomes untenable because of latency waiting for batches of data, and past that it becomes untenable because of the volume of query data.

VectorLock 1y ago

Sounds like an amazing attack vector if your prompts get mixed with other's.

taneq 1y ago

Wow, almost like Deepseek’s impressive performance is the result of optimisation by smart engineers.

perching_aix 1y ago
Not sure why the snarky tone, didn't say or imply otherwise, nor did anyone else in the thread so far that I could see.
- taneq 1y ago
  It wasn't meant to come across that snarky. Sorry about that. :/
draw_down 1y ago
[dead]

energy123 1y ago

What's the average batch size?

larodi 1y ago

Batching. Yes. And one thing it can help you locally is when you rate certain content and want to make sure it didn’t hallucinate. So you toss 3 or 5 times or… batch_size times .) Curious that batch if has been there from day one, but it takes a while for people to see/grasp/grok it.

deleted 1y ago

[deleted]