4 ms·
I'm not sure that I buy their conclusion that more compute during inference is good. Yes, batch=1 inference is mostly memory bandwidth bound, not GPU compute b
by jychang 7mo ago
I'm not sure that I buy their conclusion that more compute during inference is good.
Yes, batch=1 inference is mostly memory bandwidth bound, not GPU compute bound. But no provider does batch=1 inference. Everyone groups all the requests into a batch, and the GPU computes them together.
With a fused kernel, that means the GPU streams the tensors from VRAM, and does a bunch of compute on different conversations in the batch, at the same time.
If they increase the amount of compute required per token, that just reduces the maximum batch size a GPU can handle. In practice, yes this does mean each GPU can serve less users. Providers aren't leaving GPU cores idle normally during inference.
- zozbot234 7mo ago> Everyone groups all the requests into a batch, and the GPU computes them together. You're only saving on fetching read-only parameters, and not even on that if you're using MoE models where each inference in the batch might require a different expert (unless you rearrange batches so that sharing experts becomes more likely, but that's difficult since experts change per-token or even per-layer). Everything else - KV-cache, activations - gets multiplied by your batch size. You scale both compute and memory pressure by largely the same amount. Yes, GPUs are great at hiding memory fetch latency, but that applies also to n=1 inference.
- jychang 7mo agoWell, the actual inference providers put each expert on its own single GPU. Deepseek explicitly does this. Read-only parameters is also usually the majority of space. Deepseek is 700GB of params. Meanwhile kv cache is small (Deepseek is about 7GB at max context) and ssm/conv1d cache is even smaller- IIRC Qwen 3.5 is 146MB per token regardless of context size. Not sure about how Mamba-3 works, but I suspect read-only parameters are still a significant amount of memory bandwidth. I guess the question isn't whether compute is 1:1 with memory, but rather if you run out of compute before you run out of vram adding more users.
- zozbot234 7mo ago> Well, the actual inference providers put each expert on its own single GPU. Experts are usually chosen on a per-layer basis, not just by token, so I'd think this requires having lots of GPU's to make it worthwhile. You could do it with a single physical GPU by switching expert-layer mixes in a round-robin fashion after the batch for any single expert-layer mix is completed (essentially a refined version of expert offloading). But still, not easy.
- jychang 7mo agoCorrect, but note that's exactly what inference providers do. https://arxiv.org/pdf/2412.19437 https://arxiv.org/pdf/2412.19437 > The minimum deployment unit of the decoding stage consists of 40 nodes with 320 GPUs. The attention part employs TP4 with SP, combined with DP80, while the MoE part uses EP320. EP320 means expert parallelism, each on 320 GPUs.
- yorwba 7mo agoTheir latency measurements comparing Mamba-2 and Mamba-3 are done with a batch size of 128. It doesn't seem like Mamba-2 was compute-bound even at that batch size.
- jychang 7mo agoWell, Deepseek batch sizes are something like 8192, so 128 isn't much. https://arxiv.org/html/2412.19437v1 https://arxiv.org/html/2412.19437v1 "the batch size per expert is relatively small (usually within 256 tokens)"
- btown 7mo agoThroughput is indeed king for the standard-tier mindshare-capture play. But there are many who would pay multiple times the current cost for agentic systems for engineers and executives, if it meant a meaningful reduction in latency. The economics could work extremely well.
- Havoc 7mo agoWhy would execs need latency? I can see it for engineering - coding with slow ai is painful
- metanonsense 7mo agoImpatient execs can also be painful. EDIT: Writing this while I am waiting for Codex to complete, so I may enjoy slow AI more than the usual developer ;-)
- btown 7mo agoThe economic effect of latency is measured not by the incremental productivity itself, but by the combined economic downforce of thousands of resulting HN and Reddit comments :)
- notnullorvoid 7mo agoFocusing on needs of providers isn't a very good long term strategy if you believe compute will eventually move to self hosted and on premises solutions where large batch sizes aren't needed.
- jychang 7mo agoThat's a foolish take. That's like gamers thinking most of Nvidia's revenue coming from gaming GPUs, so Nvidia should prioritize gamers. Inference is ruled by inference providers, not local. Local inference is a rounding error, and will remain as such unless there is economic incentive otherwise.
- sroussey 7mo agoLocal has a batch size of 1. If you are already memory bound then you leave compute on the table. Why not use it? Not sure they target local though…