2 ms·

> So I simultaneously can tell you that its smart people really thinking about every facet of the problem, and I can't tell you much more than that. "we do 197

> So I simultaneously can tell you that its smart people really thinking about every facet of the problem, and I can't tell you much more than that. "we do 1970s mainframe style timesharing" there, that was easy

kstrauser 1y ago
For real. Say it takes 1 machine 5 seconds to reply, and that a machine can only possibly form 1 reply at a time (which I doubt, but for argument). If the requests were regularly spaced, and they certainly won’t be, but for the sake of argument, then 1 machine could serve 17,000 requests per day, or 120,000 per week. At that rate, you’d need about 5,600 machines to serve 700M requests. That’s a lot to me, but not to someone who owns a data center. Yes, those 700M users will issue more than 1 query per week and they won’t be evenly spaced. However, I’d bet most of those queries will take well under 1 second to answer, and I’d also bet each machine can handle more than one at a time. It’s a large problem, to be sure, but that seems tractable.
- mh- 1y ago
  Yes. And batched inference is a thing, where intelligent grouping/bin packing and routing of requests happens. I expect a good amount of "secret sauce" is at this layer. Here's an entry-level link I found quickly on Google, OP: https://medium.com/@wearegap/a-brief-introduction-to-optimized-batched-inference-with-vllm-deddf5423d0c https://medium.com/@wearegap/a-brief-introduction-to-optimiz...
rootsudo 1y ago
Makes perfect sense, completely understand now!
brookst 1y ago
But that’s not accurate. There are all sorts of tricks around KV cache where different users will have the same first X bytes because they share system prompts, caching entire inputs / outputs when the context and user data is identical, and more. Not sure if you were just joking or really believe that, but for other peoples’ sake, it’s wildly wrong.
- kossTKR 1y ago
  Really? So the system recognises someone asked the same question and serves the same answer? And who on earth shares the exact same context? I mean i get the idea but sounds so incredibly rare it would mean absolutely nothing optimisation wise.
  - fc417fc802 1y ago
    Even if that were the case you wouldn't be wrong. Adding caching and deduplication (and clever routing and sharding, and ...) on top of timesharing doesn't somehow make it not timesharing anymore. The core observation about the raw numbers still applies.
  - brookst 1y ago
    Yes. It is not incredibly rare, it's incredibly common. A huge percentage of queries to retail LLMs are things like "hello" and "what can you do", with static system prompts that make the total context identical. It's worth maybe a 3% reduction in GPU usage. So call it a half billion dollars a year or so, for a medium to large service.
    - throwaway2037 1y ago
      > It's worth maybe a 3% reduction in GPU usage. So call it a half billion dollars a year or so, for a medium to large service. So if 3% is 500M, then annual spend is ~16.6B. That is medium sized these days?
claytongulick 1y ago
I'm pretty sure that's not right. They're definitely running cluster knoppix. :-)

kstrauser 1y ago

For real. Say it takes 1 machine 5 seconds to reply, and that a machine can only possibly form 1 reply at a time (which I doubt, but for argument). If the requests were regularly spaced, and they certainly won’t be, but for the sake of argument, then 1 machine could serve 17,000 requests per day, or 120,000 per week. At that rate, you’d need about 5,600 machines to serve 700M requests. That’s a lot to me, but not to someone who owns a data center. Yes, those 700M users will issue more than 1 query per week and they won’t be evenly spaced. However, I’d bet most of those queries will take well under 1 second to answer, and I’d also bet each machine can handle more than one at a time. It’s a large problem, to be sure, but that seems tractable.

mh- 1y ago
Yes. And batched inference is a thing, where intelligent grouping/bin packing and routing of requests happens. I expect a good amount of "secret sauce" is at this layer. Here's an entry-level link I found quickly on Google, OP: https://medium.com/@wearegap/a-brief-introduction-to-optimized-batched-inference-with-vllm-deddf5423d0c https://medium.com/@wearegap/a-brief-introduction-to-optimiz...

rootsudo 1y ago

Makes perfect sense, completely understand now!

brookst 1y ago

But that’s not accurate. There are all sorts of tricks around KV cache where different users will have the same first X bytes because they share system prompts, caching entire inputs / outputs when the context and user data is identical, and more. Not sure if you were just joking or really believe that, but for other peoples’ sake, it’s wildly wrong.

kossTKR 1y ago
Really? So the system recognises someone asked the same question and serves the same answer? And who on earth shares the exact same context? I mean i get the idea but sounds so incredibly rare it would mean absolutely nothing optimisation wise.
- fc417fc802 1y ago
  Even if that were the case you wouldn't be wrong. Adding caching and deduplication (and clever routing and sharding, and ...) on top of timesharing doesn't somehow make it not timesharing anymore. The core observation about the raw numbers still applies.
- brookst 1y ago
  Yes. It is not incredibly rare, it's incredibly common. A huge percentage of queries to retail LLMs are things like "hello" and "what can you do", with static system prompts that make the total context identical. It's worth maybe a 3% reduction in GPU usage. So call it a half billion dollars a year or so, for a medium to large service.
  - throwaway2037 1y ago
    > It's worth maybe a 3% reduction in GPU usage. So call it a half billion dollars a year or so, for a medium to large service. So if 3% is 500M, then annual spend is ~16.6B. That is medium sized these days?

claytongulick 1y ago

I'm pretty sure that's not right. They're definitely running cluster knoppix. :-)