14 ms·

I work at Google on these systems everyday (caveat this is my own words not my employers)). So I simultaneously can tell you that its smart people really thinki

I work at Google on these systems everyday (caveat this is my own words not my employers)). So I simultaneously can tell you that its smart people really thinking about every facet of the problem, and I can't tell you much more than that. However I can share this written by my colleagues! You'll find great explanations about accelerator architectures and the considerations made to make things fast. https://jax-ml.github.io/scaling-book/ https://jax-ml.github.io/scaling-book/ In particular your questions are around inference which is the focus of this chapter https://jax-ml.github.io/scaling-book/inference/ https://jax-ml.github.io/scaling-book/inference/ Edit: Another great resource to look at is the unsloth guides. These folks are incredibly good at getting deep into various models and finding optimizations, and they're very good at writing it up. Here's the Gemma 3n guide, and you'll find others as well. https://docs.unsloth.ai/basics/gemma-3n-how-to-run-and-fine-tune https://docs.unsloth.ai/basics/gemma-3n-how-to-run-and-fine-...

tough 1y ago
Doesn't google have TPU's that makes inference of their own models much more profitable than say having to rent out NVDIA cards? Doesn't OpenAI depend mostly on its relationship/partnership with Microsoft to get GPUs to inference on? Thanks for the links, interesting book!
- ActorNightly 1y ago
  Yes. Google is probably gonna win the LLM game tbh. They had a massive head start with TPUs which are very energy efficient compared to Nvidia Cards.
  - fakedang 1y ago
    Yeah honestly. They could just try selling solutions and SLAs combining their TPU hardware with on-prem SOTA models and practically dominate enterprise. From what I understand, that's GCP's gameplay too for most regulated enterprise clients.
    - ActorNightly 1y ago
      Googles bread and butter is advertising, so they have a huge interest in keeping things in house. Data is more valuable to them than money from hardware sales. Even then, I think that their primary use case is going to be consumer grade good AI on phones. I dunno why Gemma QAT model fly so low on the radar, but you can basically get full scale Llamma 3 like performance from a single 3090 now, at home.
      - fakedang 1y ago
        https://www.cnbc.com/2025/04/09/google-will-let-companies-run-gemini-models-in-their-own-data-centers.html https://www.cnbc.com/2025/04/09/google-will-let-companies-ru... Google has already started the process of letting companies self-host Gemini, even on NVidia Blackwell GPUs. Although imho, they really should bundle it with their TPUs as a turnkey solution for those clients who haven't invested in large scale infra like DCs yet.
        ActorNightly 1y ago
        Its the same format as other software - you release the actual software for free but offer managed services that work with that software way better and easier.
        fakedang 1y ago
        Yeah but those are on Google's managed cloud, and not onprem. But that recent announcement has been specifically for Google Distributed Cloud, which is huge. My point was a bit more specific though. To elaborate, I know of a number of publicly traded companies (USD $200M+ market cap) globally which have identified use cases for onprem AI and want to implement them actively but cannot, because they lack the knowhow to work with onprem, and hiring talent to implement that is just extremely expensive. Google should simply provide it as a turnkey bundle and milk them for it.
        to11mtm 1y ago
        My guess is that either google want's a high level of physical control over their TPUs, or they have one sort of deal or another with NVidia and don't want to step on their toes. And also, Google's track record with hardware.
  - baxtr 1y ago
    The only one who can stop Google is Google. They’ll definitely have the best model, but there is a chance they will f*up the product / integration into their products.
    - scarface_74 1y ago
      It would take talent for them to mess up hosting businesses who want to use their TPUs on GCP. But then again even there, their reputation for abandoning products, lack of customer service, condescension when it came to large enterprises’ “legacy tech” lets Microsoft who is king of hand holding big enterprise and even AWS run rough shod over them. When I was at AWS ProServe, we didn’t even bother coming up with talking points when competing with GCP except to point out how they abandon services. Was it partially FUD? Probably. But it worked.
      - deleted 1y ago
        [deleted]
      - JoshuaDavid 1y ago
        Google employees collectively have a lot of talent.
        bee_rider 1y ago
        A truly astonishing amount of talent applied to… hosting emails very well, and losing the search battle against SEO spammers.
        big_hacker 1y ago
        Well, Search had no chance when the sites also make money from Google ads. Google fucked their Search by creating themselves incentives for bounce rate.
      - serf 1y ago
        >It would take talent for them to mess up hosting businesses who want to use their TPUs on GCP. there are few groups as talented at losing a head start as google.
  - davedx 1y ago
    But they’re ASICs so any big architecture changes will be painful for them right?
    - deleted 1y ago
      [deleted]
    - edoceo 1y ago
      I'd think no. They have the hardware and software experience, likely have next and next-next plans in place already. The big hurdle is money, which G has a bunch of.
    - llm_nerd 1y ago
      TPUs are accelerators that accelerate the common operations found in neural nets. A big part is simply a massive number of matrix FMA units to process enormous matrix operations, which comprises the bulk of doing a forward pass through a model. Caching enhancements and massively growing memory was necessary to facilitate transformers, but on the hardware side not a huge amount has changed and the fundamentals from years ago still powers the latest models. The hardware is just getting faster and with more memory and more parallel processing units. And later getting more data types to enable hardware-enabled quantization. So it isn't like Google designed a TPU for a specific model or architecture. They're pretty general purpose in a narrow field (oxymoron, but you get the point). The set of operations Google designed into a TPU is very similar to what nvidia did, and it's about as broadly capable. But Google owns the IP and doesn't pay the premium and gets to design for their own specific needs.
      - saagarjha 1y ago
        There are plenty of matrix multiplies in the backward pass too. Obviously this is less useful when serving but it's useful for training.
  - stogot 1y ago
    Hasn’t the Inferentia chip been around long enough to make the same argument? AWS and Google probably have the same order of magnitude of their own custom chips
    - saagarjha 1y ago
      Inferentia has a generally worse stack but yes
  - benreesman 1y ago
    Google will win the LLM game if the LLM game is about compute, which is the common wisdom and maybe true, but not foreordained by God. There's an argument that if compute was the dominant term that Google would never have been anything but leading by a lot. Personally right now I see one clear leader and one group going 0-99 like a five sigma cosmic ray: Anthropic and the PRC. But this is because I believe/know that all the benchmarks are gamed as hell, its like asking if a movie star had cosmetic surgery. On quality, Opus 4 is 15x the cost and sold out / backordered. Qwen 3 is arguably in next place. In both of those cases, extreme quality expert labeling at scale (assisted by the tool) seems to be the secret sauce. Which is how it would play out if history is any guide: when compute as a scaling lever starts to flatten, you expert label like its 1987 and claim its compute and algorithms until the government wises up and stops treating your success persobally as a national security priority. It's the easiest trillion Xi Xianping ever made: pretending to think LLMs are AGI too, fast following for pennies on the dollar, and propping up a stock market bubble to go with the fentanyl crisis? 9-D chess. It's what I would do about AI if I were China. Time will tell.
    - 0_____0 1y ago
      I believe Google might win the LLM game simply because they have the infrastructure to make it profitable - via ads. All the LLM vendors are going to have to cope with the fact that they're lighting money on fire, and Google have the paying customers (advertisers) and with the user-specific context they get from their LLM products, one of the juciest and most targetable ad audiences of all time.
      - throwaway2037 1y ago
        This is one of the best insights after reading 100+ posts here. You are talking about existing demand and existing relationships from their advert business. These customers are happy with Google results. The Google marketing team will be carefully defining new advert products that employ LLMs. I would only offer one disagreement with your post: There will not be a single winner in LLMs. The landscape is so large that we will have multiple winners in different areas. Example: Google might fail in B2C LLM (chatbot that answers your questions), but will certainly be (wildly?) successful in B2B for adverts.
- canyon289 1y ago
  Im a research person building models so I can't answer your questions well (save for one part) That is, as a research person using our GPUs and TPUs I see first hand how choices from the high level python level, through Jax, down to the TPU architecture all work together to make training and inference efficient. You can see a bit of that in the gif on the front page of the book. https://jax-ml.github.io/scaling-book/ https://jax-ml.github.io/scaling-book/ I also see how sometimes bad choices by me can make things inefficient. Luckily for me if my code/models are running slow I can ping colleagues who are able to debug at both a depth and speed that is quite incredible. And because were on HN I want to preemptively call out my positive bias for Google! It's a privilege to be able to see all this technology first hand, work with great people, and do my best to ship this at scale across the globe.
jackhalford 1y ago
Why does the unsloth guide for gemma 3n say: > llama.cpp an other inference engines auto add a <bos> - DO NOT add TWO <bos> tokens! You should ignore the <bos> when prompting the model! That makes the want to try exactly that? Weird
ignoramous 1y ago
> Another great resource to look at is the unsloth guides. And folks at LMSys: https://lmsys.org/blog/ https://lmsys.org/blog/ Large Model Systems (LMSYS Corp.) is a 501(c)(3) non-profit focused on incubating open-source projects and research. Our mission is to make large AI models accessible to everyone by co-developing open models, datasets, systems, and evaluation tools. We conduct cutting-edge machine learning research, develop open-source software, train large language models for broad accessibility, and build distributed systems to optimize their training and inference.
KaiserPro 1y ago
Same explanation but with less mysticism: Inference is (mostly) stateless. So unlike training where you need to have memory coherence over something like 100k machines and somehow avoid the certainty of machine failure, you just need to route mostly small amounts of data to a bunch of big machines. I don't know what the specs of their inference machines are, but where I worked the machines research used were all 8gpu monsters. so long as your model fitted in (combined) vram, you could job was a goodun. To scale the secret ingredient was industrial amounts of cash. Sure we had DGXs (fun fact, nvidia sent literal gold plated DGX machines) but they wernt dense, and were very expensive. Most large companies have robust RPC, and orchestration, which means the hard part isn't routing the message, its making the model fit in the boxes you have. (thats not my area of expertise though)
- zozbot234 1y ago
  > Inference is (mostly) stateless. ... you just need to route mostly small amounts of data to a bunch of big machines. I think this might just be the key insight. The key advantage of doing batched inference at a huge scale is that once you maximize parallelism and sharding, your model parameters and the memory bandwidth associated with them are essentially free (since at any given moment they're being shared among a huge amount of requests!), you "only" pay for the request-specific raw compute and the memory storage+bandwidth for the activations. And the proprietary models are now huge, highly-quantized extreme-MoE models where the former factor (model size) is huge and the latter (request-specific compute) has been correspondingly minimized - and where it hasn't, you're definitely paying "pro" pricing for it. I think this goes a long way towards explaining how inference at scale can work better than locally. (There are "tricks" you could do locally to try and compete with this setup, such as storing model parameters on disk and accessing them via mmap, at least when doing token gen on CPU. But of course you're paying for that with increased latency, which you may or may not be okay with in that context.)
  - patrick451 1y ago
    > The key advantage of doing batched inference at a huge scale is that once you maximize parallelism and sharding, your model parameters and the memory bandwidth associated with them are essentially free (since at any given moment they're being shared among a huge amount of requests!) Kind of unrelated, but this comment made me wonder when we will start seeing side channel attacks that force queries to leak into each other.
blibble 1y ago
> So I simultaneously can tell you that its smart people really thinking about every facet of the problem, and I can't tell you much more than that. "we do 1970s mainframe style timesharing" there, that was easy
- kstrauser 1y ago
  For real. Say it takes 1 machine 5 seconds to reply, and that a machine can only possibly form 1 reply at a time (which I doubt, but for argument). If the requests were regularly spaced, and they certainly won’t be, but for the sake of argument, then 1 machine could serve 17,000 requests per day, or 120,000 per week. At that rate, you’d need about 5,600 machines to serve 700M requests. That’s a lot to me, but not to someone who owns a data center. Yes, those 700M users will issue more than 1 query per week and they won’t be evenly spaced. However, I’d bet most of those queries will take well under 1 second to answer, and I’d also bet each machine can handle more than one at a time. It’s a large problem, to be sure, but that seems tractable.
  - mh- 1y ago
    Yes. And batched inference is a thing, where intelligent grouping/bin packing and routing of requests happens. I expect a good amount of "secret sauce" is at this layer. Here's an entry-level link I found quickly on Google, OP: https://medium.com/@wearegap/a-brief-introduction-to-optimized-batched-inference-with-vllm-deddf5423d0c https://medium.com/@wearegap/a-brief-introduction-to-optimiz...
- rootsudo 1y ago
  Makes perfect sense, completely understand now!
- brookst 1y ago
  But that’s not accurate. There are all sorts of tricks around KV cache where different users will have the same first X bytes because they share system prompts, caching entire inputs / outputs when the context and user data is identical, and more. Not sure if you were just joking or really believe that, but for other peoples’ sake, it’s wildly wrong.
catigula 1y ago
A lot of really smart people working on problems that don't even really need to be solved is an interesting aspect of market allocation.
- vermilingua 1y ago
  Well, we all thought advertising was the worst thing to come out of the tech industry, someone had to prove us wrong!
  - hirvi74 1y ago
    Just wait until the two combine.
- YossarianFrPrez 1y ago
  Can you explain what you mean about 'not needing to be solved'? There are versions of that kind of critique that would seem, at least on the surface, to better apply to finance or flash trading. I ask because scaling an system that a substantially chunk of the population finds incredibly useful, including for the more efficient production of public goods (scientific research, for example) does seem like a problem that a) needs to be solved from a business point of view, and b) should be solved from a civic-minded point of view.
  - catigula 1y ago
    [flagged]
    - hattmall 1y ago
      The only solution to those people starving to death is to kill the people that benefit from them starving to death. It's a solved problem, the solution isn't palatable. No one is starving to death because of a lack of engineering prowess.
      - catigula 1y ago
LAC-Tech 1y ago
If people at google are so smart why can't google.com get a 100% lighthouse score?
- usr1106 1y ago
  Because most smart people are not generalists. My first boss was really smart and managed to found a university institute in computer science. The 3 other professors he hired were, ahem, strange choices. We 28 year old assistents could only shake our heads. After fighting a couple of years with his own hires the founder left in frustration to found another institution. One of my colleagues was only 25, really smart in his field and became a professor less than 10 years later. But he was incredibly naive in everyday chores. Buying groceries or filing taxes resulted in major screw-ups regularly
  - jeltz 1y ago
    I have met those supersmart specialists but in my experience there are also a lot of smart people who are more generalists. The real answer is likely internal company politics and priorities. Google certainly has people with the technical skills to solve it but do they care and if they care can they allocate those skilled people to the task?
    - gregorygoc 1y ago
      My observation is that in general smart generalists are smarter than smart specialists. I work at Google, and it’s just that these generalists folks are extremely fast learners. They can cover breadth and depth of an arbitrary topic in a matter of 15 minutes, just enough to solve a problem at hand. It’s quite intimidating how fast they can break down difficult concepts into first principles. I’ve witnessed this first hand and it’s beyond intimidating. Makes you wondering what you’re doing at this company… That being said, the caliber of folks I’m talking about is quite rare, like top 10% of top 1% teams at Google.
      - jeltz 1y ago
benreesman 1y ago
I don't think it's either useful or particularly accurate to characterize modern disagg racks of inference gear, well-understood RDMA and other low-overhead networking techniques, aggressive MLA and related cache optimizations that are in the literature, and all the other stuff that goes into a system like this as being some kind of mystical thing attended to by a priesthood of people from a different tier of hacker. This stuff is well understood in public, and where a big name has something highly custom going on? Often as not it's a liability around attachment to some legacy thing. You run this stuff at scale by having the correct institutions and processes in place that it takes to run any big non-trivial system: that's everything from procurement and SRE training to the RTL on the new TPU, and all of the stuff is interesting, but if anyone was 10x out in front of everyone else? You'd be able to tell. Signed, Someone Who Also Did Megascale Inference for a TOP-5 For a Decade.
hnpolicestate 1y ago
This caught my attention "But today even “small” models run so close to hardware limits". Sounds analogous to the 60's and 70's i.e "even small programs run so close to hardware limits". If optimization and efficiency is dead in software engineering, it's certainly alive and well in LLM development.
revskill 1y ago
No, you just over complicate things.
nwhnwh 1y ago
Nothing smart about making something that is not useful for humans.

tough 1y ago

Doesn't google have TPU's that makes inference of their own models much more profitable than say having to rent out NVDIA cards? Doesn't OpenAI depend mostly on its relationship/partnership with Microsoft to get GPUs to inference on? Thanks for the links, interesting book!

ActorNightly 1y ago
Yes. Google is probably gonna win the LLM game tbh. They had a massive head start with TPUs which are very energy efficient compared to Nvidia Cards.
- fakedang 1y ago
  Yeah honestly. They could just try selling solutions and SLAs combining their TPU hardware with on-prem SOTA models and practically dominate enterprise. From what I understand, that's GCP's gameplay too for most regulated enterprise clients.
  - ActorNightly 1y ago
    Googles bread and butter is advertising, so they have a huge interest in keeping things in house. Data is more valuable to them than money from hardware sales. Even then, I think that their primary use case is going to be consumer grade good AI on phones. I dunno why Gemma QAT model fly so low on the radar, but you can basically get full scale Llamma 3 like performance from a single 3090 now, at home.
    - fakedang 1y ago
      https://www.cnbc.com/2025/04/09/google-will-let-companies-run-gemini-models-in-their-own-data-centers.html https://www.cnbc.com/2025/04/09/google-will-let-companies-ru... Google has already started the process of letting companies self-host Gemini, even on NVidia Blackwell GPUs. Although imho, they really should bundle it with their TPUs as a turnkey solution for those clients who haven't invested in large scale infra like DCs yet.
      - ActorNightly 1y ago
        Its the same format as other software - you release the actual software for free but offer managed services that work with that software way better and easier.
        fakedang 1y ago
        Yeah but those are on Google's managed cloud, and not onprem. But that recent announcement has been specifically for Google Distributed Cloud, which is huge. My point was a bit more specific though. To elaborate, I know of a number of publicly traded companies (USD $200M+ market cap) globally which have identified use cases for onprem AI and want to implement them actively but cannot, because they lack the knowhow to work with onprem, and hiring talent to implement that is just extremely expensive. Google should simply provide it as a turnkey bundle and milk them for it.
      - to11mtm 1y ago
        My guess is that either google want's a high level of physical control over their TPUs, or they have one sort of deal or another with NVidia and don't want to step on their toes. And also, Google's track record with hardware.
- baxtr 1y ago
  The only one who can stop Google is Google. They’ll definitely have the best model, but there is a chance they will f*up the product / integration into their products.
  - scarface_74 1y ago
    It would take talent for them to mess up hosting businesses who want to use their TPUs on GCP. But then again even there, their reputation for abandoning products, lack of customer service, condescension when it came to large enterprises’ “legacy tech” lets Microsoft who is king of hand holding big enterprise and even AWS run rough shod over them. When I was at AWS ProServe, we didn’t even bother coming up with talking points when competing with GCP except to point out how they abandon services. Was it partially FUD? Probably. But it worked.
    - deleted 1y ago
      [deleted]
    - JoshuaDavid 1y ago
      Google employees collectively have a lot of talent.
      - bee_rider 1y ago
        A truly astonishing amount of talent applied to… hosting emails very well, and losing the search battle against SEO spammers.
        big_hacker 1y ago
        Well, Search had no chance when the sites also make money from Google ads. Google fucked their Search by creating themselves incentives for bounce rate.
    - serf 1y ago
      >It would take talent for them to mess up hosting businesses who want to use their TPUs on GCP. there are few groups as talented at losing a head start as google.
- davedx 1y ago
  But they’re ASICs so any big architecture changes will be painful for them right?
  - deleted 1y ago
    [deleted]
  - edoceo 1y ago
    I'd think no. They have the hardware and software experience, likely have next and next-next plans in place already. The big hurdle is money, which G has a bunch of.
  - llm_nerd 1y ago
    TPUs are accelerators that accelerate the common operations found in neural nets. A big part is simply a massive number of matrix FMA units to process enormous matrix operations, which comprises the bulk of doing a forward pass through a model. Caching enhancements and massively growing memory was necessary to facilitate transformers, but on the hardware side not a huge amount has changed and the fundamentals from years ago still powers the latest models. The hardware is just getting faster and with more memory and more parallel processing units. And later getting more data types to enable hardware-enabled quantization. So it isn't like Google designed a TPU for a specific model or architecture. They're pretty general purpose in a narrow field (oxymoron, but you get the point). The set of operations Google designed into a TPU is very similar to what nvidia did, and it's about as broadly capable. But Google owns the IP and doesn't pay the premium and gets to design for their own specific needs.
    - saagarjha 1y ago
      There are plenty of matrix multiplies in the backward pass too. Obviously this is less useful when serving but it's useful for training.
- stogot 1y ago
  Hasn’t the Inferentia chip been around long enough to make the same argument? AWS and Google probably have the same order of magnitude of their own custom chips
  - saagarjha 1y ago
    Inferentia has a generally worse stack but yes
- benreesman 1y ago
  Google will win the LLM game if the LLM game is about compute, which is the common wisdom and maybe true, but not foreordained by God. There's an argument that if compute was the dominant term that Google would never have been anything but leading by a lot. Personally right now I see one clear leader and one group going 0-99 like a five sigma cosmic ray: Anthropic and the PRC. But this is because I believe/know that all the benchmarks are gamed as hell, its like asking if a movie star had cosmetic surgery. On quality, Opus 4 is 15x the cost and sold out / backordered. Qwen 3 is arguably in next place. In both of those cases, extreme quality expert labeling at scale (assisted by the tool) seems to be the secret sauce. Which is how it would play out if history is any guide: when compute as a scaling lever starts to flatten, you expert label like its 1987 and claim its compute and algorithms until the government wises up and stops treating your success persobally as a national security priority. It's the easiest trillion Xi Xianping ever made: pretending to think LLMs are AGI too, fast following for pennies on the dollar, and propping up a stock market bubble to go with the fentanyl crisis? 9-D chess. It's what I would do about AI if I were China. Time will tell.
  - 0_____0 1y ago
    I believe Google might win the LLM game simply because they have the infrastructure to make it profitable - via ads. All the LLM vendors are going to have to cope with the fact that they're lighting money on fire, and Google have the paying customers (advertisers) and with the user-specific context they get from their LLM products, one of the juciest and most targetable ad audiences of all time.
    - throwaway2037 1y ago
      This is one of the best insights after reading 100+ posts here. You are talking about existing demand and existing relationships from their advert business. These customers are happy with Google results. The Google marketing team will be carefully defining new advert products that employ LLMs. I would only offer one disagreement with your post: There will not be a single winner in LLMs. The landscape is so large that we will have multiple winners in different areas. Example: Google might fail in B2C LLM (chatbot that answers your questions), but will certainly be (wildly?) successful in B2B for adverts.
canyon289 1y ago
Im a research person building models so I can't answer your questions well (save for one part) That is, as a research person using our GPUs and TPUs I see first hand how choices from the high level python level, through Jax, down to the TPU architecture all work together to make training and inference efficient. You can see a bit of that in the gif on the front page of the book. https://jax-ml.github.io/scaling-book/ https://jax-ml.github.io/scaling-book/ I also see how sometimes bad choices by me can make things inefficient. Luckily for me if my code/models are running slow I can ping colleagues who are able to debug at both a depth and speed that is quite incredible. And because were on HN I want to preemptively call out my positive bias for Google! It's a privilege to be able to see all this technology first hand, work with great people, and do my best to ship this at scale across the globe.

jackhalford 1y ago

Why does the unsloth guide for gemma 3n say: > llama.cpp an other inference engines auto add a <bos> - DO NOT add TWO <bos> tokens! You should ignore the <bos> when prompting the model! That makes the want to try exactly that? Weird

ignoramous 1y ago

> Another great resource to look at is the unsloth guides. And folks at LMSys: https://lmsys.org/blog/ https://lmsys.org/blog/ Large Model Systems (LMSYS Corp.) is a 501(c)(3) non-profit focused on incubating open-source projects and research. Our mission is to make large AI models accessible to everyone by co-developing open models, datasets, systems, and evaluation tools. We conduct cutting-edge machine learning research, develop open-source software, train large language models for broad accessibility, and build distributed systems to optimize their training and inference.

KaiserPro 1y ago

Same explanation but with less mysticism: Inference is (mostly) stateless. So unlike training where you need to have memory coherence over something like 100k machines and somehow avoid the certainty of machine failure, you just need to route mostly small amounts of data to a bunch of big machines. I don't know what the specs of their inference machines are, but where I worked the machines research used were all 8gpu monsters. so long as your model fitted in (combined) vram, you could job was a goodun. To scale the secret ingredient was industrial amounts of cash. Sure we had DGXs (fun fact, nvidia sent literal gold plated DGX machines) but they wernt dense, and were very expensive. Most large companies have robust RPC, and orchestration, which means the hard part isn't routing the message, its making the model fit in the boxes you have. (thats not my area of expertise though)

zozbot234 1y ago
> Inference is (mostly) stateless. ... you just need to route mostly small amounts of data to a bunch of big machines. I think this might just be the key insight. The key advantage of doing batched inference at a huge scale is that once you maximize parallelism and sharding, your model parameters and the memory bandwidth associated with them are essentially free (since at any given moment they're being shared among a huge amount of requests!), you "only" pay for the request-specific raw compute and the memory storage+bandwidth for the activations. And the proprietary models are now huge, highly-quantized extreme-MoE models where the former factor (model size) is huge and the latter (request-specific compute) has been correspondingly minimized - and where it hasn't, you're definitely paying "pro" pricing for it. I think this goes a long way towards explaining how inference at scale can work better than locally. (There are "tricks" you could do locally to try and compete with this setup, such as storing model parameters on disk and accessing them via mmap, at least when doing token gen on CPU. But of course you're paying for that with increased latency, which you may or may not be okay with in that context.)
- patrick451 1y ago
  > The key advantage of doing batched inference at a huge scale is that once you maximize parallelism and sharding, your model parameters and the memory bandwidth associated with them are essentially free (since at any given moment they're being shared among a huge amount of requests!) Kind of unrelated, but this comment made me wonder when we will start seeing side channel attacks that force queries to leak into each other.

blibble 1y ago

> So I simultaneously can tell you that its smart people really thinking about every facet of the problem, and I can't tell you much more than that. "we do 1970s mainframe style timesharing" there, that was easy

kstrauser 1y ago
For real. Say it takes 1 machine 5 seconds to reply, and that a machine can only possibly form 1 reply at a time (which I doubt, but for argument). If the requests were regularly spaced, and they certainly won’t be, but for the sake of argument, then 1 machine could serve 17,000 requests per day, or 120,000 per week. At that rate, you’d need about 5,600 machines to serve 700M requests. That’s a lot to me, but not to someone who owns a data center. Yes, those 700M users will issue more than 1 query per week and they won’t be evenly spaced. However, I’d bet most of those queries will take well under 1 second to answer, and I’d also bet each machine can handle more than one at a time. It’s a large problem, to be sure, but that seems tractable.
- mh- 1y ago
  Yes. And batched inference is a thing, where intelligent grouping/bin packing and routing of requests happens. I expect a good amount of "secret sauce" is at this layer. Here's an entry-level link I found quickly on Google, OP: https://medium.com/@wearegap/a-brief-introduction-to-optimized-batched-inference-with-vllm-deddf5423d0c https://medium.com/@wearegap/a-brief-introduction-to-optimiz...
rootsudo 1y ago
Makes perfect sense, completely understand now!
brookst 1y ago
But that’s not accurate. There are all sorts of tricks around KV cache where different users will have the same first X bytes because they share system prompts, caching entire inputs / outputs when the context and user data is identical, and more. Not sure if you were just joking or really believe that, but for other peoples’ sake, it’s wildly wrong.

catigula 1y ago

A lot of really smart people working on problems that don't even really need to be solved is an interesting aspect of market allocation.

vermilingua 1y ago
Well, we all thought advertising was the worst thing to come out of the tech industry, someone had to prove us wrong!
- hirvi74 1y ago
  Just wait until the two combine.
YossarianFrPrez 1y ago
Can you explain what you mean about 'not needing to be solved'? There are versions of that kind of critique that would seem, at least on the surface, to better apply to finance or flash trading. I ask because scaling an system that a substantially chunk of the population finds incredibly useful, including for the more efficient production of public goods (scientific research, for example) does seem like a problem that a) needs to be solved from a business point of view, and b) should be solved from a civic-minded point of view.
- catigula 1y ago
  [flagged]
  - hattmall 1y ago
    The only solution to those people starving to death is to kill the people that benefit from them starving to death. It's a solved problem, the solution isn't palatable. No one is starving to death because of a lack of engineering prowess.
    - catigula 1y ago

LAC-Tech 1y ago

If people at google are so smart why can't google.com get a 100% lighthouse score?

usr1106 1y ago
Because most smart people are not generalists. My first boss was really smart and managed to found a university institute in computer science. The 3 other professors he hired were, ahem, strange choices. We 28 year old assistents could only shake our heads. After fighting a couple of years with his own hires the founder left in frustration to found another institution. One of my colleagues was only 25, really smart in his field and became a professor less than 10 years later. But he was incredibly naive in everyday chores. Buying groceries or filing taxes resulted in major screw-ups regularly
- jeltz 1y ago
  I have met those supersmart specialists but in my experience there are also a lot of smart people who are more generalists. The real answer is likely internal company politics and priorities. Google certainly has people with the technical skills to solve it but do they care and if they care can they allocate those skilled people to the task?
  - gregorygoc 1y ago
    My observation is that in general smart generalists are smarter than smart specialists. I work at Google, and it’s just that these generalists folks are extremely fast learners. They can cover breadth and depth of an arbitrary topic in a matter of 15 minutes, just enough to solve a problem at hand. It’s quite intimidating how fast they can break down difficult concepts into first principles. I’ve witnessed this first hand and it’s beyond intimidating. Makes you wondering what you’re doing at this company… That being said, the caliber of folks I’m talking about is quite rare, like top 10% of top 1% teams at Google.
    - jeltz 1y ago

benreesman 1y ago

I don't think it's either useful or particularly accurate to characterize modern disagg racks of inference gear, well-understood RDMA and other low-overhead networking techniques, aggressive MLA and related cache optimizations that are in the literature, and all the other stuff that goes into a system like this as being some kind of mystical thing attended to by a priesthood of people from a different tier of hacker. This stuff is well understood in public, and where a big name has something highly custom going on? Often as not it's a liability around attachment to some legacy thing. You run this stuff at scale by having the correct institutions and processes in place that it takes to run any big non-trivial system: that's everything from procurement and SRE training to the RTL on the new TPU, and all of the stuff is interesting, but if anyone was 10x out in front of everyone else? You'd be able to tell. Signed, Someone Who Also Did Megascale Inference for a TOP-5 For a Decade.

hnpolicestate 1y ago

This caught my attention "But today even “small” models run so close to hardware limits". Sounds analogous to the 60's and 70's i.e "even small programs run so close to hardware limits". If optimization and efficiency is dead in software engineering, it's certainly alive and well in LLM development.

revskill 1y ago

No, you just over complicate things.

nwhnwh 1y ago

Nothing smart about making something that is not useful for humans.