3 ms·

I think folks are underestimating how far we have to go on inference performance. Even ignoring LLM’s like phi-1.5. PyTorch was meant to be easy to use first, a

by lumost 3y ago

I think folks are underestimating how far we have to go on inference performance. Even ignoring LLM’s like phi-1.5. PyTorch was meant to be easy to use first, and fast second. This turned out to be the winning combo for adhoc science workloads. For pure LLM inference, you can go much further by fusing operations together and eventually building asic extensions to improve transformer perf. I’d easily believe that we will see a 10x reduction in inference costs over the next 5 years as hardware folks catch up.

brucethemoose2 3y ago
> PyTorch was meant to be easy to use first, and fast second. People have been saying this for years, yet most training and a majority(?) of inference is done in PyTorch. We have all sorts of cool ML compilers, some that directly support importing PyTorch projects, largely sitting there gathering dust. Even torch.compile is catching on like molasses in winter.
- TeMPOraL 3y ago
  > People have been saying this for years, yet most training and a majority(?) of inference is done in PyTorch. The usual symptom of "cheaper to waste more resources than do it properly"?
  - brucethemoose2 3y ago
    That, but I think many users don't even know its an option to try. I am hoping torch 2.1 improves this situation. torch.compile should "just work" much more often with `dynamic=true`, and its widespread usage might make people wonder "what other cost-reducing compilation frameworks are out there?"
- eep_social 3y ago
  I think the changed cost of money will improve this over time. It’s possible interest rates will swing back (and there is still a lot of denial about how far back) but IMO the days of throwing money at things and punting on efficiency appear to be waning.
- lumost 3y ago
  Most "production" ML models of the last 10 years were only a few millions of parameters owing to limited training data and simplistic tasks. While BERT and similar models raised the parameter count substantially, my anecdotal experience observing hundreds of ML projects was that BERT based models provided only a marginal (to the business) gain in performance over smaller models. We are now seeing mass consumer and enterprise adoption of models in the hundreds of billions of parameters (GPT-4). These models use relatively standard, building blocks - and are even chained together in a relatively standard manner. If everyone's laptop was running a 7B parameter LLM, or actively using a cloud based derivative - then LLM inference would have a chance at being the dominant compute workload in the industry. This will definitely change the way that inference is done, even if training remains the same.
moffkalast 3y ago
Well the best option is what Google's doing with its TPUs, a different architecture with far better data locality since that's the main bottleneck. https://en.wikipedia.org/wiki/Systolic_array https://en.wikipedia.org/wiki/Systolic_array
- brucethemoose2 3y ago
  I was really impressed with Microsoft's paper on llm architectures, where instead of using giant chips with giant memory busses, they pipe together a bunch of cheap, monolithic, SRAM heavy SoCs. This would work really well for something like Llama, where the layers can be easily split up into SRAM sized chunks, and the whole thing can be pipelined as long as many users are making requests.
  - marshray 3y ago
    > I was really impressed with Microsoft's paper on llm architectures Link? Please :-)
    - brucethemoose2 3y ago
      https://www.nextplatform.com/2023/07/12/microsofts-chiplet-cloud-to-bring-the-cost-of-llms-way-down/ https://www.nextplatform.com/2023/07/12/microsofts-chiplet-c... https://arxiv.org/pdf/2307.02666.pdf https://arxiv.org/pdf/2307.02666.pdf
Mockapapella 3y ago
Oh man, an ASIC designed specifically for a single model's architecture would neat to see
- mighmi 3y ago
  That's how we get locked in to that model e.g. how processors became optimized for LLVM/the C ecosystem (whereas in the past there were Lisp optimized processors, general processors etc.)

by lumost 3y ago

brucethemoose2 3y ago

> PyTorch was meant to be easy to use first, and fast second. People have been saying this for years, yet most training and a majority(?) of inference is done in PyTorch. We have all sorts of cool ML compilers, some that directly support importing PyTorch projects, largely sitting there gathering dust. Even torch.compile is catching on like molasses in winter.

TeMPOraL 3y ago
> People have been saying this for years, yet most training and a majority(?) of inference is done in PyTorch. The usual symptom of "cheaper to waste more resources than do it properly"?
- brucethemoose2 3y ago
  That, but I think many users don't even know its an option to try. I am hoping torch 2.1 improves this situation. torch.compile should "just work" much more often with `dynamic=true`, and its widespread usage might make people wonder "what other cost-reducing compilation frameworks are out there?"
eep_social 3y ago
I think the changed cost of money will improve this over time. It’s possible interest rates will swing back (and there is still a lot of denial about how far back) but IMO the days of throwing money at things and punting on efficiency appear to be waning.
lumost 3y ago
Most "production" ML models of the last 10 years were only a few millions of parameters owing to limited training data and simplistic tasks. While BERT and similar models raised the parameter count substantially, my anecdotal experience observing hundreds of ML projects was that BERT based models provided only a marginal (to the business) gain in performance over smaller models. We are now seeing mass consumer and enterprise adoption of models in the hundreds of billions of parameters (GPT-4). These models use relatively standard, building blocks - and are even chained together in a relatively standard manner. If everyone's laptop was running a 7B parameter LLM, or actively using a cloud based derivative - then LLM inference would have a chance at being the dominant compute workload in the industry. This will definitely change the way that inference is done, even if training remains the same.

moffkalast 3y ago

Well the best option is what Google's doing with its TPUs, a different architecture with far better data locality since that's the main bottleneck. https://en.wikipedia.org/wiki/Systolic_array https://en.wikipedia.org/wiki/Systolic_array

brucethemoose2 3y ago
I was really impressed with Microsoft's paper on llm architectures, where instead of using giant chips with giant memory busses, they pipe together a bunch of cheap, monolithic, SRAM heavy SoCs. This would work really well for something like Llama, where the layers can be easily split up into SRAM sized chunks, and the whole thing can be pipelined as long as many users are making requests.
- marshray 3y ago
  > I was really impressed with Microsoft's paper on llm architectures Link? Please :-)
  - brucethemoose2 3y ago
    https://www.nextplatform.com/2023/07/12/microsofts-chiplet-cloud-to-bring-the-cost-of-llms-way-down/ https://www.nextplatform.com/2023/07/12/microsofts-chiplet-c... https://arxiv.org/pdf/2307.02666.pdf https://arxiv.org/pdf/2307.02666.pdf

Mockapapella 3y ago

Oh man, an ASIC designed specifically for a single model's architecture would neat to see

mighmi 3y ago
That's how we get locked in to that model e.g. how processors became optimized for LLVM/the C ecosystem (whereas in the past there were Lisp optimized processors, general processors etc.)