3 ms·

> PyTorch was meant to be easy to use first, and fast second. People have been saying this for years, yet most training and a majority(?) of inference is done

> PyTorch was meant to be easy to use first, and fast second. People have been saying this for years, yet most training and a majority(?) of inference is done in PyTorch. We have all sorts of cool ML compilers, some that directly support importing PyTorch projects, largely sitting there gathering dust. Even torch.compile is catching on like molasses in winter.

TeMPOraL 3y ago
> People have been saying this for years, yet most training and a majority(?) of inference is done in PyTorch. The usual symptom of "cheaper to waste more resources than do it properly"?
- brucethemoose2 3y ago
  That, but I think many users don't even know its an option to try. I am hoping torch 2.1 improves this situation. torch.compile should "just work" much more often with `dynamic=true`, and its widespread usage might make people wonder "what other cost-reducing compilation frameworks are out there?"
eep_social 3y ago
I think the changed cost of money will improve this over time. It’s possible interest rates will swing back (and there is still a lot of denial about how far back) but IMO the days of throwing money at things and punting on efficiency appear to be waning.
lumost 3y ago
Most "production" ML models of the last 10 years were only a few millions of parameters owing to limited training data and simplistic tasks. While BERT and similar models raised the parameter count substantially, my anecdotal experience observing hundreds of ML projects was that BERT based models provided only a marginal (to the business) gain in performance over smaller models. We are now seeing mass consumer and enterprise adoption of models in the hundreds of billions of parameters (GPT-4). These models use relatively standard, building blocks - and are even chained together in a relatively standard manner. If everyone's laptop was running a 7B parameter LLM, or actively using a cloud based derivative - then LLM inference would have a chance at being the dominant compute workload in the industry. This will definitely change the way that inference is done, even if training remains the same.

TeMPOraL 3y ago

> People have been saying this for years, yet most training and a majority(?) of inference is done in PyTorch. The usual symptom of "cheaper to waste more resources than do it properly"?

brucethemoose2 3y ago
That, but I think many users don't even know its an option to try. I am hoping torch 2.1 improves this situation. torch.compile should "just work" much more often with `dynamic=true`, and its widespread usage might make people wonder "what other cost-reducing compilation frameworks are out there?"

eep_social 3y ago

I think the changed cost of money will improve this over time. It’s possible interest rates will swing back (and there is still a lot of denial about how far back) but IMO the days of throwing money at things and punting on efficiency appear to be waning.

lumost 3y ago

Most "production" ML models of the last 10 years were only a few millions of parameters owing to limited training data and simplistic tasks. While BERT and similar models raised the parameter count substantially, my anecdotal experience observing hundreds of ML projects was that BERT based models provided only a marginal (to the business) gain in performance over smaller models. We are now seeing mass consumer and enterprise adoption of models in the hundreds of billions of parameters (GPT-4). These models use relatively standard, building blocks - and are even chained together in a relatively standard manner. If everyone's laptop was running a 7B parameter LLM, or actively using a cloud based derivative - then LLM inference would have a chance at being the dominant compute workload in the industry. This will definitely change the way that inference is done, even if training remains the same.