4 ms·

I wonder when we as an industry will start to address the scaling issues in LLMs.It is obviously in Nvidias interest to keep pushing out bigger and better GPUs,

I wonder when we as an industry will start to address the scaling issues in LLMs.It is obviously in Nvidias interest to keep pushing out bigger and better GPUs, but what is the collective interest? It is already proven that good language models are possible given enough resources. The challenge now is to put these models in a solution which do not require unfathomable amounts of resources for the average use cases.

bayindirh 3y ago
Wasteful software development is easy and keeps momentum for development. As long as growth is king, quick and dirty will always beat well optimized and smaller systems. This is not a problem with AI only, but with every software we use. Only two groups try to optimize things and try to fit into smaller systems. Passionate programmers and people who is paid to do this (e.g.: phone manufacturers' software teams, etc.).
- Difwif 3y ago
  Not sure it's fair to characterize modern LLMs as 'wasteful software development" or unoptimized. The implementations do quite an impressive level of optimization with what hardware is available. New theoretical methods w.r.t quantization represent most of our software optimization techniques and we're probably hitting the limit of that shortly with ternary or binary gates. To your point enthusiasts and developers with strong financial motivation have fairly optimized code. AI definitely falls into the later group. These are not like your typical web app :)
  - bayindirh 3y ago
    Currently AI folks create bigger models and feed larger training data sets, because we still don't know the efficiency limits. IOW, currently we can't accelerate the learning beyond a certain point with less training data. This is esp. true on LLM/GenAI space. On the other areas where NNs are used, what I can see is training models with less data is not only plausible, but very possible, esp. in image processing, probably an image carries more information than a single sentence. I think AI falls on the spectrum with a slight bias to the latter group. Because if you can shrink a model 10% and lose a month, you'd rather have a 10% bigger model now, and reap the money^H^H^H^H^H fame. I don't know anything about a typical web app, because I'm not a typical developer developing web apps. :)
    - Difwif 3y ago
      I just think we're talking about a completely different problem with AI optimization. There's $billions of effort and research that goes into AI optimization. Scaling model size and training set size happens because it's what the research and evidence tells us will improve model performance reliably. If we could reduce any of it for the same performance we would. Top model performance is an arms race and it's happening at every expense. The largest players are all shooting to beat GPT-4 or Claude Opus and achieve AGI (whatever that means). This is very different than a program that requires zero research breakthroughs to dramatically improve and is simply slow and bloated because people have different priorities.
      - bayindirh 3y ago
        > If we could reduce any of it for the same performance we would. Nope, because all of them are harder than just going bigger. > Top model performance is an arms race and it's happening at every expense. This is also what I said. "Growth (in model performance) is king, so quick and dirty (going bigger) beating harder optimization efforts". > This is very different than a program that... [Snipped for brevity] Again this is what I said by "it keeps development momentum". Yes people have different priorities. Mostly money and fame in this point. So, we don't disagree a bit.
  - __loam 3y ago
    LLMs are efficient for programs in their class but they are substantially more power intensive than less generalized software.
- gammalost 3y ago
  I don't think it is thanks to wasteful software development. The libraries used for LLMs do a lot to squeeze out the full potential of GPUs. I think it is more of an information problem. How can we store enough information in weights so that it is possible to train models without a budget similar to OpenAI
  - bayindirh 3y ago
    Just because you're squeezing out every bit of performance from a processor doesn't mean all the work you're doing is meaningful or can't be optimized. I work on material simulations. I make processors hit their TDPs, saturate their pipelines and make them go as fast as they can. However, sometimes we come up with a formula optimization which does things 1-2% faster, which means we can save hours on a bigger computation. Utilization doesn't change, but speed does. > I think it is more of an information problem. It's an interesting point of view, and partially true. However, we're still wasting "space" by just adding bits to the network to make it contain more data. There's a long way to go. "Wasteful development" is a phase and always be part of software development. The important part is not forgetting that optimization exists. Otherwise we can't sustain ourselves much with all that energy use.

bayindirh 3y ago

Wasteful software development is easy and keeps momentum for development. As long as growth is king, quick and dirty will always beat well optimized and smaller systems. This is not a problem with AI only, but with every software we use. Only two groups try to optimize things and try to fit into smaller systems. Passionate programmers and people who is paid to do this (e.g.: phone manufacturers' software teams, etc.).

Difwif 3y ago
Not sure it's fair to characterize modern LLMs as 'wasteful software development" or unoptimized. The implementations do quite an impressive level of optimization with what hardware is available. New theoretical methods w.r.t quantization represent most of our software optimization techniques and we're probably hitting the limit of that shortly with ternary or binary gates. To your point enthusiasts and developers with strong financial motivation have fairly optimized code. AI definitely falls into the later group. These are not like your typical web app :)
- bayindirh 3y ago
  Currently AI folks create bigger models and feed larger training data sets, because we still don't know the efficiency limits. IOW, currently we can't accelerate the learning beyond a certain point with less training data. This is esp. true on LLM/GenAI space. On the other areas where NNs are used, what I can see is training models with less data is not only plausible, but very possible, esp. in image processing, probably an image carries more information than a single sentence. I think AI falls on the spectrum with a slight bias to the latter group. Because if you can shrink a model 10% and lose a month, you'd rather have a 10% bigger model now, and reap the money^H^H^H^H^H fame. I don't know anything about a typical web app, because I'm not a typical developer developing web apps. :)
  - Difwif 3y ago
    I just think we're talking about a completely different problem with AI optimization. There's $billions of effort and research that goes into AI optimization. Scaling model size and training set size happens because it's what the research and evidence tells us will improve model performance reliably. If we could reduce any of it for the same performance we would. Top model performance is an arms race and it's happening at every expense. The largest players are all shooting to beat GPT-4 or Claude Opus and achieve AGI (whatever that means). This is very different than a program that requires zero research breakthroughs to dramatically improve and is simply slow and bloated because people have different priorities.
    - bayindirh 3y ago
      > If we could reduce any of it for the same performance we would. Nope, because all of them are harder than just going bigger. > Top model performance is an arms race and it's happening at every expense. This is also what I said. "Growth (in model performance) is king, so quick and dirty (going bigger) beating harder optimization efforts". > This is very different than a program that... [Snipped for brevity] Again this is what I said by "it keeps development momentum". Yes people have different priorities. Mostly money and fame in this point. So, we don't disagree a bit.
- __loam 3y ago
  LLMs are efficient for programs in their class but they are substantially more power intensive than less generalized software.
gammalost 3y ago
I don't think it is thanks to wasteful software development. The libraries used for LLMs do a lot to squeeze out the full potential of GPUs. I think it is more of an information problem. How can we store enough information in weights so that it is possible to train models without a budget similar to OpenAI
- bayindirh 3y ago
  Just because you're squeezing out every bit of performance from a processor doesn't mean all the work you're doing is meaningful or can't be optimized. I work on material simulations. I make processors hit their TDPs, saturate their pipelines and make them go as fast as they can. However, sometimes we come up with a formula optimization which does things 1-2% faster, which means we can save hours on a bigger computation. Utilization doesn't change, but speed does. > I think it is more of an information problem. It's an interesting point of view, and partially true. However, we're still wasting "space" by just adding bits to the network to make it contain more data. There's a long way to go. "Wasteful development" is a phase and always be part of software development. The important part is not forgetting that optimization exists. Otherwise we can't sustain ourselves much with all that energy use.