7 ms·

70 billion parameters... Is each of those a 4-byte float? So, is that 280 billion bytes of just parameters?

by VikingCoder 4y ago

sudosysgen 4y ago
I'm fairly confident each of those is a 2-byte float, but yes that's over 100 GB of parameters.
- sillysaurusx 4y ago
  Welcome to the party! I joined ML because I realized I could help. You can too. I bet you’re both already thinking of clever ways to deal with massive models from an infrastructure standpoint. That’s just one of hundreds of interesting problems.
  - native_samples 4y ago
    Is 100GB of parameters really that large? 128GB of RAM on a server class machine is not unusual. Seems such a model could fit entirely in RAM.
    - andbberger 4y ago
      GPU memory is generally much smaller and more expensive
    - kristjansson 4y ago
      To elaborate on the sibling comment: main memory is much bigger, but CPUs are much, much slower. It would be a challenge to merely run a model like this on CPU, and totally infeasible to train one. So the challenge is to fit into the memory of a single GPU you can afford, coordinate multiple GPUs, or efficiently page from main memory into GPU.
- Delitio 4y ago
  Is there any source which explains what billion of parameters actually are? In my mind a parameter is: language, dialect, perhaps context parameters (food, dinner, lunch, travel) and if we than talk about language and audio perhaps sound waves, gender. Or are context parameters which gives you insight? Like a billion of parameters are literally something like travel=false, travel-europe=true people speaking=e, age, height,
  - jefft255 4y ago
    The parameters are the number of weights in a neural network, in this case.
  - matt123456789 4y ago
    A parameter is a scalar value, most of which are in the attention matrices and feedforward matrices, you also hear these called “weights”. Any intro to DL course will cover these in detail. I recommend started with Andrew Ng’s Coursera class on Intro to Machine Learning, although there may be better ones out there now.
    - Delitio 4y ago
      Input parameter vs. weights then? I see tx
      - lostmsu 4y ago
        These networks (text models) usually have around a few thousand inputs.
  - dotnet00 4y ago
    Parameters are just floating point numbers, at most they can be seen as degrees of freedom or kind of like the order of a polynomial used in curve fitting. They're too abstract to assign much meaning to individual parameters, as our understanding of why their values are exactly the way they are is extremely limited.
  - brrrrrm 4y ago
Inufu 4y ago
Yes, parameters are usually stored as float32, activations as bfloat16 or float16.

70 billion parameters... Is each of those a 4-byte float? So, is that 280 billion bytes of just parameters?

sudosysgen 4y ago

I'm fairly confident each of those is a 2-byte float, but yes that's over 100 GB of parameters.

sillysaurusx 4y ago
Welcome to the party! I joined ML because I realized I could help. You can too. I bet you’re both already thinking of clever ways to deal with massive models from an infrastructure standpoint. That’s just one of hundreds of interesting problems.
- native_samples 4y ago
  Is 100GB of parameters really that large? 128GB of RAM on a server class machine is not unusual. Seems such a model could fit entirely in RAM.
  - andbberger 4y ago
    GPU memory is generally much smaller and more expensive
  - kristjansson 4y ago
    To elaborate on the sibling comment: main memory is much bigger, but CPUs are much, much slower. It would be a challenge to merely run a model like this on CPU, and totally infeasible to train one. So the challenge is to fit into the memory of a single GPU you can afford, coordinate multiple GPUs, or efficiently page from main memory into GPU.
Delitio 4y ago
Is there any source which explains what billion of parameters actually are? In my mind a parameter is: language, dialect, perhaps context parameters (food, dinner, lunch, travel) and if we than talk about language and audio perhaps sound waves, gender. Or are context parameters which gives you insight? Like a billion of parameters are literally something like travel=false, travel-europe=true people speaking=e, age, height,
- jefft255 4y ago
  The parameters are the number of weights in a neural network, in this case.
- matt123456789 4y ago
  A parameter is a scalar value, most of which are in the attention matrices and feedforward matrices, you also hear these called “weights”. Any intro to DL course will cover these in detail. I recommend started with Andrew Ng’s Coursera class on Intro to Machine Learning, although there may be better ones out there now.
  - Delitio 4y ago
    Input parameter vs. weights then? I see tx
    - lostmsu 4y ago
      These networks (text models) usually have around a few thousand inputs.
- dotnet00 4y ago
  Parameters are just floating point numbers, at most they can be seen as degrees of freedom or kind of like the order of a polynomial used in curve fitting. They're too abstract to assign much meaning to individual parameters, as our understanding of why their values are exactly the way they are is extremely limited.
- brrrrrm 4y ago

Inufu 4y ago

Yes, parameters are usually stored as float32, activations as bfloat16 or float16.