3 ms·

Stray thought: It would be better to specify NNs in terms of their training-size to weight-size in bytes. Rather than "No. Parameters", or at least, this ratio

Stray thought: It would be better to specify NNs in terms of their training-size to weight-size in bytes. Rather than "No. Parameters", or at least, this ratio with the number of parameters. So, eg., I'd imagine ChatGPT would be, say: 100s PB in 0.5TB. The number of parameters is a nearly meaningless metric, consider, eg., that if all the parameters covary then there's one "functional" parameter. The compression ratio tells you the real reason why a NN performs. Ie., you can have 100s bns of parameters, but without that 100s PB -> 0.5TB, which you can't afford, it's all rather pointless.

moffkalast 3y ago
Well it's not a completely meaningless metric as it immediately tells you roughly how much memory you need to load it, which is kind of important?
- mjburgess 3y ago
  If you look at my suggestion, it's to state exactly that memory -- rather than to estimate based on bits/parameter.
  - moffkalast 3y ago
    Well then do explain a bit further, I still don't fully grasp what "100s PT in 0.5T" means exactly. 100 petatokens in half a trillion? Half a terrabyte? 100 seconds? Plus afaik base model training tokens don't have the same effect as fine tuning tokens, so there would need to be a way to specify each of those separately.
    - boomskats 3y ago
      FWIW I easily interpreted these as '100s of petabytes' and '0.5 terabytes' without having to give it too much thought. The original comment explicitly specified 'bytes' as the unit being suggested.
    - mjburgess 3y ago
      I edited to be TB,PB --- I was thinking of these as prefixes on bytes
YetAnotherNick 3y ago
> 100s PB Unrelated and I know it is just a representative number but I have seen the training data to be assumed something in this range few times. Entire training set of ChatGPT is almost surely less than a TB or two with compression which is 5 orders of magnitude lower. I believe that such efficient representation of text is one of the biggest reason why text models are working so well but image understanding models are not.
- mjburgess 3y ago
  Perhaps the process was an initial c. 1PB then sampled down to the TBs. Text is extremely lightweight, so I suppose everything ever written is at most 1-10PB. This is one of the illusions of text-generative NNs: a 0.5TB weight set is basically enough to store every book. Making claims to "out-sample generalisation" extremely suspicious, and indeed, fairly obviously false. eg., Ask ChatGPT to write tic-tak-toe in javascript and you get a working game; as it to write duck-hunt and you dont.
bjornsing 3y ago
I’m not sure ML researchers would agree that number of (compressed) bytes are more meaningful than number of parameters. Parameters have mathematical meaning – bytes doesn’t.
- mjburgess 3y ago
  My point is that they don't have a mathematical meaning. They have a training-time impact, but that's more relevant when creating than using. You'd need to know how many parameters were independently covarying for any given class of predictions. It certainly isnt all of them. You could cite the "average dropout percent to random-level accuracy on a given class of problems" (my guess is that this would show 5-20% of parameters could be dropped). My point, I suppose, is that users of NNs arent interested in the architecture characteristics which affect training -- they're interested in how capable any given model will be. For this we really want to know how large the training data was, and how compressed it has been. If it's 1PB -> 1MB then we can easily say that's much less useful (but much faster to use) than 1PB -> 0.5TB. Likewise we can say, if it's a video generator, that 1PB is far too small to be generally useful -- so at best it'll be domain-speicifc.
  - bjornsing 3y ago
    If you don’t believe me you can try training an LLM with just a single parameter, specified to an incredible precision using e.g a trillion bytes. Hint: it won’t perform very well.
  - tyre 3y ago
    > My point, I suppose, is that users of NNs arent interested in the architecture characteristics which affect training -- they're interested in how capable any given model will be. Yes. A more helpful bit of information could be what the model was pre-trained on. Assuming they’re trying to refine it for a more specific task. Size is helpful for “what can I run on my machine” (or how much would it cost to run on a server.) Not all models are created equal, given a byte size, for a given task.
- piker 3y ago

moffkalast 3y ago

Well it's not a completely meaningless metric as it immediately tells you roughly how much memory you need to load it, which is kind of important?

mjburgess 3y ago
If you look at my suggestion, it's to state exactly that memory -- rather than to estimate based on bits/parameter.
- moffkalast 3y ago
  Well then do explain a bit further, I still don't fully grasp what "100s PT in 0.5T" means exactly. 100 petatokens in half a trillion? Half a terrabyte? 100 seconds? Plus afaik base model training tokens don't have the same effect as fine tuning tokens, so there would need to be a way to specify each of those separately.
  - boomskats 3y ago
    FWIW I easily interpreted these as '100s of petabytes' and '0.5 terabytes' without having to give it too much thought. The original comment explicitly specified 'bytes' as the unit being suggested.
  - mjburgess 3y ago
    I edited to be TB,PB --- I was thinking of these as prefixes on bytes

YetAnotherNick 3y ago

> 100s PB Unrelated and I know it is just a representative number but I have seen the training data to be assumed something in this range few times. Entire training set of ChatGPT is almost surely less than a TB or two with compression which is 5 orders of magnitude lower. I believe that such efficient representation of text is one of the biggest reason why text models are working so well but image understanding models are not.

mjburgess 3y ago
Perhaps the process was an initial c. 1PB then sampled down to the TBs. Text is extremely lightweight, so I suppose everything ever written is at most 1-10PB. This is one of the illusions of text-generative NNs: a 0.5TB weight set is basically enough to store every book. Making claims to "out-sample generalisation" extremely suspicious, and indeed, fairly obviously false. eg., Ask ChatGPT to write tic-tak-toe in javascript and you get a working game; as it to write duck-hunt and you dont.

bjornsing 3y ago

I’m not sure ML researchers would agree that number of (compressed) bytes are more meaningful than number of parameters. Parameters have mathematical meaning – bytes doesn’t.

mjburgess 3y ago
My point is that they don't have a mathematical meaning. They have a training-time impact, but that's more relevant when creating than using. You'd need to know how many parameters were independently covarying for any given class of predictions. It certainly isnt all of them. You could cite the "average dropout percent to random-level accuracy on a given class of problems" (my guess is that this would show 5-20% of parameters could be dropped). My point, I suppose, is that users of NNs arent interested in the architecture characteristics which affect training -- they're interested in how capable any given model will be. For this we really want to know how large the training data was, and how compressed it has been. If it's 1PB -> 1MB then we can easily say that's much less useful (but much faster to use) than 1PB -> 0.5TB. Likewise we can say, if it's a video generator, that 1PB is far too small to be generally useful -- so at best it'll be domain-speicifc.
- bjornsing 3y ago
  If you don’t believe me you can try training an LLM with just a single parameter, specified to an incredible precision using e.g a trillion bytes. Hint: it won’t perform very well.
- tyre 3y ago
  > My point, I suppose, is that users of NNs arent interested in the architecture characteristics which affect training -- they're interested in how capable any given model will be. Yes. A more helpful bit of information could be what the model was pre-trained on. Assuming they’re trying to refine it for a more specific task. Size is helpful for “what can I run on my machine” (or how much would it cost to run on a server.) Not all models are created equal, given a byte size, for a given task.
piker 3y ago