3 ms·

I’m not sure ML researchers would agree that number of (compressed) bytes are more meaningful than number of parameters. Parameters have mathematical meaning –

I’m not sure ML researchers would agree that number of (compressed) bytes are more meaningful than number of parameters. Parameters have mathematical meaning – bytes doesn’t.

mjburgess 3y ago
My point is that they don't have a mathematical meaning. They have a training-time impact, but that's more relevant when creating than using. You'd need to know how many parameters were independently covarying for any given class of predictions. It certainly isnt all of them. You could cite the "average dropout percent to random-level accuracy on a given class of problems" (my guess is that this would show 5-20% of parameters could be dropped). My point, I suppose, is that users of NNs arent interested in the architecture characteristics which affect training -- they're interested in how capable any given model will be. For this we really want to know how large the training data was, and how compressed it has been. If it's 1PB -> 1MB then we can easily say that's much less useful (but much faster to use) than 1PB -> 0.5TB. Likewise we can say, if it's a video generator, that 1PB is far too small to be generally useful -- so at best it'll be domain-speicifc.
- bjornsing 3y ago
  If you don’t believe me you can try training an LLM with just a single parameter, specified to an incredible precision using e.g a trillion bytes. Hint: it won’t perform very well.
- tyre 3y ago
  > My point, I suppose, is that users of NNs arent interested in the architecture characteristics which affect training -- they're interested in how capable any given model will be. Yes. A more helpful bit of information could be what the model was pre-trained on. Assuming they’re trying to refine it for a more specific task. Size is helpful for “what can I run on my machine” (or how much would it cost to run on a server.) Not all models are created equal, given a byte size, for a given task.
piker 3y ago
Bytes does imply a level of precision, however, which affects the mathematical meaning. Perhaps there’s a metric that captures both.
- xpe 3y ago
  Allocating more bits for each parameter increases precision, by definition. But that doesn’t come for free.* So it is useful to optimize network performance for a given number of total parameter bytes. I haven’t done a recent literature review, but my hand-wavy guessplanation is that a NN (as a whole) can adapt to relatively low precision parameters. Up to a point. * In general. Given actual hardware designs, there are places where you have slack in the system. So adding some extra parameters, e.g. to fully utilize a GPU’s core’s threads (e.g. 32), might actually cost you nothing.

I’m not sure ML researchers would agree that number of (compressed) bytes are more meaningful than number of parameters. Parameters have mathematical meaning – bytes doesn’t.

mjburgess 3y ago

My point is that they don't have a mathematical meaning. They have a training-time impact, but that's more relevant when creating than using. You'd need to know how many parameters were independently covarying for any given class of predictions. It certainly isnt all of them. You could cite the "average dropout percent to random-level accuracy on a given class of problems" (my guess is that this would show 5-20% of parameters could be dropped). My point, I suppose, is that users of NNs arent interested in the architecture characteristics which affect training -- they're interested in how capable any given model will be. For this we really want to know how large the training data was, and how compressed it has been. If it's 1PB -> 1MB then we can easily say that's much less useful (but much faster to use) than 1PB -> 0.5TB. Likewise we can say, if it's a video generator, that 1PB is far too small to be generally useful -- so at best it'll be domain-speicifc.

bjornsing 3y ago
If you don’t believe me you can try training an LLM with just a single parameter, specified to an incredible precision using e.g a trillion bytes. Hint: it won’t perform very well.
tyre 3y ago
> My point, I suppose, is that users of NNs arent interested in the architecture characteristics which affect training -- they're interested in how capable any given model will be. Yes. A more helpful bit of information could be what the model was pre-trained on. Assuming they’re trying to refine it for a more specific task. Size is helpful for “what can I run on my machine” (or how much would it cost to run on a server.) Not all models are created equal, given a byte size, for a given task.

piker 3y ago

Bytes does imply a level of precision, however, which affects the mathematical meaning. Perhaps there’s a metric that captures both.

xpe 3y ago
Allocating more bits for each parameter increases precision, by definition. But that doesn’t come for free.* So it is useful to optimize network performance for a given number of total parameter bytes. I haven’t done a recent literature review, but my hand-wavy guessplanation is that a NN (as a whole) can adapt to relatively low precision parameters. Up to a point. * In general. Given actual hardware designs, there are places where you have slack in the system. So adding some extra parameters, e.g. to fully utilize a GPU’s core’s threads (e.g. 32), might actually cost you nothing.