4 ms·

Naive question, how many parameters do most large models have?

gwern 10y ago
Low millions to a max of 1b or so (excluding special cases like "Outrageously large neural networks" paper). The recent Google Translate RNN was in the neighborhood of 50m parameters, IIRC. It can be a little difficult to calculate since you have a lot of weight-sharing and tied-weights going on in CNNs and other architectures.
- Moshe_Silnorin 10y ago
  So the largest are about the size of a bee brain? Amazing how many applications you can get from insect-brain sized networks, about a hundred thousand times smaller than a human brain if a parameter is at all analogous to a synapse. Seems like human-level AI is a ways off though.
  - argonaut 10y ago
    A parameter != a neuron. A single neuron probably has thousands, if not hundreds of thousands, of "parameters" (of course the vast majority are probably unrelated to learning - but given we barely understand neurological learning it's hard to say).
    - Moshe_Silnorin 10y ago
      I was comparing parameter counts to synapse counts, not neuron counts.
      - deepnotderp 10y ago
        Please don't make these sorts of comparisons between biological neurons and our "neural networks". They're really not the same thing at all. We'd rather call them "differentiable networks", and IIRC, the human brain is supposed to have only 6 layers in the visual cortex.
        apl 10y ago
        Cortical layers are in no way equivalent to DNN layers.
        deleted 10y ago
        [deleted]
      - argonaut 10y ago
        The fair comparison is comparing the number of "free variables" in the system. This is how machine learning models are compared. You don't compare "number of free parameters in linear regression" with "number of layers of neural nets," you compare number of parameters with number of parameters.
  - eva1984 10y ago
    Talking about size, but not specific architecture is pretty meaningless at this moment. The assumption that only when we have comparable number of parameters to the number of brain cells we can have nearly human performance is no longer true. It gradually loses traction once people discover that stochastic connections may be more effective than merely increase layers/parameters.
  - gwern 10y ago
    Wouldn't that imply the other way around? If you can get these amazing, often human-level, performances out of tiny insect-brain equivalent NNs - which I don't believe anyone predicted in advance - that suggests that the estimates of thousands of parameters or megaflops per neurons (see the Whole Brain Emulation Roadmap for a compilation of estimates) are irrelevant and human-level AI requires much less computation than expected.
    - Moshe_Silnorin 10y ago
      Naive trend-extrapolation (assuming 1 synapse == 1 parameter) implies 17 iterations of Moore's law before we get human level AI, or some sort of ASIC, I suppose. This is what made me think human-level AI is a ways off. Brain dead trend-extrapolation is a pretty good prediction tool, but I suppose my assumption that modern NNs aren't more efficient was sort of an arbitrary assumption that makes the exercise useless. Is there any work on this? I'm not sure what evidence the performance of these systems show us, as bees are pretty smart for their size. These system are about as optimized for lip-reading and translation as bees are for their various tasks, so it wouldn't be surprise me if it turned out that they are similarly powerful per synapse to NNs.
      - tshadley 10y ago
        17 years doesn't seem too far off. Tim Detmer's detailed analysis puts it anywhere from 2037 to 2078, with his preference the latter (or later). http://timdettmers.com/2015/07/27/brain-vs-deep-learning-singularity/ http://timdettmers.com/2015/07/27/brain-vs-deep-learning-sin... A key assumption I don't share with Detmer is that we must match the brain's raw computational power to have a chance at human-level AI. From an evolutionary point of view, most of the brain's computations surely must be superfluous to intelligence and necessary instead to a vast host of other biological functions.
      - gwern 10y ago
        > Naive trend-extrapolation (assuming 1 synapse == 1 parameter) implies 17 iterations of Moore's law before we get human level AI, or some sort of ASIC, I suppose. I'm not sure how you get that. My point is that artificial NNs may well be more efficient than biological ones because they are punching so far above their weight based on what you would expect from a naive comparison of parameters, in which case it's not 17 iterations but a lot less. Further, modern NNs don't typically max out the memory of a single GPU (since we want to do minibatches of at least n=10 for the training phase and researchers care a lot less about the forward passes or deployment), and groups like Google Brain or Deepmind have shown the ability to use ~1000 GPUs, so that's 4 orders of magnitude right there (doable with asychronous training like synthetic gradients). > I'm not sure what evidence the performance of these systems show us, as bees are pretty smart for their size. What do bees do that is remotely as demanding as being able to translate between English, French, German, Japanese etc at very high quality?
  - j2kun 10y ago
    I read somewhere[1] that because we know far more now about human neurons than we did five years ago, modern CNN's are better compared to a _single_ neuron (but are probably still simpler than them). I'm far from an expert in brain biology, but what I took away from that is that it's specious to make these comparisons based on our current knowledge of either field. [1]: http://timdettmers.com/2015/07/27/brain-vs-deep-learning-singularity/ http://timdettmers.com/2015/07/27/brain-vs-deep-learning-sin...
    - gwern 10y ago
      > modern CNN's are better compared to a _single_ neuron You obviously cannot reach human-level performance on ImageNet or do anything at all with a single neuron, so such an absurd equation refutes any such calculation or attempt to use a comparison as an upper bound.
      - j2kun 10y ago
        The linked article explains how a single human brain neuron is a lot more complex than previously thought. Obvious is also a word we should avoid using in discussions about brain biology.
  - ma2rten 10y ago
    I am not surprised at all how many applications you can get from insect-brain sized networks, since they only solve one specific purpose. If the same network could solve all those problems that would be surprising. I also think neural networks could save alot of parameters if they use sparse connections. That just doesn't happen to work well with our current hardware.
    - gwern 10y ago
      > I am not surprised at all how many applications you can get from insect-brain sized networks, since they only solve one specific purpose. If the same network could solve all those problems that would be surprising. Transfer learning demonstrates that the 'specialization' objection is moot. The CNNs are learning large parts of what goes into general-purpose vision, not super-narrow specific things, otherwise it would be impossible to take a classification CNN and reuse only a tweaked version for image captioning, localization, tagging, segmentation etc and find the intermediate layers learning semantically meaningful things.

gwern 10y ago

Low millions to a max of 1b or so (excluding special cases like "Outrageously large neural networks" paper). The recent Google Translate RNN was in the neighborhood of 50m parameters, IIRC. It can be a little difficult to calculate since you have a lot of weight-sharing and tied-weights going on in CNNs and other architectures.

Moshe_Silnorin 10y ago
So the largest are about the size of a bee brain? Amazing how many applications you can get from insect-brain sized networks, about a hundred thousand times smaller than a human brain if a parameter is at all analogous to a synapse. Seems like human-level AI is a ways off though.
- argonaut 10y ago
  A parameter != a neuron. A single neuron probably has thousands, if not hundreds of thousands, of "parameters" (of course the vast majority are probably unrelated to learning - but given we barely understand neurological learning it's hard to say).
  - Moshe_Silnorin 10y ago
    I was comparing parameter counts to synapse counts, not neuron counts.
    - deepnotderp 10y ago
      Please don't make these sorts of comparisons between biological neurons and our "neural networks". They're really not the same thing at all. We'd rather call them "differentiable networks", and IIRC, the human brain is supposed to have only 6 layers in the visual cortex.
      - apl 10y ago
        Cortical layers are in no way equivalent to DNN layers.
      - deleted 10y ago
        [deleted]
    - argonaut 10y ago
      The fair comparison is comparing the number of "free variables" in the system. This is how machine learning models are compared. You don't compare "number of free parameters in linear regression" with "number of layers of neural nets," you compare number of parameters with number of parameters.
- eva1984 10y ago
  Talking about size, but not specific architecture is pretty meaningless at this moment. The assumption that only when we have comparable number of parameters to the number of brain cells we can have nearly human performance is no longer true. It gradually loses traction once people discover that stochastic connections may be more effective than merely increase layers/parameters.
- gwern 10y ago
  Wouldn't that imply the other way around? If you can get these amazing, often human-level, performances out of tiny insect-brain equivalent NNs - which I don't believe anyone predicted in advance - that suggests that the estimates of thousands of parameters or megaflops per neurons (see the Whole Brain Emulation Roadmap for a compilation of estimates) are irrelevant and human-level AI requires much less computation than expected.
  - Moshe_Silnorin 10y ago
    Naive trend-extrapolation (assuming 1 synapse == 1 parameter) implies 17 iterations of Moore's law before we get human level AI, or some sort of ASIC, I suppose. This is what made me think human-level AI is a ways off. Brain dead trend-extrapolation is a pretty good prediction tool, but I suppose my assumption that modern NNs aren't more efficient was sort of an arbitrary assumption that makes the exercise useless. Is there any work on this? I'm not sure what evidence the performance of these systems show us, as bees are pretty smart for their size. These system are about as optimized for lip-reading and translation as bees are for their various tasks, so it wouldn't be surprise me if it turned out that they are similarly powerful per synapse to NNs.
    - tshadley 10y ago
      17 years doesn't seem too far off. Tim Detmer's detailed analysis puts it anywhere from 2037 to 2078, with his preference the latter (or later). http://timdettmers.com/2015/07/27/brain-vs-deep-learning-singularity/ http://timdettmers.com/2015/07/27/brain-vs-deep-learning-sin... A key assumption I don't share with Detmer is that we must match the brain's raw computational power to have a chance at human-level AI. From an evolutionary point of view, most of the brain's computations surely must be superfluous to intelligence and necessary instead to a vast host of other biological functions.
    - gwern 10y ago
      > Naive trend-extrapolation (assuming 1 synapse == 1 parameter) implies 17 iterations of Moore's law before we get human level AI, or some sort of ASIC, I suppose. I'm not sure how you get that. My point is that artificial NNs may well be more efficient than biological ones because they are punching so far above their weight based on what you would expect from a naive comparison of parameters, in which case it's not 17 iterations but a lot less. Further, modern NNs don't typically max out the memory of a single GPU (since we want to do minibatches of at least n=10 for the training phase and researchers care a lot less about the forward passes or deployment), and groups like Google Brain or Deepmind have shown the ability to use ~1000 GPUs, so that's 4 orders of magnitude right there (doable with asychronous training like synthetic gradients). > I'm not sure what evidence the performance of these systems show us, as bees are pretty smart for their size. What do bees do that is remotely as demanding as being able to translate between English, French, German, Japanese etc at very high quality?
- j2kun 10y ago
  I read somewhere[1] that because we know far more now about human neurons than we did five years ago, modern CNN's are better compared to a _single_ neuron (but are probably still simpler than them). I'm far from an expert in brain biology, but what I took away from that is that it's specious to make these comparisons based on our current knowledge of either field. [1]: http://timdettmers.com/2015/07/27/brain-vs-deep-learning-singularity/ http://timdettmers.com/2015/07/27/brain-vs-deep-learning-sin...
  - gwern 10y ago
    > modern CNN's are better compared to a _single_ neuron You obviously cannot reach human-level performance on ImageNet or do anything at all with a single neuron, so such an absurd equation refutes any such calculation or attempt to use a comparison as an upper bound.
    - j2kun 10y ago
      The linked article explains how a single human brain neuron is a lot more complex than previously thought. Obvious is also a word we should avoid using in discussions about brain biology.
- ma2rten 10y ago
  I am not surprised at all how many applications you can get from insect-brain sized networks, since they only solve one specific purpose. If the same network could solve all those problems that would be surprising. I also think neural networks could save alot of parameters if they use sparse connections. That just doesn't happen to work well with our current hardware.
  - gwern 10y ago
    > I am not surprised at all how many applications you can get from insect-brain sized networks, since they only solve one specific purpose. If the same network could solve all those problems that would be surprising. Transfer learning demonstrates that the 'specialization' objection is moot. The CNNs are learning large parts of what goes into general-purpose vision, not super-narrow specific things, otherwise it would be impossible to take a classification CNN and reuse only a tweaked version for image captioning, localization, tagging, segmentation etc and find the intermediate layers learning semantically meaningful things.