4 ms·

> Well humans have about 100 trillion synapses. We already know that the higher the parameter count, the lower the training data required. Do you have any refe

> Well humans have about 100 trillion synapses. We already know that the higher the parameter count, the lower the training data required. Do you have any reference to back this claim, because it sounds is very curious to me. My understanding was pretty much the opposite, that current LLM technology require a bigger training set as you increase the parameter count. I'm no NN expert in any way though.

famouswaffles 4y ago
You can look at any training run comparing multiple parameter sizes on the dataset. https://arxiv.org/abs/2204.02311 https://arxiv.org/abs/2204.02311 If you're increasing parameter size, it's a no brainer to increase data too as that will still also increase performance. The point is that for any arbitrary performance x, the data required to reach it reduces with size.
- littlestymaar 4y ago
  > If you're increasing parameter size, it's a no brainer to increase data too as that will still also increase performance. It also increases the cost by a lot, so it's not a no-brainer at all. If they could beat the state of the art with only a fraction of the training cost, I suspect that they'd do so… > The point is that for any arbitrary performance x, the data required to reach it reduces with size. This is the claim you're making, but it's not substantiated.
  - famouswaffles 4y ago
    >It also increases the cost by a lot, so it's not a no-brainer at all. Okay?.. Parameter size increases also increase cost a lot. Far more than more training data. Costs that stay well beyond training. Training on 1T tokens vs 500b won't change how resources it takes to run. Not the cases with parameter sizes. >If they could beat the state of the art with only a fraction of the training cost, I suspect that they'd do so… Not sure what this has to do with anything lol >This is the claim you're making, but it's not substantiated. I'm sorry but can you perhaps just read the paper sent ? Google trained 3 differently sized models of the same architecture (8b, 62b, 540b) on the same dataset of 780b tokens and evaluated all 3 on various tasks.
    - coldtea 4y ago
      >Google trained 3 differently sized models of the same architecture (8b, 62b, 540b) on the same dataset of 780b tokens and evaluated all 3 on various tasks. That's quite a small sample to argue the generic point that "for any arbitrary performance x, the data required to reach it reduces with size". Key part being: "for any arbitrary performance".
      - famouswaffles 4y ago
        Any paper that's trained more than one model size on the same data affirms the same thing. Llama 13b was better than 7b and Llama 66b was better than 33b. If you're bothered with how general a statement I'm making then Ok, point is that all training so far has pointed towards that.
    - littlestymaar 4y ago
      > Okay?.. Parameter size increases also increase cost a lot. Far more than more training data. Yup, and that's why lots of work goes into smaller model trained beyond the Chinchilla-optimality. But increasing the model size alone doesn't seem to make sense to anyone for some reason. > I'm sorry but can you perhaps just read the paper sent? I did skim it, and it's not making the claim you are. > Google trained 3 differently sized models of the same architecture (8b, 62b, 540b) on the same dataset of 780b tokens and evaluated all 3 on various tasks. This has nothing to do with your claim that “We already know that the higher the parameter count, the lower the training data required”. To back such a claim we'd need a 540b model trained on 10b token beating / rivaling with a 8b parameters trained on 400b. I'm not aware of anything like this existing today. That a big model trained with enough data can beat a smaller model on the same data isn't the same claim at all.
      - famouswaffles 4y ago
        >But increasing the model size alone doesn't seem to make sense to anyone for some reason. It's not Economically viable or efficient to just scale model size. >This has nothing to do with your claim that “We already know that the higher the parameter count, the lower the training data required”. To back such a claim we'd need a 540b model trained on 10b token beating / rivaling with a 8b parameters trained on 400b. I'm not aware of anything like this existing today. Literally this is what I said >a 50 billion parameter model will far outperform a 5 billion one TRAINED ON THE SAME DATA. A 400b dataset is not the same training data as a 10b dataset

famouswaffles 4y ago

You can look at any training run comparing multiple parameter sizes on the dataset. https://arxiv.org/abs/2204.02311 https://arxiv.org/abs/2204.02311 If you're increasing parameter size, it's a no brainer to increase data too as that will still also increase performance. The point is that for any arbitrary performance x, the data required to reach it reduces with size.

littlestymaar 4y ago
> If you're increasing parameter size, it's a no brainer to increase data too as that will still also increase performance. It also increases the cost by a lot, so it's not a no-brainer at all. If they could beat the state of the art with only a fraction of the training cost, I suspect that they'd do so… > The point is that for any arbitrary performance x, the data required to reach it reduces with size. This is the claim you're making, but it's not substantiated.
- famouswaffles 4y ago
  >It also increases the cost by a lot, so it's not a no-brainer at all. Okay?.. Parameter size increases also increase cost a lot. Far more than more training data. Costs that stay well beyond training. Training on 1T tokens vs 500b won't change how resources it takes to run. Not the cases with parameter sizes. >If they could beat the state of the art with only a fraction of the training cost, I suspect that they'd do so… Not sure what this has to do with anything lol >This is the claim you're making, but it's not substantiated. I'm sorry but can you perhaps just read the paper sent ? Google trained 3 differently sized models of the same architecture (8b, 62b, 540b) on the same dataset of 780b tokens and evaluated all 3 on various tasks.
  - coldtea 4y ago
    >Google trained 3 differently sized models of the same architecture (8b, 62b, 540b) on the same dataset of 780b tokens and evaluated all 3 on various tasks. That's quite a small sample to argue the generic point that "for any arbitrary performance x, the data required to reach it reduces with size". Key part being: "for any arbitrary performance".
    - famouswaffles 4y ago
      Any paper that's trained more than one model size on the same data affirms the same thing. Llama 13b was better than 7b and Llama 66b was better than 33b. If you're bothered with how general a statement I'm making then Ok, point is that all training so far has pointed towards that.
  - littlestymaar 4y ago
    > Okay?.. Parameter size increases also increase cost a lot. Far more than more training data. Yup, and that's why lots of work goes into smaller model trained beyond the Chinchilla-optimality. But increasing the model size alone doesn't seem to make sense to anyone for some reason. > I'm sorry but can you perhaps just read the paper sent? I did skim it, and it's not making the claim you are. > Google trained 3 differently sized models of the same architecture (8b, 62b, 540b) on the same dataset of 780b tokens and evaluated all 3 on various tasks. This has nothing to do with your claim that “We already know that the higher the parameter count, the lower the training data required”. To back such a claim we'd need a 540b model trained on 10b token beating / rivaling with a 8b parameters trained on 400b. I'm not aware of anything like this existing today. That a big model trained with enough data can beat a smaller model on the same data isn't the same claim at all.
    - famouswaffles 4y ago
      >But increasing the model size alone doesn't seem to make sense to anyone for some reason. It's not Economically viable or efficient to just scale model size. >This has nothing to do with your claim that “We already know that the higher the parameter count, the lower the training data required”. To back such a claim we'd need a 540b model trained on 10b token beating / rivaling with a 8b parameters trained on 400b. I'm not aware of anything like this existing today. Literally this is what I said >a 50 billion parameter model will far outperform a 5 billion one TRAINED ON THE SAME DATA. A 400b dataset is not the same training data as a 10b dataset