4 ms·

If the model weights aren’t open source, the model isn’t open source in my opinion. It’s not like a switch transformer is that much more complicated to slap int

If the model weights aren’t open source, the model isn’t open source in my opinion. It’s not like a switch transformer is that much more complicated to slap into pytorch than any other new research. Also in my personal opinion, while yes this model performs great, it’s ridiculously parameter inefficient. 1.6T parameters is 3.12TB using bfloat 16. I’d rather take 7x longer to train and have a model I can actually put places.

amelius 6y ago
> it’s ridiculously parameter inefficient How do you know that? Perhaps with this methodology you really need those 3.12TB to reach comparable performance?
- buildbot 6y ago
  If you read through the paper where they present switch transformers, they train all models to the same performance, so for example t5-large, at 770M parameters[1] is just as good as this model. That’s only 1.54GB [1] https://huggingface.co/t5-large https://huggingface.co/t5-large
- thesz 6y ago
  I beg to disagree. [1] provides one with a whole-data-set training method (ADMM, one of such methods). Page 8 contains figure 2(b) - accuracy of training after specified amount of time. Note that ADMM start where stochastic gradient stops. [1] https://arxiv.org/pdf/1605.02026.pdf https://arxiv.org/pdf/1605.02026.pdf At [2] I tried to apply logistic regression trained using reweighted least squares algorithm on the same Higgs boson data set. I've got the same accuracy (64%) as mentioned in the ADMM paper with much less number of coefficients - basically, just the size of input vector + 1 instead of 300 such rows of coefficients and then 300x1 affine transformation. When I added squares of inputs (for the simplest approximation of polynomial regression) and used the same reweighted iterative least squares algorithm, I've got even better accuracy (66%) for double the number of coefficients. [2] https://github.com/thesz/higgs-logistic-regression https://github.com/thesz/higgs-logistic-regression There's a hypothesis [3] that SGD and ADAM are best optimizers because that everyone use and report on. Rarely if ever you get anything that differ. [3] https://parameterfree.com/2020/12/06/neural-network-maybe-evolved-to-make-adam-the-best-optimizer/ https://parameterfree.com/2020/12/06/neural-network-maybe-ev... So, answering your question of "how do you know" - researchers at Google cannot do IRLS (search provides IRLS only for logistic regression in Tensorflow), they cannot do Hessian-free optimization ([4], closed due lack of activity - notice the "we can't support RNN due to the WHILE loop" bonanza), etc. All due to the fact they have to use Tensorflow - it just does not support these things. https://github.com/tensorflow/tensorflow/issues/2682 https://github.com/tensorflow/tensorflow/issues/2682 I haven't seen anything about whole-data-set optimization from Google at all. That's why I (and only me - due to standing I take and experiments I did) conclude that they do not quite care about parameter efficiency.
  - vladf 6y ago
    > they do not quite care about parameter efficiency. Google Research is pretty big, I used to think like you did but I think it's mostly b/c DeepMind just hogs all the spotlight. Check out PRESS [0] for example. [0]: https://research.google/pubs/pub46141 https://research.google/pubs/pub46141
    - thesz 6y ago
      Thank you! I skimmed over the abstract and will read the paper later, it seems interesting. But you gave me another point to support my view: PRESS uses stochastic gradient, not second-order method like IRLS.
      - vladf 6y ago
        I agree, I was really only proposing PRESS for the "parameter efficiency" part of your comment. It'd be interesting to see some modern takes on IRLS. I think generally this goes against the grain of the Cheap Gradient Principle which is why you see less of it (edit: eh, I think Fisher scoring can be cast in this light). For instance, on modern modelling problems with non-linearities and change points, it's a lot less easy to do something like IRLS in an end-to-end system, but interesting as a research direction.
        thesz 6y ago
        I also agree with "Cheap Gradient Principle". I see it as a case of "width of two horse backs from Ancient Rome determine Shuttle buster width" (which is untrue but cool as a reference). The very SGD thing was developed because it was the only way to train something like neural network with small memory and, more importantly, in reasonable time. Multiplying of training time by N (number of parameter) meant having good result in a year, not in a day. And today we have large batch training with complex synchronization systems to speed up training even more. Which bring us closer to the whole-dataset training and, I guess, second-order optimization as well.
gradys 6y ago
Mixture of experts architectures like this are a specific design decision to increase the parameter count but use and update those parameters sparsely. Sure, that design decision doesn't fit all scenarios, but it fits some, and it has its own advantages, like faster training time.
- ogrisel 6y ago
  But do you see any practical scenario where you have to keep 3TB of parameters in accelerator's RAM at inference time? The accuracy would have to be significantly higher that any alternative to justify monopolizing that many hardware resources.
  - gradys 6y ago
    Yep, it would definitely be difficult to justify running it in production. Accuracy would need to be higher as you said or it would need to be applicable to more tasks such that you can take other models out of production. This kind of model could be used as the teacher in a distillation setup too though. Then faster training of the teacher is actually a huge benefit since it speeds up model development iteration cycles. But even if it weren't practical to use in production in any sense, I'd argue there's value in doing the basic research of exploring design space of architectures in this way. This came out of a research team at Google. It may inspire and inform smaller, more practical architectures.
    - gwern 6y ago
      "Yep, it would definitely be difficult to justify running it in production. Accuracy would need to be higher as you said or it would need to be applicable to more tasks such that you can take other models out of production." Part of the justification is the MoE sparsity by design means only a small part of the model will be activated by a given query. Don't think of it as a single giant 1t model, think of it as 50 small models which happen to share a glue layer at the input. So at deployment, you could, for example, keep the gating-layer in RAM and only pull the necessary sub-model off disk as necessary. Or you could shard the sub-models over 51 GPUs and feed the master gating-layer+GPU lots of queries, and each query will be dispatched to a different expert+GPU pair. This could easily be competitive with running a lot of dense models in parallel trying to keep up with the same load.
summerlight 6y ago
Google's legal team would be freaking out if any of its engineers propose open sourcing model weights. This is a mine field full of legal and privacy headaches and I guess it is infeasible to audit 1.6T parameters anyway...
- Closi 6y ago
  This is a really good point, and also depends on what the training data is - although arguably if they are using this model to power their services and it can leak personal data that might be an issue in it's own right.
- robkop 6y ago
  I have to say that I really doubt this as they've released model weights for their several last few models that have been of nearly this size (and have been trained on the same dataset). See https://github.com/google-research/text-to-text-transfer-transformer https://github.com/google-research/text-to-text-transfer-tra...
  - deleted 6y ago
    [deleted]
  - GrantZvolsky 6y ago
    I'm not sure whether I'd call 10e10 nearly the size of 10e12, but the dataset is open, so it shouldn't be an issue. https://github.com/google-research/text-to-text-transfer-transformer/blob/master/released_checkpoints.md https://github.com/google-research/text-to-text-transfer-tra...
pabs3 6y ago
Some other thoughts on what makes open source machine learning, from Debian: https://salsa.debian.org/deeplearning-team/ml-policy https://salsa.debian.org/deeplearning-team/ml-policy

amelius 6y ago

> it’s ridiculously parameter inefficient How do you know that? Perhaps with this methodology you really need those 3.12TB to reach comparable performance?

buildbot 6y ago
If you read through the paper where they present switch transformers, they train all models to the same performance, so for example t5-large, at 770M parameters[1] is just as good as this model. That’s only 1.54GB [1] https://huggingface.co/t5-large https://huggingface.co/t5-large
thesz 6y ago
I beg to disagree. [1] provides one with a whole-data-set training method (ADMM, one of such methods). Page 8 contains figure 2(b) - accuracy of training after specified amount of time. Note that ADMM start where stochastic gradient stops. [1] https://arxiv.org/pdf/1605.02026.pdf https://arxiv.org/pdf/1605.02026.pdf At [2] I tried to apply logistic regression trained using reweighted least squares algorithm on the same Higgs boson data set. I've got the same accuracy (64%) as mentioned in the ADMM paper with much less number of coefficients - basically, just the size of input vector + 1 instead of 300 such rows of coefficients and then 300x1 affine transformation. When I added squares of inputs (for the simplest approximation of polynomial regression) and used the same reweighted iterative least squares algorithm, I've got even better accuracy (66%) for double the number of coefficients. [2] https://github.com/thesz/higgs-logistic-regression https://github.com/thesz/higgs-logistic-regression There's a hypothesis [3] that SGD and ADAM are best optimizers because that everyone use and report on. Rarely if ever you get anything that differ. [3] https://parameterfree.com/2020/12/06/neural-network-maybe-evolved-to-make-adam-the-best-optimizer/ https://parameterfree.com/2020/12/06/neural-network-maybe-ev... So, answering your question of "how do you know" - researchers at Google cannot do IRLS (search provides IRLS only for logistic regression in Tensorflow), they cannot do Hessian-free optimization ([4], closed due lack of activity - notice the "we can't support RNN due to the WHILE loop" bonanza), etc. All due to the fact they have to use Tensorflow - it just does not support these things. https://github.com/tensorflow/tensorflow/issues/2682 https://github.com/tensorflow/tensorflow/issues/2682 I haven't seen anything about whole-data-set optimization from Google at all. That's why I (and only me - due to standing I take and experiments I did) conclude that they do not quite care about parameter efficiency.
- vladf 6y ago
  > they do not quite care about parameter efficiency. Google Research is pretty big, I used to think like you did but I think it's mostly b/c DeepMind just hogs all the spotlight. Check out PRESS [0] for example. [0]: https://research.google/pubs/pub46141 https://research.google/pubs/pub46141
  - thesz 6y ago
    Thank you! I skimmed over the abstract and will read the paper later, it seems interesting. But you gave me another point to support my view: PRESS uses stochastic gradient, not second-order method like IRLS.
    - vladf 6y ago
      I agree, I was really only proposing PRESS for the "parameter efficiency" part of your comment. It'd be interesting to see some modern takes on IRLS. I think generally this goes against the grain of the Cheap Gradient Principle which is why you see less of it (edit: eh, I think Fisher scoring can be cast in this light). For instance, on modern modelling problems with non-linearities and change points, it's a lot less easy to do something like IRLS in an end-to-end system, but interesting as a research direction.
      - thesz 6y ago
        I also agree with "Cheap Gradient Principle". I see it as a case of "width of two horse backs from Ancient Rome determine Shuttle buster width" (which is untrue but cool as a reference). The very SGD thing was developed because it was the only way to train something like neural network with small memory and, more importantly, in reasonable time. Multiplying of training time by N (number of parameter) meant having good result in a year, not in a day. And today we have large batch training with complex synchronization systems to speed up training even more. Which bring us closer to the whole-dataset training and, I guess, second-order optimization as well.

gradys 6y ago

Mixture of experts architectures like this are a specific design decision to increase the parameter count but use and update those parameters sparsely. Sure, that design decision doesn't fit all scenarios, but it fits some, and it has its own advantages, like faster training time.

ogrisel 6y ago
But do you see any practical scenario where you have to keep 3TB of parameters in accelerator's RAM at inference time? The accuracy would have to be significantly higher that any alternative to justify monopolizing that many hardware resources.
- gradys 6y ago
  Yep, it would definitely be difficult to justify running it in production. Accuracy would need to be higher as you said or it would need to be applicable to more tasks such that you can take other models out of production. This kind of model could be used as the teacher in a distillation setup too though. Then faster training of the teacher is actually a huge benefit since it speeds up model development iteration cycles. But even if it weren't practical to use in production in any sense, I'd argue there's value in doing the basic research of exploring design space of architectures in this way. This came out of a research team at Google. It may inspire and inform smaller, more practical architectures.
  - gwern 6y ago
    "Yep, it would definitely be difficult to justify running it in production. Accuracy would need to be higher as you said or it would need to be applicable to more tasks such that you can take other models out of production." Part of the justification is the MoE sparsity by design means only a small part of the model will be activated by a given query. Don't think of it as a single giant 1t model, think of it as 50 small models which happen to share a glue layer at the input. So at deployment, you could, for example, keep the gating-layer in RAM and only pull the necessary sub-model off disk as necessary. Or you could shard the sub-models over 51 GPUs and feed the master gating-layer+GPU lots of queries, and each query will be dispatched to a different expert+GPU pair. This could easily be competitive with running a lot of dense models in parallel trying to keep up with the same load.

summerlight 6y ago

Google's legal team would be freaking out if any of its engineers propose open sourcing model weights. This is a mine field full of legal and privacy headaches and I guess it is infeasible to audit 1.6T parameters anyway...

Closi 6y ago
This is a really good point, and also depends on what the training data is - although arguably if they are using this model to power their services and it can leak personal data that might be an issue in it's own right.
robkop 6y ago
I have to say that I really doubt this as they've released model weights for their several last few models that have been of nearly this size (and have been trained on the same dataset). See https://github.com/google-research/text-to-text-transfer-transformer https://github.com/google-research/text-to-text-transfer-tra...
- deleted 6y ago
  [deleted]
- GrantZvolsky 6y ago
  I'm not sure whether I'd call 10e10 nearly the size of 10e12, but the dataset is open, so it shouldn't be an issue. https://github.com/google-research/text-to-text-transfer-transformer/blob/master/released_checkpoints.md https://github.com/google-research/text-to-text-transfer-tra...

pabs3 6y ago

Some other thoughts on what makes open source machine learning, from Debian: https://salsa.debian.org/deeplearning-team/ml-policy https://salsa.debian.org/deeplearning-team/ml-policy