7 ms·

The model size in deep learning is one of the most problematic aspects of the tech. I'm surprised we haven't seen more research on modular models. Random fore

The model size in deep learning is one of the most problematic aspects of the tech. I'm surprised we haven't seen more research on modular models. Random forests and boosting both work by building a collection of weak models, and combining their results to produce final predictions. It seems like building an ensemble of some sort of unsupervised learners, then training a smaller supervised network on that ensemble could produce good results, while also being more reusable and easier to train.

belval 6y ago
This is true for classical ML, but unfortunately it does not hold very well for hard problems. The reason is that GPT-3 at al. are meant as multipurpose pretrained models their size is their strength in that they can model highly complex datasets. In my opinion the right approach to downsizing is to train at full size and then either distill or prune the model to keep the parts that are relevant to your problem.
- h-jones 6y ago
  This helps lessen issues like inference speed / requirements but it doesn't address the environmental impact of training.
  - klipt 6y ago
    The environmental impact of training deep models is a tiny fraction of the impact of eg mining bitcoin.
    - ma2rten 6y ago
      Also it's tiny compared to the server cost of big tech companies like facebook and google.
  - danieldk 6y ago
    That's only for pretraining a model. Very few groups pretrain models, since it is so expensive in terms of GPU time. Finetuning a model for a specific task typically only takes a few hours. E.g. I regularly train multitask syntax models (POS tagging, lemmatization, morphological tagging, dependency relations, topological fields), which takes just a few hours on a consumer-level RTX 2060 super. Unfortuntaly, distillation of smaller models can take a fair bit of time. However, there is a lot of recent work to make distillation more efficient, e.g. by not just training on the label distributions of of a teacher model, but by also learning to emulate the teacher's attention, hidden layer outputs, etc.
    - ericd 6y ago
      Is there one model that you use more frequently than others as a base for these disparate fine tuning tasks? Basically, are there any that are particularly flexible?
      - ma2rten 6y ago
        In general, BERT would be the most common one. RoBERTa is the same model but trained for longer, which turns out to work better. T5 is a larger model, which works better on many tasks but is more expensive.
        ericd 6y ago
        Thanks for the summary! I'm familiar with BERT, but less so the different variants, so that's quite helpful. I'll take a look at how RoBERTa works.
      - danieldk 6y ago
        So far, of the models that run on GPUs with 8-16GiB VRAM XLM-RoBERTa has been the best for these specific tasks. It worked better than the multi-lingual BERT model and language-specific BERT models by quite a wide margin.
        ericd 6y ago
  - dnautics 6y ago
    the environmental impact of training is, generally considered to be a one-time cost.
  - ma2rten 6y ago
    The pretraining is shared between models, inference is generally such more expensive.
nestorD 6y ago
The Rocket paper [0] for time serie classification works along those lines. They use a linear classifier on top of an ensemble of random convolutions. [0]: https://arxiv.org/abs/1910.13051 https://arxiv.org/abs/1910.13051
double_descent 6y ago
Unfortunately, this is not what the science says in 2020. See for example the graph at https://youtu.be/OBCciGnOJVs?t=1250 https://youtu.be/OBCciGnOJVs?t=1250. More parameters is inherently better, with the knowledge we have now.
emcq 6y ago
You can make a model of any size with deep learning. If your concerns are about over fitting there are lots of regularization techniques used in practice like dropout, weight decay, and data augmentation. There's nothing preventing you from sharing weights across layers, and would be interesting to see some research about that.
- danieldk 6y ago
  There's nothing preventing you from sharing weights across layers, and would be interesting to see some research about that. E.g. the ALBERT model does that: https://arxiv.org/abs/1909.11942 https://arxiv.org/abs/1909.11942 I have done model distillation of XLM-RoBERTa into ALBERT-based models with multiple layer groups and for the tasks that I was working on (syntax) it works really well. E.g. we have gone from a finetuned ~1000MiB XLM-R base model to a 74MiB ALBERT-based model with barely any loss in accuracy.
- auraham 6y ago
  You can also train a neural network with a single shared weight parameter: https://weightagnostic.github.io/ https://weightagnostic.github.io/

belval 6y ago

This is true for classical ML, but unfortunately it does not hold very well for hard problems. The reason is that GPT-3 at al. are meant as multipurpose pretrained models their size is their strength in that they can model highly complex datasets. In my opinion the right approach to downsizing is to train at full size and then either distill or prune the model to keep the parts that are relevant to your problem.

h-jones 6y ago
This helps lessen issues like inference speed / requirements but it doesn't address the environmental impact of training.
- klipt 6y ago
  The environmental impact of training deep models is a tiny fraction of the impact of eg mining bitcoin.
  - ma2rten 6y ago
    Also it's tiny compared to the server cost of big tech companies like facebook and google.
- danieldk 6y ago
  That's only for pretraining a model. Very few groups pretrain models, since it is so expensive in terms of GPU time. Finetuning a model for a specific task typically only takes a few hours. E.g. I regularly train multitask syntax models (POS tagging, lemmatization, morphological tagging, dependency relations, topological fields), which takes just a few hours on a consumer-level RTX 2060 super. Unfortuntaly, distillation of smaller models can take a fair bit of time. However, there is a lot of recent work to make distillation more efficient, e.g. by not just training on the label distributions of of a teacher model, but by also learning to emulate the teacher's attention, hidden layer outputs, etc.
  - ericd 6y ago
    Is there one model that you use more frequently than others as a base for these disparate fine tuning tasks? Basically, are there any that are particularly flexible?
    - ma2rten 6y ago
      In general, BERT would be the most common one. RoBERTa is the same model but trained for longer, which turns out to work better. T5 is a larger model, which works better on many tasks but is more expensive.
      - ericd 6y ago
        Thanks for the summary! I'm familiar with BERT, but less so the different variants, so that's quite helpful. I'll take a look at how RoBERTa works.
    - danieldk 6y ago
      So far, of the models that run on GPUs with 8-16GiB VRAM XLM-RoBERTa has been the best for these specific tasks. It worked better than the multi-lingual BERT model and language-specific BERT models by quite a wide margin.
      - ericd 6y ago
- dnautics 6y ago
  the environmental impact of training is, generally considered to be a one-time cost.
- ma2rten 6y ago
  The pretraining is shared between models, inference is generally such more expensive.

nestorD 6y ago

The Rocket paper [0] for time serie classification works along those lines. They use a linear classifier on top of an ensemble of random convolutions. [0]: https://arxiv.org/abs/1910.13051 https://arxiv.org/abs/1910.13051

double_descent 6y ago

Unfortunately, this is not what the science says in 2020. See for example the graph at https://youtu.be/OBCciGnOJVs?t=1250 https://youtu.be/OBCciGnOJVs?t=1250. More parameters is inherently better, with the knowledge we have now.

emcq 6y ago

You can make a model of any size with deep learning. If your concerns are about over fitting there are lots of regularization techniques used in practice like dropout, weight decay, and data augmentation. There's nothing preventing you from sharing weights across layers, and would be interesting to see some research about that.

danieldk 6y ago
There's nothing preventing you from sharing weights across layers, and would be interesting to see some research about that. E.g. the ALBERT model does that: https://arxiv.org/abs/1909.11942 https://arxiv.org/abs/1909.11942 I have done model distillation of XLM-RoBERTa into ALBERT-based models with multiple layer groups and for the tasks that I was working on (syntax) it works really well. E.g. we have gone from a finetuned ~1000MiB XLM-R base model to a 74MiB ALBERT-based model with barely any loss in accuracy.
auraham 6y ago
You can also train a neural network with a single shared weight parameter: https://weightagnostic.github.io/ https://weightagnostic.github.io/