6 ms·

This is nice work. Summarized: by untying word piece embedding sizes from the hidden layer size + sharing parameters between layers, the number of parameters in

This is nice work. Summarized: by untying word piece embedding sizes from the hidden layer size + sharing parameters between layers, the number of parameters in the model are drastically reduced. They use this headroom to make the hidden layer sizes larger and given that layers share parameters, they can also use more layers without increasing model size. Another interesting finding is that this speeds up training due to the smaller number of parameters. However, my worry is that those of use that do not readily have access to TPUs will get even slower models when using CPUs for prediction due to the additional and wider hidden layers. (Of course, one could use ALBERT base, which still have 12 layers and a hidden layer size of 768, at a small loss.) Did anyone measure the CPU prediction performance of ALBERT models compared to BERT? Edit: I guess one solution would be to use a pretrained ALBERT and finetune to get the initial model and then use model distillation to get a smaller, faster model.

Nimitz14 7y ago
Yeah, trading off embedding dim vs hidden layer dim is a common trick that effectively means trading size on disk (and in memory) for inference speed. It's obvious the model will need to be more powerful if the embedding carry less information. Still cool they got the size down by 90%.
mapleshamrock 7y ago
On CPU, assuming inference is compute bound rather than bandwidth bound, the compute time will scale quadratically with the size of the FC layers (which account for almost all compute time in these networks). So if the hidden size was 768 in BERT-Base, and 4096 in ALBERT, inference will approximately be 28.4x slower... yikes.
zuzun 7y ago
Is there a speed-up? In their paper in table 3, once you compare each ALBERT model with the smaller BERT model, you're looking at similar accuracies and longer training times.
- nl 7y ago
  They are comparing the speed to execute training to 125K steps, not speed to a given accuracy. In section 4.8 they compare accuracy at the same amount of training time for the biggest of each model and show that ALBERT is substantially better.
samcodes 7y ago
> I guess one solution would be to use a pretrained ALBERT and finetune to get the initial model and then use model distillation to get a smaller, faster model. IME this is the way to go; take an ensemble of big, accurate models, then distill them down to the smallest model you can get away with. There are really good tricks in this paper https://www.aclweb.org/anthology/D19-5632/ https://www.aclweb.org/anthology/D19-5632/
ZeroCool2u 7y ago
I'm sorry, but why would you run a model like ALBERT on a CPU in the first place? It's common knowledge not to bother running a deep model like ALBERT, BERT, or XLNet, etc without at least a GPU. Training and inference with models of this size on CPU's is typically considered to be intractable. Obviously, this paper is from Google, so they have free access to TPU's, which is arguably optimal for an arbitrary TensorFlow model in terms of performance, but if you don't have access to TPU's, the clear choice is to use GPU's instead. It won't be quite as fast, but it is sufficient and GPU's are relatively cheap and available.
- nl 7y ago
  I'm sorry, but why would you run a model like ALBERT on a CPU in the first place? It's pretty common to run inference on CPUs. There are lots of operational and cost reasons why this makes sense in at least some cases.
  - peterjussi 7y ago
    I would even say: in every case, except where a GPU or TPU is necessary to achieve a certain speed. Unless there are very specific reasons for it, GPU/TPU is just unnecessarily cost inefficient.
- danieldk 7y ago
  It's common knowledge not to bother running a deep model like ALBERT, BERT, or XLNet, etc without at least a GPU. Training and inference with models of this size on CPU's is typically considered to be intractable. I am not sure if this is common knowledge, and if it is, it is wrong. With read-ahead and sorting batches by length, we can easily reach 100 sentences per second on modern CPUs (8 threads) with BERT base. We use BERT in a multi-task setup, typically annotating 5 layers at the same time. This is many times faster than old HPSG parsers (which typically had exponential time complexity) and as fast as other neural methods used in a pipelined setup.
nl 7y ago
I've been using ALBERT (the HuggingFace port) for a few weeks. It works fine on GPUs, and it isn't noticeably slower for inference than other large models on CPUs. It's worth noting that TPUs are available for free on Google Colab.
- danieldk 7y ago
  It's worth noting that TPUs are available for free on Google Colab. Yes, and you can also get a research grant, which gives you several TPUs for a month. But that does not mean that you can easily deploy TPUs in your own infrastructure, unless you use Google Cloud and suck up the costs (which may not be possible in academia).
nl 7y ago
Summarized: by untying word piece embedding sizes from the hidden layer size + sharing parameters between layers, the number of parameters in the model are drastically reduced. They use this headroom to make the hidden layer sizes larger and given that layers share parameters, they can also use more layers without increasing model size. This summary is wrong. It's true that they use parameter sharing to reduce the number of parameters. But for any given "class" of model (base/large/xlarge) ALBERT has the same size hidden layer and the same number of layers. If you try to compare by model size (measured by number of parameters) then ALBERT xxlarge (235M parameters) has less layers than BERT large (334M parameters) - 12 vs 24 - a larger hidden layer (4096 vs 1024) and smaller embedding size (128 vs 1024).
- nl 7y ago
  they can also use more layers without increasing model size. Additionally, in section 4.9 they compare more layers and find "The difference between 12-layer and 24-layerALBERT-xxlarge configurations in terms of downstream accuracy is negligible, with the Avg score being the same. We conclude that, when sharing all cross-layer parameters (ALBERT-style), there is no need for models deeper than a 12-layer configuration"
- danieldk 7y ago
  The summary is only wrong if you want to compare the categories. I am not sure why the category names are important, except for identifying individual models.
halflings 7y ago
> Edit: I guess one solution would be to use a pretrained ALBERT and finetune to get the initial model and then use model distillation to get a smaller, faster model. Huggingface did just that: https://medium.com/huggingface/distilbert-8cf3380435b5 https://medium.com/huggingface/distilbert-8cf3380435b5

Nimitz14 7y ago

Yeah, trading off embedding dim vs hidden layer dim is a common trick that effectively means trading size on disk (and in memory) for inference speed. It's obvious the model will need to be more powerful if the embedding carry less information. Still cool they got the size down by 90%.

mapleshamrock 7y ago

On CPU, assuming inference is compute bound rather than bandwidth bound, the compute time will scale quadratically with the size of the FC layers (which account for almost all compute time in these networks). So if the hidden size was 768 in BERT-Base, and 4096 in ALBERT, inference will approximately be 28.4x slower... yikes.

zuzun 7y ago

Is there a speed-up? In their paper in table 3, once you compare each ALBERT model with the smaller BERT model, you're looking at similar accuracies and longer training times.

nl 7y ago
They are comparing the speed to execute training to 125K steps, not speed to a given accuracy. In section 4.8 they compare accuracy at the same amount of training time for the biggest of each model and show that ALBERT is substantially better.

samcodes 7y ago

> I guess one solution would be to use a pretrained ALBERT and finetune to get the initial model and then use model distillation to get a smaller, faster model. IME this is the way to go; take an ensemble of big, accurate models, then distill them down to the smallest model you can get away with. There are really good tricks in this paper https://www.aclweb.org/anthology/D19-5632/ https://www.aclweb.org/anthology/D19-5632/

ZeroCool2u 7y ago

I'm sorry, but why would you run a model like ALBERT on a CPU in the first place? It's common knowledge not to bother running a deep model like ALBERT, BERT, or XLNet, etc without at least a GPU. Training and inference with models of this size on CPU's is typically considered to be intractable. Obviously, this paper is from Google, so they have free access to TPU's, which is arguably optimal for an arbitrary TensorFlow model in terms of performance, but if you don't have access to TPU's, the clear choice is to use GPU's instead. It won't be quite as fast, but it is sufficient and GPU's are relatively cheap and available.

nl 7y ago
I'm sorry, but why would you run a model like ALBERT on a CPU in the first place? It's pretty common to run inference on CPUs. There are lots of operational and cost reasons why this makes sense in at least some cases.
- peterjussi 7y ago
  I would even say: in every case, except where a GPU or TPU is necessary to achieve a certain speed. Unless there are very specific reasons for it, GPU/TPU is just unnecessarily cost inefficient.
danieldk 7y ago
It's common knowledge not to bother running a deep model like ALBERT, BERT, or XLNet, etc without at least a GPU. Training and inference with models of this size on CPU's is typically considered to be intractable. I am not sure if this is common knowledge, and if it is, it is wrong. With read-ahead and sorting batches by length, we can easily reach 100 sentences per second on modern CPUs (8 threads) with BERT base. We use BERT in a multi-task setup, typically annotating 5 layers at the same time. This is many times faster than old HPSG parsers (which typically had exponential time complexity) and as fast as other neural methods used in a pipelined setup.

nl 7y ago

I've been using ALBERT (the HuggingFace port) for a few weeks. It works fine on GPUs, and it isn't noticeably slower for inference than other large models on CPUs. It's worth noting that TPUs are available for free on Google Colab.

danieldk 7y ago
It's worth noting that TPUs are available for free on Google Colab. Yes, and you can also get a research grant, which gives you several TPUs for a month. But that does not mean that you can easily deploy TPUs in your own infrastructure, unless you use Google Cloud and suck up the costs (which may not be possible in academia).

nl 7y ago

Summarized: by untying word piece embedding sizes from the hidden layer size + sharing parameters between layers, the number of parameters in the model are drastically reduced. They use this headroom to make the hidden layer sizes larger and given that layers share parameters, they can also use more layers without increasing model size. This summary is wrong. It's true that they use parameter sharing to reduce the number of parameters. But for any given "class" of model (base/large/xlarge) ALBERT has the same size hidden layer and the same number of layers. If you try to compare by model size (measured by number of parameters) then ALBERT xxlarge (235M parameters) has less layers than BERT large (334M parameters) - 12 vs 24 - a larger hidden layer (4096 vs 1024) and smaller embedding size (128 vs 1024).

nl 7y ago
they can also use more layers without increasing model size. Additionally, in section 4.9 they compare more layers and find "The difference between 12-layer and 24-layerALBERT-xxlarge configurations in terms of downstream accuracy is negligible, with the Avg score being the same. We conclude that, when sharing all cross-layer parameters (ALBERT-style), there is no need for models deeper than a 12-layer configuration"
danieldk 7y ago
The summary is only wrong if you want to compare the categories. I am not sure why the category names are important, except for identifying individual models.

halflings 7y ago

> Edit: I guess one solution would be to use a pretrained ALBERT and finetune to get the initial model and then use model distillation to get a smaller, faster model. Huggingface did just that: https://medium.com/huggingface/distilbert-8cf3380435b5 https://medium.com/huggingface/distilbert-8cf3380435b5