3 ms·
I'm sorry, but why would you run a model like ALBERT on a CPU in the first place? It's common knowledge not to bother running a deep model like ALBERT, BERT, o
by ZeroCool2u 7y ago
I'm sorry, but why would you run a model like ALBERT on a CPU in the first place?
It's common knowledge not to bother running a deep model like ALBERT, BERT, or XLNet, etc without at least a GPU. Training and inference with models of this size on CPU's is typically considered to be intractable.
Obviously, this paper is from Google, so they have free access to TPU's, which is arguably optimal for an arbitrary TensorFlow model in terms of performance, but if you don't have access to TPU's, the clear choice is to use GPU's instead. It won't be quite as fast, but it is sufficient and GPU's are relatively cheap and available.
- nl 7y agoI'm sorry, but why would you run a model like ALBERT on a CPU in the first place? It's pretty common to run inference on CPUs. There are lots of operational and cost reasons why this makes sense in at least some cases.
- peterjussi 7y agoI would even say: in every case, except where a GPU or TPU is necessary to achieve a certain speed. Unless there are very specific reasons for it, GPU/TPU is just unnecessarily cost inefficient.
- danieldk 7y agoIt's common knowledge not to bother running a deep model like ALBERT, BERT, or XLNet, etc without at least a GPU. Training and inference with models of this size on CPU's is typically considered to be intractable. I am not sure if this is common knowledge, and if it is, it is wrong. With read-ahead and sorting batches by length, we can easily reach 100 sentences per second on modern CPUs (8 threads) with BERT base. We use BERT in a multi-task setup, typically annotating 5 layers at the same time. This is many times faster than old HPSG parsers (which typically had exponential time complexity) and as fast as other neural methods used in a pipelined setup.