4 ms·
Abstract: > Language models only really need to use an exponential fraction of their neurons for individual inferences. As proof, we present UltraFastBERT, a B
by ndr 3y ago
Abstract:
> Language models only really need to use an exponential fraction of their neurons for individual inferences. As proof, we present UltraFastBERT, a BERT variant that uses 0.3% of its neurons during inference while performing on par with similar BERT models. UltraFastBERT selectively engages just 12 out of 4095 neurons for each layer inference. This is achieved by replacing feedforward networks with fast feedforward networks (FFFs). While no truly efficient implementation currently exists to unlock the full acceleration potential of conditional neural execution, we provide high-level CPU code achieving 78x speedup over the optimized baseline feedforward implementation, and a PyTorch implementation delivering 40x speedup over the equivalent batched feedforward inference. We publish our training code, benchmarking setup, and model weights.
Conclusions
> We present UltraFastBERT, a modified version of the
(crammed)BERT architecture that uses fast feedforward instead of feedforward networks in its intermediate layers. UltraFastBERT serves as proof that large language models only
really need to engage an exponential fraction of their parameters to perform individual inferences. UltraFastBERT-1x11,
our deepest model with the highest promise of acceleration,
uses only 0.3% of its neurons during inference and already
achieves a 78x CPU speedup over the inference time of
the corresponding feedforward layer. With a theoretical
speedup promise of 341x at the scale of BERT-base models,
we hope that our work will inspire an effort to implement
primitives for conditional neural execution as a part of device programming interfaces.
- iNic 3y agoDo I understand correctly that the difficulty of making this useful is writing code to run this idea on GPUs?
- PaulHoule 3y agoThey are getting a 78x speedup w/o hardware support which is pretty good: they think they can speed it up another 4x if they had the right hardware support. So it looks useful now with possibility to get better. So long as I've been involved with neural networks for text analysis it's seemed to me that we really should be using sparse activations because any particular document only involves a limited set of concepts. For instance a search engine for patents might be looking at a patent for adhesive tape which activates a certain set of concepts but is not going to activate concepts involved with bicycle derailleurs or public key cryptography: a sparse representation reflects this and dense representations don't.
- nolist_policy 3y agoAs far as I understood it: Forget GPUs, this thing is plenty fast on CPUs. In general, GPUs are bad at branching. The fastest way to implement it on GPUs is probably to let it calculate both sides of the branch and then only use the result of the one that was taken. Which won't be faster than a normal NN.
- falcor84 3y agoI wonder then, if each inference only uses a small part of the net, could you possibly perform multiple inferences in the same forward pass?
- lawlessone 3y ago> Forget GPUs, this thing is plenty fast on CPUs. Does this mean everyone could be running the 100+b models from ram? This opens up a lot , some models could be run very fast on small machines with this. Bundling a small model inside a game to act as part of the mind for ingame NPC's (obviously with some tuning) becomes practical with this.
- hobofan 3y agoThe bottleneck for "easy integration" into games and applications right now is as much the RAM usage as is the slowness. This would probably bring the speed to an acceptable level but you would still have to hold the whole model in RAM. That would make it a lot more feasible to run models in the cloud (triple digit RAM is a lot more abundant than VRAM), but wouldn't do that much for consumer hardware.
- nolist_policy 3y agoI wonder if the model takes similar branches while in the same context? Then you can fault in parts of the model from disk as needed.
- entropicdrifter 3y agoInteresting idea. Like texture streaming, you'd just stream in the parts of the model from disk to fill up all available RAM. If the NPC needed to think about something not cached in RAM, you'd throw up a "hmm, let me think about this" while stuff loads from disk.
- kristianp 3y ago> a PyTorch implementation delivering 40x speedup over the equivalent batched feedforward inference Does this not indicate a 40x speedup on the GPU? Edit: looking at the paper, their "Naive CUDA" implementation also shows a 117x speedup in Table 2.
- deleted 3y ago[deleted]