12 ms·
Exponentially faster language modelling
- fgfm 3y agoThis approach feels like pruning, but the speedup is considerably higher. Interestingly, I'm curious how this will play out on more recent transformer architectures though: I guess the speedup will be more important for the largest architectures, but even if we can get 2x or 10x speedup on Mistral/Zephyr, Orca 2 or OpenChat3.5, that would be a tremendous achievement!
- webmaven 3y agoI'm curious as to how applicable this approach might be for text-to-image models like Stable Diffusion.
- tokai 3y agoWhy not use the real title? Its short and precise.
- ndr 3y agoAbstract: > Language models only really need to use an exponential fraction of their neurons for individual inferences. As proof, we present UltraFastBERT, a BERT variant that uses 0.3% of its neurons during inference while performing on par with similar BERT models. UltraFastBERT selectively engages just 12 out of 4095 neurons for each layer inference. This is achieved by replacing feedforward networks with fast feedforward networks (FFFs). While no truly efficient implementation currently exists to unlock the full acceleration potential of conditional neural execution, we provide high-level CPU code achieving 78x speedup over the optimized baseline feedforward implementation, and a PyTorch implementation delivering 40x speedup over the equivalent batched feedforward inference. We publish our training code, benchmarking setup, and model weights. Conclusions > We present UltraFastBERT, a modified version of the (crammed)BERT architecture that uses fast feedforward instead of feedforward networks in its intermediate layers. UltraFastBERT serves as proof that large language models only really need to engage an exponential fraction of their parameters to perform individual inferences. UltraFastBERT-1x11, our deepest model with the highest promise of acceleration, uses only 0.3% of its neurons during inference and already achieves a 78x CPU speedup over the inference time of the corresponding feedforward layer. With a theoretical speedup promise of 341x at the scale of BERT-base models, we hope that our work will inspire an effort to implement primitives for conditional neural execution as a part of device programming interfaces.
- iNic 3y agoDo I understand correctly that the difficulty of making this useful is writing code to run this idea on GPUs?
- PaulHoule 3y agoThey are getting a 78x speedup w/o hardware support which is pretty good: they think they can speed it up another 4x if they had the right hardware support. So it looks useful now with possibility to get better. So long as I've been involved with neural networks for text analysis it's seemed to me that we really should be using sparse activations because any particular document only involves a limited set of concepts. For instance a search engine for patents might be looking at a patent for adhesive tape which activates a certain set of concepts but is not going to activate concepts involved with bicycle derailleurs or public key cryptography: a sparse representation reflects this and dense representations don't.
- nolist_policy 3y agoAs far as I understood it: Forget GPUs, this thing is plenty fast on CPUs. In general, GPUs are bad at branching. The fastest way to implement it on GPUs is probably to let it calculate both sides of the branch and then only use the result of the one that was taken. Which won't be faster than a normal NN.
- falcor84 3y agoI wonder then, if each inference only uses a small part of the net, could you possibly perform multiple inferences in the same forward pass?
- lawlessone 3y ago> Forget GPUs, this thing is plenty fast on CPUs. Does this mean everyone could be running the 100+b models from ram? This opens up a lot , some models could be run very fast on small machines with this. Bundling a small model inside a game to act as part of the mind for ingame NPC's (obviously with some tuning) becomes practical with this.
- sdrg822 3y agoCool. Important note: """ One may ask whether the conditionality introduced by the use of CMM does not make FFFs incompatible with the processes and hardware already in place for dense matrix multiplication and deep learning more broadly. In short, the answer is “No, it does not, save for some increased caching complexity." """ It's hard to beat the hardware lottery!
- algo_trader 3y agoInfact, as stated in the paper, this is bad news > We therefore leave the attention layers untouched Meaning, presumably, that the GPU memory remains the bottleneck Flops really are quite cheap by now, e.g. vision inference chip ~$2/teraflop/s !!
- deleted 3y ago[deleted]
- marcinzm 3y agoBottleneck for larger models however this would presumably allow for cheaper models at scale or on compute constrained devices (like phones).
- entropicdrifter 3y agoAnd potentially for distributing a model across several devices at inference time. You could devote a cluster of smaller/weaker machines to inference.
- sroussey 3y agoYou can do that today, the only advantage today though is being able to fix the model in memory. It’s sequential and slower due to communication costs, though batching might be faster?
- ashirviskas 3y ago>Flops really are quite cheap by now, e.g. vision inference chip ~$2/teraflop/s !! I'm really interested, can you share where you got these numbers?
- Klaster_1 3y agoWhat are the potential consequences? Does this open doors to faster edge inference or improved capabilities?
- yvdriess 3y agoBoth. Cheaper CPU-based inference, GPUs are not as competitive for sparse linear algebra. This could lead to much larger models, as you only touch a small portion of the matrix during inference. However, the training here is still dense-LA on a GPU, so you still blow up the compute cost when increasing model size.
- WithinReason 3y agoNote this doesn't speed up training
- deleted 3y ago[deleted]
- swalsh 3y agoHas anyone used SIMD instructions to try and speed up cpu inference?
- hobofan 3y agoMost inference builds on top of BLAS libraries, which in their implementation take advantage of SIMD.
- singhrac 3y agoA lot of CPU inference libraries (llama.cpp included) use as much SIMD as possible, sometimes by hand-writing loops. The one I hack on, llama.rs, uses portable_simd but specializes to your CPU at compile time. My experience has been that most CPU inference is actually not compute limited, but memory bandwidth limited, since most weights are used for a few operations per token (how quickly can you load and unload the entire 70 GB of weights into your registers?). It's not quite that bad but I found most vectorization changes didn't meaningfully change performance.
- millisecond 3y agoCould this be applied to other models like Llama2 or Mistral?
- andy99 3y agoJust from the abstract I don't see why not, it's just replacing the feed forward network that's part of all of these models with a very sparse one. The bigger problem is you seemingly have to retrain the model, so you couldn't just drop in llama2 weights from meta and have it work. Which makes it much more limiting. Something that used existing weights would be a lot more practical (like quantization for example). For BERT, I can see this being useful if you had to make a really fast embedding model. There was a discussion about a fast embedding use case not long ago https://news.ycombinator.com/item?id=37898001 https://news.ycombinator.com/item?id=37898001
- MinusGix 3y agoIt certainly could, and I wouldn't be surprised if the authors want to try it out on those. You do have issues of past improvements often not quite enhancing more powerful models nearly as much. I'd expect this to possibly not work as well, something like the bigger models ending up with more polysemantic neurons because they're given more ''incentive'' (training time, neuron count, dataset size which they're encouraged to be able to reconstruct) to extract as much possible. This might make so the method performs worse due to this intermingling. (See the transformer circuits website for that) (Though I expect there's ways to recover a good chunk of extra lost throughput/accuracy, maybe by doing extra steps to directly steer the training towards breaking apart polysemantic neurons)
- lopuhin 3y agoThere are two issues here -- for one, in big transformers, more compute is in the attention layers, while this work improves only feed-forward layers, which are more important for smaller models and smaller sequence lengths. Second, in many typical scenarios LLM inference is memory bandwidth bound, I'm not sure if it's possible to utilize their approach to reduce required memory bandwidth.
- 3y ago
- OneOffAsk 3y agoIs this similar to what iOS 17 uses for its new autocomplete?
- WithinReason 3y agoLink to previous paper: https://arxiv.org/abs/2308.14711 https://arxiv.org/abs/2308.14711 An attempt at a summary: They use a sigmoid function to make differentiable "soft" branches, and stack them to construct a binary tree, with the goal of only taking one branch at inference time (but training the whole tree) leading to log(W) instead of W inference cost. They gradually harden the branches so they become hard branches at the end of training. A branch is computed as branch(input, N), with a neural network N computing a scalar c=N(input), then using a sigmoid to do a soft branch by returning the weighted sum of the recursive call s(c)*branch(input, N_left) + (1-s(c)) * branch(input, N_right) (the two weights s(c) and 1-s(c) sum to 1). They only do "proper processing" using the leaf nodes. Then they add a new loss term that encourages hard decisions by minimising the entropy of the Bernoulli distribution, making the 2 weights converge to 0 and 1, at which point only one branch needs to be taken at inference. They also state that this hardening often happens automatically though. It's a simple idea but the loss formulation is nice, you usually want your loss terms to be a measure of information.
- WithinReason 3y agoAlso, this didn't come from OpenAI or DeepMind, or even industry. What are those guys even doing? :)
- mmaunder 3y agoMany labs doing foundational work like this and making progress don’t have the anything near the budget or compute to implement at scale. In other words they don’t have a Sam and his backers or a Zuck and his budget.
- Micoloth 3y agoThey sure as hell have no incentives to make Neural Network faster and more accessible, for starters.. (Considering they right now make more money and have more control, the less accessible and the more computation-hungry AI models are) To be fair, this approach (claims to) only speed up inference, not training, so all the GPUs are needed anyway.
- vorticalbox 3y agohugging face model https://huggingface.co/pbelcak/UltraFastBERT-1x11-long https://huggingface.co/pbelcak/UltraFastBERT-1x11-long
- lawlessone 3y ago> This model is provided only as sanity check for research purposes, it is untested and unfit for deployment. I guess this means it isn't pretrained yet? Is it still just random weights?
- mjn 3y ago"Unfit for deployment" or "not intended for deployment" is semi-standard wording for research models that are just raw language models with none of the safety/bias/offensiveness filtering that is usually desired for product applications. For example, if you deploy it as a customer-service chatbot, it might tell your customers to kill themselves, or call them racial slurs. It doesn't mean that there's anything technically wrong with the language model per se as a model of language, just that there has been no effort made to ensure it's fit to be deployed as-is for any given generative-AI use case, and the model authors would prefer you didn't do that.
- gpm 3y agoIs it even fine tuned for question answering?
- ilaksh 3y agoIs it possible to use this with something like Llama 2?
- sanxiyn 3y agoThis requires retraining from scratch, so no, you can't use Llama 2 pretrained weight. As far as I can tell you can take Llama 2 modelling code, training infrastructure, training data and apply proposed modification (they provide PyTorch nn.Module which should be drop in replacement of nn.Linear) and run the training if you have enough compute and it should work. Doesn't mean it would work, there are always lots of practical problems, but it should work in principle.
- vouaobrasil 3y agoThis is rather scary. I feel we are witnessing the evolution of language models and artificial intelligence, which seems intellectually laudable until you realize that the underlying evolutionary framework for this evolution is the global capitalistic system whose only criteria for selection in short-term monetary gain. We are creating a monster.
- caerwy 3y ago[flagged]
- hendler 3y agoRather than looking to capitalism which has provided tremendous benefits to society as well as unintended consequences you may want to update your thinking to focus on the incentives alignment problem in general. This TED talk articulates it well: https://youtu.be/WX_vN1QYgmE https://youtu.be/WX_vN1QYgmE What is after capitalism?
- deleted 3y ago[deleted]
- vouaobrasil 3y agoI absolutely disagree that reformism as in the video via incentives will be enough.
- dicroce 3y agoI think it's good to be concerned and cautious but I also think you are being a bit extreme here.
- vouaobrasil 3y agoI absolutely disagree - I believe everyone else is blind, the same way we are blind that our current lifestyles are an exercise in extreme violence on the nonhuman world.
- qntty 3y agoAccording to scientists, we only use 0.3% of our neural networks. Imagine if we could use 100%.
- deleted 3y ago[deleted]
- kleiba 3y agoNice. I know HN can sometimes be the place where humor goes to die, but I found this comment hilarious.
- zzzcsgo 3y ago[dead]
- madprofessor 3y agoWonderful.
- jredwards 3y agoThank you for taking one of my most hated memes and turning it into hilarity.
- 29athrowaway 3y agoThen you would have to eat a full refrigerator worth of food every day to survive.
- simne 3y agoReally good humor :) But in reality, sparse NN is just loose it's performance, mean loose precision and recall. Precision, means, larger probability of errors; recall - if you work with piece of information, which could consist of few, ie predicates, it will see not all predicates. To be concrete, for good trained full-scale NN, usually considered 70-90% for precision and for recall; but if use small fraction of weights, usually will got drop of performance to about 40-70%, which is good enough for many cases, considering saves on size and computations.
- baq 3y agoMix this with yesterday's matmul approximation (maddness) in HW for a casual... three orders of magnitude speed increase?
- nulld3v 3y agoCan you link the post for the matmul approximation?
- hobofan 3y agoI'm not 100% sure, but those seem mostly mutually exclusive (or redundant), with the decision tree in maddness taking on a similar function as the binary tree in FFF that decides which neurons to activate.
- terafo 3y agoThey are mostly incompatible.
- measured_step 3y agoHow would this scale for a use case like writing code? I could imagine that some inputs would require a large number of neurons. Would this architecture be able to do that if it were scaled up? I'm also curious if this model architecture would achieve the grokking of more complex concepts at scale.
- jasonjmcghee 3y agoDoes anyone understand why they are using B x H instead of B x S x H? Why is the context size and batch size represented as a single parameter?
- numeri 3y agoI would have to go back and reread the paper to be sure, but FF layers are applied position-wise, meaning independently and in parallel on all input tokens/positions. Because of that, I could imagine contexts where the sequence dimension isn't relevant, i.e., for computational complexity.
- rsolva 3y agoI find running 7B models on my 6 year old small form factor HP EliteDesk to be fast enough for casual everyday use. If this speedup can be applied generally to commonly used models, I can serve a local ChatGPT experience for both friends and family from my tiny homelab in my basement. mind blown
- FooBarWidget 3y agoI find 7B models to be too stupid. They often respond with nonsense or fail to follow instructions.
- cooper_ganglia 3y agoEven 65B models approach a level of being almost usable, but still fall short in my personal experience.
- cloudking 3y agoThis is why I'm not understanding the excitement around open source models, they pale in comparison to GPT-4 quality, so I have no use for them until we have something comparable.
- lettergram 3y agoWe utilize a LLama that frankly competes directly with gpt-4, to the point I can't tell which model we are using (we randomly switch between providers to ensure a robust backend). I think it likely depends on the use case, but many llama models can be returned and there are quite literally thousands of free versions available.
- WhrRTheBaboons 3y agoWhat is your use case?
- 3y ago
- jafitc 3y ago[flagged]
- infecto 3y agoNot a fan of what looks to be an AI summary in the comments.
- EForEndeavour 3y agoYep. And we only notice the obvious ones. And the quality of AI comments will only improve over time. Which raises the question (unanswered as far as I can tell) of what to do about it, and whether it even really matters...
- dragonwriter 3y agoIMO: If the comment is a quality comment, I don't care if the poster used AI, if its a low-quality or inappropriate comment (and a summary of the existing comments counts), then downvote or, for egregious cases, flag it.
- thomasahle 3y agoThe issue (temporarily) is that many consider a long comment = high effort = probably high quality. Now we have to change our heuristics.
- dllthomas 3y agoThat's... not been a major factor in my heuristics.
- kristianp 3y agoAlso a bit pointless when it's just summarising < 90 HN comments. One can skim that in a minute or two anyway.
- quickthrower2 3y agoIf anyone is on the ball enough to turn this into a colab or notebook that would be appreciated! Would love to see the code
- itissid 3y agoNoob Question: So is the idea to load only specific branches (and by extension log(n) order neurons), right based on the input data. Would this be something that a compiler would do using a JIT trick(because the input needs to be known to get the right branch) to issue a call to the right neurons into memory(SIMD?) to do the Feed Forward?
- itissid 3y agoFor those not familiar with Bert transformer arch. You can read a bunch of their torch benchmark code to measure speed up in just the FFF: https://github.com/pbelcak/UltraFastBERT/blob/main/benchmark_pytorch/main.py https://github.com/pbelcak/UltraFastBERT/blob/main/benchmark... some of which is from the pytorch docs here: https://pytorch.org/tutorials/intermediate/torch_compile_tutorial.html https://pytorch.org/tutorials/intermediate/torch_compile_tut..., e.g. the `timed` function and how they generate data. Also its not just the same 12 neurons, its the 12 neurons based on the previous dot product. So some kind of JIT is needed to load the right ones?
- itissid 3y agoAnother noob Question: So a 50% size reduction in BERT? let's see if I am getting these numbers right. At inference time you need a fraction of the neurons in the FF layer to do the inference based on the input data and the previous dot product. Here some quick math for BERT-Base which has 110M params according to the original paper: ---- L (Number of Layers): 12 transformer blocks. H (Hidden Size): 768 units in the hidden layers. A (Number of Attention Heads): 12 attention heads. Embedding Layers: WordPiece Embeddings: 768 (hidden size) * 30,522 (vocab size) = 23,440,896 parameters. Positional Embeddings: 768 * 512 (max sequence length) = 393,216 parameters. Segment Embeddings: 768 * 2 (number of segments) = 1,536 parameters. Total Embedding Parameters: 23,440,896 + 393,216 + 1,536 = 23,835,648 parameters. Transformer Blocks: Each transformer block has the following components: Self-Attention Layer: Each attention head has 768 / 12 = 64 units. Query (Q), Key (K), Value (V) matrices: 3 * (64 * 768) = 147,456 parameters per head. Across 12 heads: 147,456 * 12 = 1,769,472 parameters. Output layer of the attention mechanism: 768 * 768 = 589,824 parameters. Feed-Forward Network (FFN): First layer: 768 (input) * 3,072 (intermediate size) = 2,359,296 parameters. Second layer: 3,072 * 768 = 2,359,296 parameters. Total FFN parameters per block: 2,359,296 + 2,359,296 = 4,718,592 parameters. -----------------> *This is the number to keep in mind.* Total Parameters per Block: 1,769,472 (self-attention) + 589,824 (output) + 4,718,592 (FFN) = 7,077,888 parameters. Total for 12 Blocks: 7,077,888 * 12 = 84,934,656 parameters. Layer Norm and Other Parameters: Each transformer block also includes layer normalization and other small components, which add a relatively small number of parameters. Total Parameters: Embeddings: 23,835,648 Transformer Blocks: 84,934,656 Layer Norm and Others: A small number, completing the total to around 110 million. -------------------------------------- 4.718M FF Params per block * 12 ~ 56.6 Million/110M Params which is a staggering ~50% reduction in size at inference time if you use 0.3% of the FF neurons for FFF??
- matmulbro 3y agoTimewaster All valuable AI research is secret now, they just churn out papers to waste time
- dartos 3y agoAFAIK GPU core are quite slow with branching logic. I wonder if the conditional in this would hurt performance at scale
- OliverPersson 3y ago[dead]