5 ms·
Hardware Acceleration of LLMs: A comprehensive survey and comparison
- deleted 2y ago[deleted]
- moffkalast 2y agoIn-memory sounds like the way to go not just in terms of performance, but in that it makes no sense to build an ASIC or program an FPGA for a model that will most likely be obsolete in a few months at best if you're lucky.
- limit499karma 2y agohttps://arxiv.org/pdf/2402.09709 https://arxiv.org/pdf/2402.09709
- throwawaymaths 2y agoYeah, it's not like foundational models ever share compute kernels, or anything.
- moffkalast 2y agoEh, there's so much shenanigans these days even in fine tuning, people adding empty layers and pruning and whatnot, it's unlikely that even models based on the same one will have the same architecture. For new foundation models it's even worse, because there's some fancy experiment every time and the llama.cpp team needs two weeks to figure out how to implement it so the model can even run.
- throwawaymaths 2y agoYeah so you might need to implement a weird activation or positional enciding in software or something but I suspect 90% will probably be the same... If it's just layer count or skipped matrices I assume it should be possible to write an orchestrator that could run most of those models... Unless we move to mamba or something
- yjftsjthsd-h 2y agoI'm unfamiliar; in this context is "in-memory" specialized hardware that combines CPU+RAM?
- limit499karma 2y agoIn-mem (generally) means no (re)loading of data from a storage device.
- yjftsjthsd-h 2y agoSure, but I don't think that makes sense here; when I run an LLM on CPU, I load to memory and run it, when I run on GPU I load the model into the GPU's memory and run it, and I don't have anything like that much money to burn but I imagine if I used an FPGA then I would load the model into its memory and then run it from there. So the fact that they're saying "in-memory" in contrast to ex. GPU makes me think that they're talking about something different here.
- mmoskal 2y agoIt's a different kind of memory chip that also does some computation. See https://en.m.wikipedia.org/wiki/In-memory_processing https://en.m.wikipedia.org/wiki/In-memory_processing
- adrian_b 2y agoWhile this has been proposed repeatedly for many decades, I doubt that it will ever become useful. Combining memory with computation seems good in theory, but it is difficult to do in practice. The fabrication technologies for DRAM and for computational devices are very different. If you implement computational units on a DRAM chip, they will have a much worse performance than those implemented with a dedicated fabrication process, so for instance their performance per watt and per occupied area will be worse, leading to higher costs than for using separate memories and computational devices. The higher cost might be acceptable in certain cases if a much higher performance is obtained. However it is unavoidable that unlike with a CPU/GPU/FPGA, where you can easily reprogram the device to implement a completely different algorithm, a device with in-memory computation would be much less flexible, so it either will implement extremely simple operations, like adding to memory or multiplying the memory, which would not increase much the performance due to communication overheads, or it would implement some more complex operations, which might implement some ML/AI algorithm that is popular for the moment, but which would be hard to use to implement better algorithms when such algorithms are discovered.
- deleted 2y ago[deleted]
- deleted 2y ago[deleted]
- deleted 2y ago[deleted]
- deleted 2y ago[deleted]
- jumploops 2y agoCurious if anyone is making AccelTran ASICs?
- refibrillator 2y agoThis paper is light on background so I’ll offer some additional context: As early as the 90s it was observed that CPU speed (FLOPs) was improving faster than memory bandwidth. In 1995 William Wulf and Sally Mckee predicted this divergence would lead to a “memory wall”, where most computations would be bottlenecked by data access rather than arithmetic operations. Over the past 20 years peak server hardware FLOPS has been scaling at 3x every 2 years, outpacing the growth of DRAM and interconnect bandwidth, which have only scaled at 1.6 and 1.4 times every 2 years, respectively. Thus for training and inference of LLMs, the performance bottleneck is increasingly shifting toward memory bandwidth. Particularly for autoregressive Transformer decoder models, it can be the dominant bottleneck. This is driving the need for new tech like Compute-in-memory (CIM), also known as processing-in-memory (PIM). Hardware in which operations are performed directly on the data in memory, rather than transferring data to CPU registers first. Thereby improving latency and power consumption, and possibly sidestepping the great “memory wall”. Notably to compare ASIC and FPGA hardware across varying semiconductor process sizes, the paper uses a fitted polynomial to extrapolate to a common denominator of 16nm: > Based on the article by Aaron Stillmaker and B.Baas titled ”Scaling equations for the accurate prediction of CMOS device performance from 180 nm to 7nm,” we extrapolated the performance and the energy efficiency on a 16nm technology to make a fair comparison But extrapolation for CIM/PIM is not done because they claim: > As the in-memory accelerators the performance is not based only on the process technology, the extrapolation is performed only on the FPGA and ASIC accelerators where the process technology affects significantly the performance of the systems. Which strikes me as an odd claim at face value, but perhaps others here could offer further insight on that decision. Links below for further reading. https://arxiv.org/abs/2403.14123 https://arxiv.org/abs/2403.14123 https://en.m.wikipedia.org/wiki/In-memory_processing https://en.m.wikipedia.org/wiki/In-memory_processing http://vcl.ece.ucdavis.edu/pubs/2017.02.VLSIintegration.TechScale/ http://vcl.ece.ucdavis.edu/pubs/2017.02.VLSIintegration.Tech...
- bilsbie 2y agoThanks for the background. Whatever happened to memristors and the promise of memory living alongside cpu?
- deleted 2y ago
- next_xibalba 2y agoCould a FPGA + ASICs + in-mem hybrid architecture have any role to play in scaling/flexibility? Each one has its own benefits (e.g., FPGAs for flexibility, ASICs for performance, in-memory for energy efficiency), so could a hybrid approach integrating each to juice LLM perf even further?
- synergy20 2y agonormally it's FPGA + memory first, when it hits a sweet spot in the market with volume, you then turn FPGA to ASIC for performance and cost saving. For big companies they will go ASIC directly.
- synergy20 2y agoMemory move is the bottleneck these days, thus the expensive HBM, Nvidia's design is also memory-optimized since it's the true bottleneck chip wise and system wise.
- DrNosferatu 2y agoWhy haven’t all GPUs migrated to HBMx? You seldom see it.
- iml7 2y agoexpensive
- deepnotderp 2y agoIt’s expensive, not needed for most consumer workloads and ironically, is actually often worse for latency for many patterns, even though it’s much higher bandwidth
- ska 2y agowhy is that ironic? Many ways to increase bandwidth come at the cost of latency...
- koolala 2y agoI'd love to watch a LLM run in WebGL where everything is Textures. Would be neat to visually see the difference in architectures.
- vanviegen 2y agoWouldn't that be just like watching static noise?
- archerx 2y agoI think some patterns would appear.
- Twirrim 2y agoWhy would you expect patterns to appear?
- koolala 2y agoEverything is a different pattern. Noise isn't random - we just call 1 very specific unique pattern of it 'random'.
- archerx 2y agoI was really into Convolutional Neural Networks back in the day and when you visualize the layers you’d get interesting patterns. I’m guessing the same would happen with LLMs but the patterns will be a lot more abstract but still some what recognizable. Here’s an example of visualizing CNN layers; https://www.researchgate.net/figure/Visualization-of-example-features-of-eight-layers-of-a-deep-convolutional-neural_fig2_279068412 https://www.researchgate.net/figure/Visualization-of-example...
- iml7 2y agoDoesn't Google have a tool that allows you to check the activation status of the matrix? Gemma scope
- smcleod 2y agoIs there a "nice" way to read content on Arxiv? Every time I land on that site I'm so confused / lost in it's interface (or lack there of) I usually end up leaving without getting to the content.
- buildbot 2y agoIt’s a paper pre-publishing website, so everything is formatted in PDFs by default. They recently added html: https://arxiv.org/html/2409.03384v1 https://arxiv.org/html/2409.03384v1 That’s the best way per paper. There’s a few arxiv frontends, like https://arxiv-sanity-lite.com/ https://arxiv-sanity-lite.com/
- Noumenon72 2y agoSame here -- I visited this link earlier today and thought "Oh, it's just an abstract, I'm out". I've read Arxiv papers before but the UI just doesn't look like it offers any content.
- johndough 2y agoClick on "View PDF" or "HTML (experimental)" on the top right to get to the content.
- TheMysteryTrain 2y agoI've always had the same issue as OP - it's never bothered me much because I'm rarely in the mood to read something so dense. But I find it quite interesting that I've managed to completely miss the big obvious blue buttons every time, I just immediately scan down to the first paragraph. The cynic in me guesses it's because I'm so used to extraneous content taking up space that I instinctively skim past, but maybe that's too pessimistic & there's another UX/psychological reason for it.
- beefnugs 2y agoIts the idea of the gateway: we click on a link and expect it to be what we are looking for. Having some "summary" or paywall or anything between us and what we thought we are getting is , QUICK CLOSE THE STUPID WEBSITE as fast as possible triggering.
- fulafel 2y agoRelated: https://arxiv.org/pdf/2406.08413 https://arxiv.org/pdf/2406.08413 Memory Is All You Need: An Overview of Compute-in-Memory Architectures for Accelerating Large Language Model Inference
- smusamashah 2y agoThere was a paper about LLM running on same power as a light bulb. https://arxiv.org/abs/2406.02528 https://arxiv.org/abs/2406.02528 https://news.ucsc.edu/2024/06/matmul-free-llm.html https://news.ucsc.edu/2024/06/matmul-free-llm.html
- transpute 2y agoClaims 90% memory reduction with OSS code for replication on standard GPUs, https://github.com/ridgerchu/matmulfreellm https://github.com/ridgerchu/matmulfreellm > ..avoid using matrix multiplication using two main techniques. The first is a method to force all the numbers within the matrices to be ternary, meaning they can take one of three values: negative one, zero, or positive one. This allows the computation to be reduced to summing numbers rather than multiplying.. Instead of multiplying every single number in one matrix with every single number in the other matrix.. the matrices are overlaid and only the most important operations are performed.. researchers were able to maintain the performance of the neural network by introducing time-based computation in the training of the model. This enables the network to have a “memory” of the important information it processes, enhancing performance. > ... On standard GPUs.. network achieved about 10 times less memory consumption and operated about 25 percent faster.. could provide a path forward to enabling the algorithms to run at full capacity on devices with smaller memory like smartphones.. Over three weeks, the team created a [FPGA] prototype of their hardware.. surpasses human-readable throughput.. on just 13 watts of power. Using GPUs would require about 700 watts of power, meaning that the custom hardware achieved more than 50 times the efficiency of GPUs.
- mikewarot 2y agoI've always been partial to systolic arrays. I iterated through a bunch of options over the past few decades, and settled upon what I think is the optimal solution, a cartesian grid of cells. Each cell would have 4 input bits, 1 each from the neighbors, and 4 output bits, again, one to each neighbor. In the middle would be 64 bits of shift register from a long scan chain, the output of which goes to 4 16:1 multiplexers, and 4 bits of latch. Through the magic of graph coloring, a checkerboard pattern would be used to clock all of the cells to allow data to flow in any direction without preference, and without race conditions. All of the inputs to any given cell would be stable. This allows the flexibility of an FPGA, without the need to worry about timing issues or race conditions, glitches, etc. This also keeps all the lines short, so everything is local and fast/low power. What it doesn't do is be efficient with gates, nor give the fastest path for logic. Every single operation happens effectively in parallel. All computation is pipelined. I've had this idea since about 1982... I really wish someone would pick it up and run with it. I call it the BitGrid.
- covoeus 2y agoSounds similar to the GA144 chip from the inventor of Forth
- mikewarot 2y agoThe GA144 chip is a 12x12 grid of RISC Stack oriented CPUs with networking hardware between them. Each can be doing something different. The BitGrid is bit oriented, not bytes or words, which makes if flexible for doing FP8, FP16, or whatever other type of calculation is required. Since everything is clocked at the same rate/step, data flows continuously, without need for queue/dequeue, etc. Ideally, a chip could be made that's a billion cells. The structure is simple, and uniform. An emulator exists, and that would return the exact same answer (albeit much slower) than a billion cell chip. You could divide up simulation among a network of CPUs to speed it up.
- ericlewis 2y agoThis reminds of a TPU.
- DrNosferatu 2y agoThe values (namely the FPGAs) should have been normalized also by price.
- fsndz 2y agothis explains the success of Groq's ASIC-powered LPUs. LLM inference on Groq Cloud is blazingly fast. Also, the reduction in energy consumption is nice.