23 ms·
There are two findings I find shocking in this work: * In existing LLMs, we can replace all parameter floating-point values representing real numbers with tern
by cs702 3y ago
There are two findings I find shocking in this work:
* In existing LLMs, we can replace all parameter floating-point values representing real numbers with ternary values representing (-1, 0, 1).
* In matrix multiplications (e.g., weights by vectors), we can replace elementwise products in each dot product (a₁b₁ + a₂b₂ ...) with elementwise additions (a₁+b₁ + a₂+b₂ ...), in which signs depend on each value. See the paper for exact details.
On existing hardware, the gains in compute and memory efficiency are significant, without performance degradation (as tested by the authors).
If the proposed methods are implemented in hardware, we will see even greater gains in compute and memory efficiency.
Wow.
- rhaps0dy 3y agoI think you need more evidence than this paper (which is very short and light on actual numbers) to be this shocked. For example, most of the plots in the paper are actually of throughput, memory, etc. all performance characteristics that are better on the ternary version. Which, of course. The only thing that contains perplexities are Table 1 and 2. There, they compare "BitNet b1.58 to our reproduced FP16 LLaMA LLM in various sizes" on the RedPajama data set. The first thing to note is the perplexities are very high: they're all at least ~9.9, which compared for example with quantized Llama on wikitext-2 which is 6.15 (https://www.xzh.me/2023/09/a-perplexity-benchmark-of-llamacpp.html https://www.xzh.me/2023/09/a-perplexity-benchmark-of-llamacp...). Maybe RedPajama is a lot harder than wikitext-2, but that's a big gap. I think probably their benchmark (their "reproduced FP16 LLaMA LLM") is just not very good. They didn't invest much in training their baseline and so they handily beat it.
- cs702 3y agoThank you. I think the paper as it is provides enough evidence to support the claims. If I understand the authors correctly, they trained the compared models on only 100B tokens, all drawn from RedPajama, to make the comparisons apples-to-apples. That's sensible. It allows for easier replication of the results. Otherwise, I agree with you that more extensive testing, after more extensive pretraining, is still necessary.
- craq 3y agoAnd that's true, but why do they limit it to 100B tokens? And why not provide the loss curves in the end to show that both models have converged? What's not proven to me, in this paper, is the ability of the model to scale and generalize to bigger datasets. It's easy to see how a model of sufficient size can overcome the quantization bottleneck, when trained on such a small dataset. Which is perhaps why smaller variations failed.
- lr1970 3y agoAuthors reported perplexity only for small up to 3B weights models. On the other hand, they reported throughput for 70B model, but not its performance (perplexity, end-to-end tasks). Very unfortunate omission. Overall, the paper is rather poorly written.
- cs702 3y agoIf I understand the authors correctly, they trained the compared models on only 100B tokens, all drawn from RedPajama, to make the comparisons apples-to-apples. That's sensible. It allows for easier replication of the results. Otherwise, I agree with you that more extensive testing, after more extensive pretraining, at larger model sizes, is still necessary.
- beagle3 3y agoI haven’t been keeping tabs, but this seems very much like RIP / Achilioptas version of the Johnson Lindenstrauss lemma. Perhaps the rest of the JL lemma promise applies as well - compressing the number of parameters by a few orders of magnitude as well.
- jandrese 3y agoIt seems like the AI space is slowly coming back around to the old Thinking Machines CM-1 architecture. It's not too often in computing where you see ideas a full 40 years ahead of their time make it into production.
- giantrobot 3y agoIIUC the main issue with the CM-1 architecture was feeding the processor cluster with data. That required a heftier front end system than was practical/affordable at the time. With modern CPUs and memory subsystems the GPUs can be saturated pretty easily. So going back to huge clusters of super narrow cores won't starve them for work.
- theendisney 3y agoMemristors any moment now
- TMWNN 3y agoI'm holding out for Josephson junctions
- abeppu 3y ago> On existing hardware, the gains in compute and memory efficiency are significant, without performance degradation (as tested by the authors). Did they actually show absence of performance degradation? I think it's conspicuous that Table 1 and Table 2 in the paper, which show perplexity and accuracy results respectively, are only for small model sizes, whereas Figure 2, Figure 3 (latency, memory, energy consumption) and Table 3 (throughput) all show larger model sizes. So it seems like they had every opportunity to show the perplexity/accuracy comparisons at the larger model sizes, but did not include them.
- cs702 3y agoOthers have already made the same point in this thread. See my response here: https://news.ycombinator.com/item?id=39539508 https://news.ycombinator.com/item?id=39539508
- vessenes 3y agoI'd be VERRY cautious about being excited here. My priors are like this: 1. Initial training of a neural network moves all weights around a large amount at first. 2. Later training of the network adjusts them a small amount. 3. An undertrained network will therefore look a lot like figuring out "positive, negative, or 0?" for each node during early training. If all these things are true, then 1. Early training of an fp16 network and a bitnet with 0 added will be roughly similar in results 2. Later training will yield different / worse results, as the network gets into the 'fine tuning' part of the training. I think the paper's stats back these priors up -- they say "this works on (3B+) large networks, but not small ones." They then imply there's something about the structure of a large network that allows a bitnet to do well. It seems more likely to me it works on large networks because they have not put the compute into 3B+ networks to get past the 'gross tuning' phase. The networks they have compute to put in to get them 'fully' trained -- those networks don't show the results. Also, a quick reminder that Perplexity 12 is really terrible. You would not want to use such a network. Hopefully I'm wrong and we can get something for free here! But, I'm cautious - to - skeptical.
- mise_en_place 3y agoIntuitively I've always been a bit skeptical of quantization. Wouldn't there be a tiny loss in precision by doing this type of quantization? I could imagine the error function increasing by utilizing these types of techniques.
- eightysixfour 3y agoIt does increase the “error” (meaning it is less likely to predict the next word when compared against a dataset) but the losses are lower than your intuition would guide you to believe.
- int_19h 3y agoQuantization does reduce quality of the outputs. But the point is that you save enough memory doing so that you can cram a larger model into the same hardware, and this more than compensates for lost precision.
- 3y ago
- flockonus 3y agoConsidering how much faster additions are processed, and how a particular silicon chip could be optimized for this very specific case; all parts added together perhaps could show >100x speed up vs current systems. I must concur, "wow".
- Nevermark 3y agoFor hardware, 2-argument ternary additions and multiplications should be very close in terms of the tiny circuit required for either. If you are doing ternary calculations on 32/16-bit hardware, then the additions would be simpler.
- sva_ 3y agoI'm also curious about the potential speed gains in automatic differentiation, as there are way less branches to 'go up'. Or am I wrong here?
- lumost 3y agoThey actually use a relu to represent the model weights. But I'm not convinced that this can't be avoided. We do gradient boosted decision tree training without this trick.
- PaulHoule 3y agoI am not startled at all. Dense vector representations are pretty silly, they can’t really be the road to knowledge representation.
- p1esk 3y agoTernary networks have been used since 2015. There are hundreds of papers. They all require full QAT (training from scratch). Not sure why you’re shocked.
- cs702 3y agoBecause it's not just the use ternary values. It's also that there are no dot-products; there are only additions. And when we apply both changes to existing LLMs, there's no performance degradation (as tested by the authors).
- acchow 3y agoConversely, this also implies our current model sizes can still embed a ton more “understanding”
- Noe2097 3y agoThere is another _shocking_ realization in this work: there are 11 types of people: those who know what binary means, those who don't, and those who say they do but actually don't. "The era of 1-bit LLMs" Representing { -1, 0, 1 } can't be done with 1-bit, I'm sorry -- and sad, please let's all get back to something vaguely sound and rigorous.
- npunt 3y agoTernary supporters are always bitter about this (I'll let myself out)
- deleted 3y ago[deleted]
- esrauch 3y agoOne trit but that's not a word anyone knows.
- baq 3y agoThat used to be true yesterday…
- gpderetta 3y agoThere are 10 types of people, those who don't know binary, those who do and those who know ternary.
- hk__2 3y ago> please let's all get back to something vaguely sound and rigorous Something rigorous would be to actually read the paper rather than stop at the first part of its title. The authors are not claiming their LLM is 1-bit.
- AaronFriel 3y agoIn undergrad, some of us math majors would joke that there's really only three quantities: 0, 1, infinity. So, do we need the -1, and/or would a 2.32 bit (5 state, or 6 with +/-0) LLM perform better than a 1.58 bit LLM?
- fzliu 3y agoThis will be big for FPGAs - adders are extremely cheap compared to multipliers and other DSP blocks.
- deleted 3y ago[deleted]
- eru 3y agoMultipliers for eg 8 bit or 4 bit floating point values should also be pretty cheap? (I assume multipliers have a cost that grows quadratically with the number of bits?)
- imtringued 3y agoYou use DSPs for that. Effinix has direct bfloat16 support in their FPGAs. The real game changer is using the carry chain with your LUT based adders. Assuming 16 LUTs, you could be getting 11 teraops out of a Ti180 using a few watts. Of course that is just a theoretical number though but I could imagine using four FPGAs for speech recognition and synthesis and vision based LLMs operating in real time.
- paul_mk1 3y agoFun to see ternary weights making a comeback. This was hot back in 2016 with BinaryConnect and TrueNorth chip from IBM research (disclosure, I was one of the lead chip architects there). Authors seemed to have missed the history. They should at least cite Binary Connect or Straight Through Estimators (not my work). Helpful hint to authors: you can get down to 0.68 bits / weight using a similar technique, good chance this will work for LLMs too. https://arxiv.org/abs/1606.01981 https://arxiv.org/abs/1606.01981 This was a passion project of mine in my last few months at IBM research :). I am convinced there is a deep connection to understanding why backprop is unreasonably effective, and the result that you can train low precision DNNs; for those note familiar, the technique is to compute the loss wrt to the low precision parameters (eg project to ternary) but apply the gradient to high precision copy of parameters (known as the straight through estimator). This is a biased estimator and there is no theoretical underpinning for why this should work, but in practice it works well. My best guess is that it is encouraging the network to choose good underlying subnetworks to solve the problem, similar to Lottery Ticket Hypothesis. With ternary weights it is just about who connects to who (ie a graph), and not about the individual weight values anymore.
- nxobject 3y agoAs aside, I'm curious: what was it like to work at IBM research, especially as a legacy industrial research org?
- WhitneyLand 3y agoThat’s really interesting to see the breadcrumb trail goes back that far. So what are the most important insights in this paper compared to what was previously done? I assume there’s more context to the story and it’s not just that no one thought to apply the concepts to LLM’s until now?
- paul_mk1 3y agoI don't think there is anything conceptually new in this work, other than it is applied to LLMs. But in fairness, getting these techniques to work at scale is no small feat. In my experience quantization aware training at these low bit depths was always finicky and required a very careful hand. I'd be interested to know if it has become easier to do, now that there are so many more parameters in LLMs. In any case full kudos to the authors and I'm glad to see people continuing this work.
- fragmede 3y ago> In existing LLMs, we can replace all parameter floating-point values representing real numbers with ternary values representing (-1, 0, 1). does that mean we can do integer instead of floating point math for some parts of the training? that seems like a really big win
- verytrivial 3y ago> If the proposed methods are implemented in hardware .. And the paper is _true_ of course, indeed, this sort of compounding quantum leap in efficiency due to representational change starts to get towards the Black Mirror / SciFi foundational mythology level of acceleration. Wild (if true!)
- eru 3y agoSlight tangent: in physics a quantum leap is the smallest possible change.
- bjornsing 3y ago> * In matrix multiplications (e.g., weights by vectors), we can replace elementwise products in each dot product (a₁b₁ + a₂b₂ ...) with elementwise additions (a₁+b₁ + a₂+b₂ ...), in which signs depend on each value. See the paper for exact details. Aren’t you over complicating it a bit here? A dot product between a vector of activations (a₁, a₂, …) and a vector of ternary weights (b₁, b₂, …) can of course be computed as the sum of all activations for which the weight is 1, minus the sum of all activations for which the weight is -1. It can’t however be computed as (a₁+b₁ + a₂+b₂ ...). You must have gotten that wrong.
- api 3y agoQuestion is whether you can train in this domain or whether you need increased precision to properly represent gradients. If we could train in this domain it would be an even bigger game changer.
- phkahler 3y ago>> we can replace elementwise products in each dot product (a₁b₁ + a₂b₂ ...) with elementwise additions (a₁+b₁ + a₂+b₂ ...), in which signs depend on each value Thinking out loud here. If you encode 64 weights in 2 64-bit words you can have the bits in one word indicating +1 if they're 1, and the bits in the other word indicating -1 if they are 1. You should be able to do the "products" with a few boolean operations on these 2 words to get a pair of 64 bit words for the result. Then summing becomes a matter of using a count-of-1's instruction on each word and subtracting the "negative" count from the positive. If AVX instructions can do this too, it seems like equivalent of 10-100 TOPS might be possible on a multi-core CPU.
- cs702 3y agoYes. More generally, this will enable implementation via crazy-cheap bit-wise ops in binary hardware, and possibly, maybe, via crazy-cheap trit-wise ops in ternary hardware that manipulates ternary digits, or trits. Note that any binary op over trits has only nine possible (trit, trit) input pairs and only three possible trit outputs. Maybe ternary hardware for AI will become a thing?
- phkahler 3y agoFleshing out my thought above. If we want to multiply A*B = C and all operands are stored in 2 separate bits Ap and An (Ap = 1 if A = +1 while An = 1 if A = -1). We can do a product with: Cp = (Ap & Bp) | (An & Bn) Cn = (An & Bp) | (Ap & Bn) So 64 products in 6 instructions, or 256 in 6 instructions with AVX2, or 512 in six instructions using AVX512. If you can execute 2 instructions at a time on different words, this becomes 1024 "products" in 6 cycles or between 0.5 and 1 TOP per core. The summing still involves using popcount on the positive and negative bits - I doubt AVX supports that but its still a fast way to "sum" individual bits. I don't see custom hardware for this as a short term thing - they need to prove out the quantization concept more first.
- cs702 3y ago> I don't see custom hardware for this as a short term thing - they need to prove out the quantization concept more first. Yes, I agree. This still needs to be more extensively tested.
- chrsw 3y agoIt almost seems too good to be true
- tbalsam 3y agoIt's not too surprising, honestly! I've poked around with similar in the past and am of a perspective that ternary is a very good thing for a lot of neural networks. Training CIFAR-10 speedily w/ ternary weights on an fp16 interface (using fp16 buffers, and norm params unchanged): https://gist.github.com/tysam-code/a43c0fab332e50163b74141bccdf4588 https://gist.github.com/tysam-code/a43c0fab332e50163b74141bc...
- jsnelgro 3y agoIt all seems too good to be true but your comment helped me develop a mental model for how this could work. The most inspiring aspect to me here is just realizing how much potential low-hanging fruit there is in this space! What other seemingly naïve optimizations are there to try out?
- creshal 3y ago> * In existing LLMs, we can replace all parameter floating-point values representing real numbers with ternary values representing (-1, 0, 1). Why is this so shocking? Quantization has been widely explored, driving that to its extreme (and blowing up parameter count to make up for it) just seems like a natural extension of that. Easier said than done, of course, and very impressive that they pulled it off. > In matrix multiplications (e.g., weights by vectors), we can replace elementwise products in each dot product (a₁b₁ + a₂b₂ ...) with elementwise additions (a₁+b₁ + a₂+b₂ ...), in which signs depend on each value I feel like this follows naturally from having only ternary values, multiplication doesn't really bring much to the table here. It's a bit surprising that it's performing so well on existing hardware, usually multiplication hardware sees more optimization, especially for GPGPU hardware.
- cs702 3y ago> Why is this so shocking? Quantization has been widely explored, driving that to its extreme (and blowing up parameter count to make up for it) just seems like a natural extension of that. I find it shocking that we don't even need lower floating-point precision. We don't need precision at all. We only need three symbols to represent every value. > I feel like this follows naturally from having only ternary values, multiplication doesn't really bring much to the table here. It's a bit surprising that it's performing so well on existing hardware, usually multiplication hardware sees more optimization, especially for GPGPU hardware. I find it shocking. Consider that associative addition over ternary digits, or trits, represented by three symbols (a,b,c) has only three possible input pairs, (a,b), (a,c), or (b,c) (within each pair, order doesn't matter), and only three possible outputs, a, b, or c. Matrix multiplications could be executed via crazy-cheap tritwise operations in hardware. Maybe ternary hardware[a] will become a thing in AI? --- [a] https://en.wikipedia.org/wiki/Ternary_computer https://en.wikipedia.org/wiki/Ternary_computer
- jerf 3y agoAn integer is just a concatenation of bits. Floating point appears more complicated but from an information theory perspective it is also just a concatenation of bits. If, for the sake of argument, one replaced a 64-bit int with 64 individual bits, that's really the same amount of information and a structure could hypothetically then either choose to recreate the original 64-bit int, or use the 64-bits more efficiently by choosing from the much larger set of possibilities of ways to use such resources. Trits are helpful for neural nets, though, since they really love signs and they need a 0. So from the perspective that it's all just bits in the end the only thing that is interesting is how useful it is to arrange those bits into trits for this particular algorithm, and that the algorithm seems to be able to use things more effectively that way than with raw bits. This may seem an absolutely bizarre zigzag, but I am reminded of Busy Beavers, because of the way they take very the very small primitives of a Turing Machine, break it down to the smallest pieces, then combine them in ways that almost immediately cease to be humanly comprehensible. Completely different selection mechanism for what appears, but it turns out Turing Machine states can do a lot "more" than you might think simply by looking at human-designed TMs. We humans have very stereotypical design methodologies and they have their advantages, but sometimes just letting algorithms rip can result in much better things than we could ever hope to design with the same resources.
- nutanc 3y agoWe have been experimenting with the paper(https://www.researchgate.net/publication/372834606_ON_NON-ITERATIVE_NEURAL_ALGORITHMS_BASED_ON_THE_SEPARATION_OF_HIGH_DIMENSIONED_SAMPLE_POINTS_BY_HYPERPLANES https://www.researchgate.net/publication/372834606_ON_NON-IT...). There is a mathematical proof that binary representation is enough to capture the latent space. And in fact we don't even need to do "training" to get that representation. The practical application we tried out for this algorithm was to create an alternate space for mpnet embeddings of Wikipedia paragraphs. Using Bit embedding we are able to represent 36 million passages of Wikipedia in 2GB.(https://gpt3experiments.substack.com/p/building-a-vector-database-in-2gb https://gpt3experiments.substack.com/p/building-a-vector-dat...)
- cs702 3y agoYou're talking about mapping floating-point vector representations, i.e., embeddings, computed by a pretrained LLM to binary vector representations, right? And you're talking about doing this by first having someone else's pretrained LLM compute the embeddings, right? Sorry, but that seems only minimally, tangentially related to the topic of running LLMs in ternary space. I don't see how your comment is relevant to the discussion here.
- nutanc 3y agoYeah, sorry, needed a much bigger canvas than a comment to explain. Let me try again. The example I took was to show mapping from one space to another space and it may have just come across as not learning anything. Yes. You are right it was someone else's pretrained LLM. But this new space learnt the latent representations of the original embedding space. Now, instead of the original embedding space it could also have been some image representation or some audio representation. Even neural networks take input in X space and learn a representation in Y space. The paper shows that any layer of a neural network can in fact be replaced with a set of planes and we can represent a space using those planes and that those planes can be created in a non iterative way. Not sure if I am being clear, but have written a small blog post to show for MNIST how an NN creates the planes(https://gpt3experiments.substack.com/p/understanding-neural-networks-and https://gpt3experiments.substack.com/p/understanding-neural-...). Will write more on how once these planes are written, how we can use a bit representation instead of floating point values to get similar accuracy in prediction and next how we can draw those planes without the iterative training process.