12 ms·
“Imprecise” language models are smaller, speedier, and nearly as accurate
- aussieguy1234 2y agoWhen it needs to do more precise numerical calculations, perhaps, like a human, it could just use a calculator?
- kromem 2y agoThat's not really a concern. If you have a trillion parameter 8-bit fp network or a trillion parameter 1.5-bit ternary network, based on the scaling in Microsoft's paper the latter will actually perform better. A lot of the current thinking is that the nodes themselves act as superpositions for a virtualized network in a multidimensional vector space, so precision is fairly arbitrary for the base nodes and it may be that constraining the individual node values actually allows for a less fuzzy virtualized network by the end of the training. You could still have a very precise 'calculator' feature in the virtual space no matter the underlying parameter precision, and because each parameter is being informed by overlapping virtual features, may even have less unexpected errors and issues with lower precision nodes.
- xwolfi 2y agoYup, your response makes me think they should just use a calculator, like everyone.
- imtringued 2y agoI don't know what you mean. They already use the GPU as a calculator.
- ben_w 2y agoI believe you four are talking about different things; the models are executed on very good "calculators" (if you want to call the GPUs that), but themselves are not very good at being used as calculators. LLMs are sufficiently good hammers that people see everything as a nail, then talk about how bad they are at driving screws.
- CuriouslyC 2y agoPeople are trying to make these monolithic god models right now because everyone's chasing OpenAI's example with ChatGPT, but that approach is going to run out of steam for a lot of reasons. We're eventually going to end up with an agent model that does just what you say, recognizes specialized problem types such as math and calls the appropriate tool, such as a calculator or symbolic computation engine.
- zozbot234 2y agoMany models support function calling/RAG already, these are very similar features from a structural POV. But of course it's harder to train a model for such tasks, compared to just fitting some existing training set.
- CuriouslyC 2y agoI'm not sure having the LLM as the top level piece is the right approach though. Async is the direction we want to go, and LLMS are inherently synchronous. Additionally, LLMs are cumbersome to train and tune, having an agent that calls smaller models would unlock the power of the community to build customized models for specific needs.
- szundi 2y agoWe would like to have something like us. My daughter is just 3.5yrs old, she is inherently hard to train too, decades left. However, we find ourselves quite good.
- CuriouslyC 2y agoYou're assuming you're a monolith, but in fact you have many subnetworks in your brain. The neocortex acts to process information from other parts of the brain, and it itself comprised of a network of interacting modules.
- szundi 2y agoChatGPT routinely does this already
- darkerside 2y agoIt should just be able to program. A programming language is the ultimate verbal-to-math translation layer. Tangent but... programming languages aren't designed for computers. Computers are perfectly happy with assembly or even binary. Programming languages are designed for humans, not just so we can see what others have done, but so that we can understand what we ourselves have done. We give the variable a name because we can't remember 0x0010101; but the computer remembers them both just fine.
- gslin 2y agoThis makes me watch https://en.wikipedia.org/wiki/11001001 https://en.wikipedia.org/wiki/11001001 again.
- sleepybrett 2y agosmaller bucket of shit just as effective at getting you stinky as a large bucket of shit
- blurbleblurble 2y agoI know there are 1-bit experiments out there but from what I can tell it's BitNet 1.58 that's really exciting. So why do people keep focusing on "1-bit" when the whole reason 1-trit models are so successful in the first place just might have everything to do with the ternary weights and the symmetry they're able to encode? I'm not sold on the suggestion that "imprecise language models" in general are "nearly as accurate" when it might actually be that ternary weights are more precise in a totally difference sense: in that they just might be capturing a minimal representation of the very symmetries responsible for making all those parameters effective in the first place.
- lavelganzu 2y agoMake a tiny language model and then use it for autocorrect & autocomplete. Current autocorrect & autocomplete on smartphones is so bad that it spawns countless jokes. This use case doesn't require a large language model able to write whole paragraphs -- it only needs a model that's just barely big enough to make contextually appropriate suggestions of one or a few words.
- hiddencost 2y agoNope nope nope. Any time we find more efficiency, we can trade it for more quality by doing more compute. We'll always use as much compute as we can afford, until we stop getting quality gains that are worth the added cost.
- XorNot 2y agoIsn't this article just about an optimization though, sans the title? I don't much care for all the "oh but the energy usage" claims in most tech things: it's all electricity, and it's all fungible. It usually seems to roll out as a proxy for "I don't like this thing". Like even with cryptocurrency, there were a lot of people mistaking the issue of scalability - namely that "as a store of value" crypto would consume incredible amounts of other resources (and a lot of people got stuck trying to figure out how somehow "a hash" could be reclaimed for useful resources) to do less then alternatives, with "the energy usage itself is the problem". Finding optimizations for LLMs is good because it means we can build cheaper LLMs, which means we can build larger LLMs then we otherwise could for some given constraint, which means we can miniaturize (or in this case specialize) more capable hardware. The thing which really matter is, can the energy usage be meaningfully limited to a sensible scaling factor given the capability that makes them useful? Because environmentally, I can install solar panels to do zero-carbon training (and if LLMs are as valuable as they're currently being priced, this is a no-brainer - if people aren't lying about solar being "cheaper then fossil fuels").
- alisonkisk 2y agoNo, because the article is about sacrificing quality by using less energy. But the quality is already bad for non-toy purposes, so there is plenty of demand for higher-energy higher-quality systems.
- dartos 2y agoIt’s good that we can build cheaper LLMs, but the problem companies guzzling energy for LLM training won’t use less energy, they’ll just have better models
- jandrese 2y agoI'm starting to feel like the Thinking Machines CM-1 was 40 years ahead of its time.
- chuckadams 2y agoIt had the most wonderful blinkenlights too! :)
- deepfriedchokes 2y agoOne of my favorites, CM-1 t-shirt: https://www.tamikothiel.com/cm/cm-tshirt.html https://www.tamikothiel.com/cm/cm-tshirt.html
- boringg 2y agoEasier just to deploy more renewable energy
- alex_duf 2y agoor is it cheaper to optimize a model to crush the competition on price?
- adtac 2y agoLike often, both will probably happen
- boringg 2y agoRight - it obviously isn't a binary option you can pursue both in unison. I would want to continue the statement that we are still early innings on renewable energy -- and let's keep deploying it rapidly to manage increased compute demand.
- brcmthrowaway 2y agoAre there any 1-bit LLMs available on GitHub?
- hughesjj 2y agoHeck they have some zero bit llm's https://github.com/kelseyhightower/nocode https://github.com/kelseyhightower/nocode
- lhl 2y agoBitNet Implementations: * https://huggingface.co/1bitLLM/bitnet_b1_58-large https://huggingface.co/1bitLLM/bitnet_b1_58-large * https://github.com/Oxen-AI/BitNet-1.58-Instruct https://github.com/Oxen-AI/BitNet-1.58-Instruct * https://github.com/nkotak/1.58BitNet https://github.com/nkotak/1.58BitNet See some followups that has some training advice: https://github.com/microsoft/unilm/tree/master/bitnet https://github.com/microsoft/unilm/tree/master/bitnet
- Der_Einzige 2y agoQuantization is never free, and you can rest assured that even the "good" quants of the best models are highly crippled compared to their unquantized versions. The "nearly as accurate" is only on their contrived benchmarks. I've never met a quantized model that actually behaved "98%" as good as the unquantized model, and I do LLM work daily and have since well before the ChatGPT era.
- rockskon 2y agoI've never met an LLM that consistently behaved well at all. Quantized or unquantized. Honestly it feels like the bulk of the industry is acting out one big LARP where they're always right around the corner from developing AGI and doing something big and amazing and....it just never materializes. Obnoxious AI agents, worthless AI-generated websites crowding out useful resources, unreliable AI search results. The AI industry has done very well for itself selling hype. Now it needs actual good products.
- hn_throwaway_99 2y agoThis makes 0 sense to me. I get tons of real world value and productivity benefits from AI, most specifically ChatGPT and cursor.sh. I don't disagree that there is a ton of hype, a lot of it unwarranted, and I cringe when I see tons of companies trying to "throw AI against the wall and see if it sticks" (I personally nominate LinkedIn's AI blurbs on their feed as "most useless and annoying use of AI"). But still, I'm blown away with how much value I get from AI. It makes me a bit sad that so many of us have become so jaded that they see it as "one big LARP".
- Capricorn2481 2y agoIt's dismissive to call it jaded. I don't think you, me, or the people you're talking about are intellectually different. We just don't like what AI makes even if we think it's impressive.
- hn_throwaway_99 2y ago
- light_hue_1 2y agoThe performance loss from BiLLM is disastrous. It's basically useless. No one would ever want to use the resulting models. They hide their main results in the appendix: Table 8. page 15. https://arxiv.org/pdf/2402.04291 https://arxiv.org/pdf/2402.04291 I won't go over the entire table in detail, but PIQA, BoolQ, HellaSwag, and WinoGrande should in the mid-to-high 70s for LLaMa2-7B. They drop that to 58, 62, 32, and 51. There are 700M parameter models that perform much better. What they should have reported is effective number of parameters. Does LLaMa2-7B with their quantization method outperform a model that has amount of computation but uses that compute with say.. 16-bit quantization? If the answer is no, and it seems like it very clearly is, then the method is wholly worthless. Just use a smaller model to begin with. The BitNet paper is better. But for some reason they only consider very small models. It's the obvious question and in their FAQ they don't provide a solid answer to it. Despite having all of the compute resources of MS. They could have easily run this experiment in the past year; I'm suspicious.
- imtringued 2y agoBiLLM is about post training quantization and BitNet trains models from scratch. You do realize that one of those is going to give significantly worse results and the other is going to be significantly more expensive, well into the millions of dollars?
- light_hue_1 2y agoThere's no mathematical reason why doing quantization after would be worse than training from scratch. That's nonsense. There's also no practical reason. Training quantized networks is often harder! This is why people quantize after the fact or do distillation. Nor is there any reason why we won't find some projection of weights onto the BitNet manifold. If it was published by academics I'd believe the cost argument. This was published by MS. They can run this experiment trivially. I have friends at MS with access to enough compute to do it in days. Either the authors ran it and saw it doesn't work or they're playing with us. Not a good look. The reviewers shouldn't have accepted the paper in this state without an explanation of why the authors can't do this. This is the question that determines if this work matters or is useless. Publishing before knowing that isn't responsible on anyone's part.
- WiSaGaN 2y agoThis seems saying that the 1bit model is a bit better than GPTQ Q2. However, I find there are few situations where you would want to use GPTQ Q2 in the first places. You would want to run the F16 version if you want quality, and if you want to have a sweet spot, you usually find something like Q5_K_M of the biggest model you can run.
- qeternity 2y agoNobody is running llama.cpp in production…
- luke-stanley 2y agoWhat do you mean? What makes you think that?
- qeternity 2y agoBecause for anything other than CPU inference it is inferior to TensorRT and vLLM.
- luke-stanley 2y agoAre you sure about that? what are the benchmarks it fails on when setup like-for-like with GPU drivers? Even still, it can do constrained grammar and is really easy to setup with wrappers like Ollama and the Python server too. I struggled to find support for that with other inference engines - though things change fast!
- szundi 2y agoProbably more often than not
- qeternity 2y agoHighly doubtful. If anyone is running it here in a production setting, please post and prove me wrong.
- mlsu 2y agoI don't believe that quantization comes for free. Someone made the observation that llama3 models quantize "worse" than llama2 models do -- they suffer in quality from quantization far more. My intuition is that a model which is undertrained suffers less from quantization, because the training process has not utilized each weight to its full potential. One of the key findings with llama, and why it punches above its weight for its size, is that they trained it for longer on a much larger dataset then was "optimal" according to the literature up to that point. Putting two and two together, it seems that: small model, lots of data, long training > large model + quantization That basically, quantization is a lossy shortcut to the tail of training long. Amount and quality of data is, as always, the most important part about all of this.
- kir-gadjello 2y agoWhile llama3-8b might be slightly more brittle under quantization, llama3-70b really surprised myself and others[1] in how well it performs even in the 2..3 bits per parameter regime. It requires one of the most advanced quantization methods (IQ2_XS specifically) to work, but the reward is a SoTA LLM that fits on one 4090 GPU with 8K context (KV-cache uncompressed btw) and allows for advanced usecases such as powering the agent engine I'm working on: https://github.com/kir-gadjello/picoagent-rnd https://github.com/kir-gadjello/picoagent-rnd For me it completely replaced strong models such as Mixtral-8x7B and DeepSeek-Coder-Instruct-33B. 1. https://www.reddit.com/r/LocalLLaMA/comments/1cst400/result_llama_3_mmlu_score_vs_quantization_for/ https://www.reddit.com/r/LocalLLaMA/comments/1cst400/result_...
- d13 2y agoHow does it compare against unequalised Llama 3 8B at 16fp? I’ve been using that locally and it’s almost replaced GPT4 for me. Runs in about 14GB of VRAM.
- LordDragonfang 2y agoWhat is your use case where you find it comparable to gpt4?
- roschdal 2y ago"Nearly accurate" as in "incorrect"
- CGamesPlay 2y agoDon't worry, the LLMs were only "fairly accurate" (as in "incorrect") to begin with.
- akira2501 2y agoThe lesson being... We spent billions for incremental gains masquerading as tectonic shifts. Whoops.
- 13alvone 2y agoI've been thinking about how far we've come with large language models (LLMs) and the challenge of making them almost perfect. It feels a lot like trying to get a spaceship to travel at the speed of light. We’ve made impressive progress, getting these models to be quite accurate. But pushing from 90% to 99.9999999% accuracy? That takes an insane amount of data and computing power. It's like needing exponentially more energy as you get closer to light speed. And just like we can’t actually reach the speed of light, there might be a practical limit to how accurate LLMs can get. Language is incredibly complex and full of ambiguities. The closer we aim for perfection, the harder it becomes. Each tiny improvement requires significantly more resources, and the gains become marginal. To get LLMs to near-perfect accuracy, we'd need an infinite amount of data and computing power, which isn't feasible. So while LLMs are amazing and have come a long way, getting them to be nearly perfect is probably impossible—like reaching the speed of light. Regardless I hope to appreciate the progress we've made but also be realistic about the challenges ahead. What do you think? Is this a fair analogy?
- jjayj 2y agoI think I agree with your analogy, but would say 99% rather than 99.99999%. Beyond that, I'm not entirely sure what a "perfect" LLM would even be defined as.
- 13alvone 2y agoThat makes sense, and just harder and harder to get more accurate. Same as humans I supposed :)
- jjayj 2y agoPast 99%, what does "more accurate" mean? I think it will vary from person to person and use case to use case, which is why I personally don't foresee a world where an LLM or any form of AI/ML is ever perfectly accurate. I'm struggling to think of any medium that has ever reached 100% accuracy, so to target that for an ML algorithm seems foolhardy
- usgroup 2y agoIs there some work on how small a model with some specific epsilon perplexity could theoretically be? Given a fixed architecture and a fixed dataset, I presume there is a minimal number of parameters required for optimal representation.
- renonce 2y agoIf you are referring to what is theoretically possible with arbitrary computation in the model, it's called Kolmogorov complexity and it's not computable.
- usgroup 2y agoWith a fixed architecture, and a fixed dataset, as mentioned. So, a specific kind of neural network, and a fixed dataset.
- jononor 2y agoYou can estimate it empirically. However large changes in model parameters/capacity tends to interact with hyperparameters, so would want to do runs with multiple values of hyperparameters. And training processes give noisy results, so might want to do multiple repetitions. And each run may take several GPU days. So even a small experiment of 10 repetitions X 10 hyperparameters X 10 model sizes takes several thousand GPU days. But there are many papers from the large labs that do such. And the whole result is also conditional on the optimization/training process used. Which is an area where we have no reason to think that we are optimal... So we can do studies with practical results (given sufficient money), but we are far from being able to identify the actual maximums available.
- renonce 2y agoThe closest research would be the Chinchilla scaling laws, which estimates the final loss as a function of number of parameters and tokens. Set the number of tokens to infinity would give a good estimate of minimum achievable loss.
- chx 2y agoNearly as accurate? I guess zero is close to zero. https://hachyderm.io/@inthehands/112006855076082650 https://hachyderm.io/@inthehands/112006855076082650 > You might be surprised to learn that I actually think LLMs have the potential to be not only fun but genuinely useful. “Show me some bullshit that would be typical in this context” can be a genuinely helpful question to have answered, in code and in natural language — for brainstorming, for seeing common conventions in an unfamiliar context, for having something crappy to react to. > Alas, that does not remotely resemble how people are pitching this technology.
- Last5Digits 2y agoAre you going to spam this same link in every single thread about LLMs on HN? People have provided good arguments refuting whatever you're trying to say here, but you just keep posting the same thing while not engaging with anyone.
- chx 2y ago[flagged]
- dang 2y agoYes but please try to avoid repetition on HN. The GP's response was rude and broke the site guidelines but https://hn.algolia.com/?dateRange=all&page=0&prefix=true&query=hachyderm.io%2F%40inthehands&sort=byDate&type=comment https://hn.algolia.com/?dateRange=all&page=0&prefix=true&que... does look excessive to me.
- Last5Digits 2y agoNo, the answers aren't just "plausible", they are correct the vast majority of the time. You can try this for yourself or look at any benchmark, leaderboard or even just listen to the millions of people using them every day. I fact check constantly when I use any LLM, and I can attest to you that I don't just believe that the answers I'm getting are correct, but that they actually are just that. But they apparently actually don't get better even though every metric tells us they do, because they can't? How about making an actual argument? Why is correctness "not a property of LLMs"? Do you have a point here that I'm missing? Whether or not Kahneman thinks that there are two different systems of thinking in the human mind has absolutely no relevance here. Factualness isn't some magical circuit in the brain. > No such thing can exist. In the same way there can exist no piece of clothing, piece of tech, piece of furniture, book, toothpick or paperclip that is environmentally friendly; yes. In any common usage, "environmentally friendly" simply means reduced impact, which is absolutely possible with LLMs, as is demonstrated by bigger models being distilled into smaller more efficient ones. Discussing the environmental impact of LLMs has always been silly, given that we regularly blow more CO2 into the atmosphere to produce and render the newest Avengers movie or to spend one week in some marginally more comfortable climate.
- bambax 2y agoI'm surprised the comments here are so negative / cynical. Of course "quantization doesn't come for free": it's a trade off. High precision models are very expensive to run: they require expensive hardware and lots of energy. Isn't it possible, or even likely, that low-precision models would be good enough for many tasks?
- az09mugen 2y agoI totally agree with your statement. I consider LLMs imprecise in any case, it's not a perfect/exact science, just statistics. I only use LLMs for tasks where an error margin can be allowed. In that perspective I'm totally fine with using a 5 MB LLM like this one : https://neuml.hashnode.dev/train-a-language-model-from-scratch https://neuml.hashnode.dev/train-a-language-model-from-scrat...
- kromem 2y agoPost-training quantization doesn't come for free, but the pretraining on constrained precision weights actually counterintuitively results in a performance increase per parameter as the number of parameters grows in the ternary BitNet paper. Even if there was zero efficiency gain in ternary weights, large models should probably be trained on a network of precision limited weights from here on out given the research so far. I suspect it relates to each weight relating to multiple 'features' in the network. The greater the precision, the more room it gives for competing features to compromise on node values that aren't best for either feature instead of reorganizing the feature mapping to avoid conflicts.
- trashtester 2y agoThe number of bits used per weight during training, could be included in the regularization, perhaps? For instance, one could extend dropout regularization to several levels, where each weight could have random chances to include the most significant 2-16 bits part of the time (and still 0 part of the time), and where the impact on the gradient of having fewer bits could be used to tune the ideal number of bits for each weight. Then one could add L1 regularization for the total number of bits used to squeeze the total down down to whatever size one aims for.
- renonce 2y agoI study LLM quantization and I have surveyed GPTQ and QuIP# and lots of quantization algorithms (specifically PTQ, post-training quantization) to develop my own, and my experience has led me to become extremely skeptical of many of the papers. I've seen lots of headlines like "1-bit quantization" (including this one and https://arxiv.org/abs/2310.16795 https://arxiv.org/abs/2310.16795). What I've found in this space is that the headlines can often be intentionally misleading about what is actually achieved. If you read closer the abstract of this paper, it claims 8.41 perplexity on LLaMA2-70B at 1 bit, which is a HUGE decrease from 3.120 perplexity in FP16 and they will never mention that in the headline. Even LLaMA2-7B at INT8 achieves 5.677 perplexity with half of storage place (better with LESS space and LESS training). Some claim 1.58-bit quantization (each weight is either -1, 0 or 1) but in practice require very small group sizes, which means one or two extra FP16 numbers for every 64 weights and that means another 0.5 bit, so it's actually 2-bit quantization. And every quantization algorithm can claim they make language models smaller, speedier, and use less energy, so there's nothing special about these. Here are the key metrics that I suggest checking when comparing quantization schemes: * Perplexity. Note that it also depends on the dataset (either WikiText2 or C4, WikiText2 numbers are usually lower than C4) and context size (1024, 2048 or 4096, higher context sizes usually means less perplexity). Dataset and context size must match to make a meaningful comparison. * Quantization bits. Many algorithms claiming 2-bit or 1-bit quantization has lots of extra parameters elsewhere, such as grouping. Download the quantized version of the model, check its file size, multiply by 8 and divide by the number of parameters. That gets you the ACTUAL quantization bits. * Performance. Weights may need to be dequantized during inference which could introduce overhead. Some libraries have custom matmul kernel for dequantization that achieves performance close to FP16, others can be slower. Check its generation speed and inference speed. Newer architectures such as Ampere contains INT8 cores which may make quantized version even faster than FP16, I haven't tried out yet. There is also a lot of misleading comparisons in this space. Some methods like GPTQ only provide vector-matrix multiplication kernels, which means only a single token can be generated and batched inference (which is needed for generating initial KV cache, or for serving multiple users) can be much slower. If an algorithm claims a 3x speedup for something, check if they refer to single stream latency or multiple stream throughput. Some of that speedup comes from running a model on 2 cards instead of 5 cards without specifying if the cards have NVLink configured (you shouldn't run inference on multiple cards without NVLink, or you should expecet huge slowdown simply because of using 5 cards). * Base model. Pick a STRONG base model like Llama-2-7b or Llama-3-8b etc. Not an undertrained model like SwitchTransformer etc which may have lots of redundant parameters in itself. My personal favourite remains QuIP# (https://github.com/Cornell-RelaxML/quip-sharp https://github.com/Cornell-RelaxML/quip-sharp). It lacks in the "performance" part as its matrix multiplication performance isn't on par yet but there is room for improvement, and it wins every other metric. And sad news: it's very likely we won't have practical 1-bit LLMs, never ever. We are reaching the end game between 2.5~4 bits. By "practical" I mean it should beat 3-bit LLMs with 3x less parameters or 2-bit LLMs with half as many parameters. There is a Shannon limit to quantization whatever methods you use.
- Decabytes 2y agoI feel like we are getting closer and closer to a finding the Goldilocks Llm, that with some smarter training and the right set of parameters, will get us close to got 3.5 turbo performance but at a size, cost, and time effort that is significantly lower and that is runnable locally. Combine that with what seems like every chip adding a neural engine and it feels like we are in the early days of high performance graphics again. Right now we are in the unreal engine voodoo era, where graphics cards/neural engines are expensive/rare. But give it a few generations and soon we can assume that even standard computers will have pretty decent NPUs and developers will be able to rely on that for models
- moffkalast 2y ago3.5-turbo performance wasn't very good though, and according to API statistic analysis it's a Nx7B model so it's already rather small. Ultimately Llama-3-8B is already better in all measurable metrics except multilingual translation, but that's not saying much.
- djeastm 2y ago>3.5 turbo performance Is this the level of performance people are relying upon? While I've always been impressed with the technology itself, it's only starting with GPT 4 that I think it approaches adequate performance.
- Decabytes 2y agoFor the work that I do (which is mostly rag with a little bit of content generation) GPT 3.5-turbo-0125 with a 16k context window is the sweet spot for me. I started using the api when it was only a 4k context window, so the extra breathing room provided by the 16k context window feels cavernous. Plus the fact that it's $0.50 per 1Million tokens means that I can augment my software with LLM capabilities at a cost that is attractive to me as a small time developer. The way I rationalize it is that using 3.5-turbo is like programming on an 8-bit computer with Kilobytes of Ram, and gpt-4o is like programming on a 64bit computer with a 4080 ti and 32gb of ram. If I can make things work on the 8-bit system, they will work nicely on the more powerful system.
- Gaazrukk 2y ago.999936565
- bunderbunder 2y agoThey keep calling them 1-bit LLMs, but they're really 1-trit LLMs. If you can have 3 states, it's not a BInary uniT, it's a TRInary uniT. I don't think that this is just a nit, it implies a real mismatch between how these models work, and how modern computing hardware works. People have built ternary computers in the past, but to the best of my knowledge nobody's made a serious attempt at it in half a century. You can always use two bits to store a ternary value, of course. But then it wouldn't be a 1-bit LLM, now, would it? And that doubling in compute resources required would make a tangible difference in how one has to think about potential efficiency improvements. Also, ternary tritwise logic implemented on binary hardware is unlikely to be anywhere near as efficient as binary bitwise logic on hardware that was specifically engineered for the purpose. This leaves me thinking that the research teams involved continuing to refer to these as 1-bit LLMs must be interpreted as knowingly engaging in dishonest marketing tactics.
- freeone3000 2y agoIt’s stored in and calculated using a floating-point value, and uses log(2) bits. So it’s ~1.2 bits with float packing over 32m parameters.
- IanCal 2y agoI don't think it's quite so clear cut, and the papers and names are often a bit more precise anyway. BitNet is (I think) actually one bit per parameter. BitNet 1.58b isn't, but then to be fair isn't describing itself as a 1 bit llm. I'm less sure but it seems OneBit is 1 bit. One of the processes mentioned in the article is a mix of one and two bits for the weights.
- orlp 2y agoPerhaps in the future we will see MsoTriState neural networks: https://learn.microsoft.com/en-us/dotnet/api/microsoft.office.core.msotristate https://learn.microsoft.com/en-us/dotnet/api/microsoft.offic... For those unaware, the MsoTriState is fairly self-explanatory: it is a tri-state boolean type with five possible values.
- neolefty 2y agoThe article describes both. The 1,0,-1 networks are "1.58 bits" — that is, log2(3). But yeah it mostly focuses on 1-bit networks.
- celltalk 2y agoYou all run on 2-bit (DNA) and you all seem pretty stable to me. You run your brain with couple of watts per day and add big numbers without much hassle.
- JoeAltmaier 2y agoLots of folks have the intuition that, 'slightly worse' LLM models means unacceptable rates of nonsense answers. What's an acceptable rate? Is it 99%? 99.9%? The closer it gets to 99.999% good answers, the more damaging the wrong ones become. Because people have been trained, too. Trained to trust the answers, which makes them lazy and vulnerable to lies.