9 ms·
Towards 1-bit Machine Learning Models
- corysama 3y agoI look forward to the inevitable probabilistic sub-bit machine learning models :)
- deleted 3y ago[deleted]
- Cieric 3y agoI assume this is ment to be a joke, but isn't this actually being worked on? (minus the probabilistic portion) https://arxiv.org/abs/2310.16795 https://arxiv.org/abs/2310.16795
- Cheer2171 3y agoMy 0-bit model can predict if you have cancer with 99.5% accuracy. It doesn't even need input data! Don't ask about the false negative rate though.
- paul_mk1 3y agoSub 1-bit has been done at least as far back as 2016 for VGG style networks (my work). I was able to get 0.68 "effective" bits. The idea is that in each forward pass you add noise to each weight independently drawn from normal distribution, and when you calculate snr, it's sub 1 bit. Points to the idea that a stochastic memory element can be used. https://arxiv.org/abs/1606.01981 https://arxiv.org/abs/1606.01981
- overengineer 3y agoIf noise is desirable in this way, why not to just switch back to analog computing which comes with free noise
- iamgopal 3y agoSo new AI chip will be coming optimised for large scale non-byte bit only AI world ?
- thesz 3y agoEvery CPU and GPU now has popcount [1] instruction. You can thank US three letter agencies for that [2]. [1] https://en.wikipedia.org/wiki/Hamming_weight#Processor_support [2] https://vaibhavsagar.com/blog/2019/09/08/popcount/ Multiplication then is just bitwise AND and a popcount.
- JKCalhoun 3y agoAs an AI layman, I am for some reason excited by the low-bit learning models. Apple, now leaving behind the albatros that was the Apple car, could be the one to roll their own silicon and put an LLM in our back pockets. Goodbye Siri, hello Seriously.
- talldayo 3y agoOr we could end up getting another Google integration, with iOS being too rigid and locked-down to feasibly support third-party private AI alternatives. Fate has a cruel sense of humor, sometimes.
- odyssey7 3y agoAre you referring to the Swift version of TensorFlow? https://www.infoworld.com/article/3608151/swift-for-tensorflow-project-shuts-down.html https://www.infoworld.com/article/3608151/swift-for-tensorfl...
- Me1000 3y agoI believe they're referring to these rumors: https://www.macrumors.com/2024/03/21/apple-iphone-ai-talks-google-openai-ongoing/ https://www.macrumors.com/2024/03/21/apple-iphone-ai-talks-g...
- JKCalhoun 3y agoCould be a way for Apple to buy time as well until they do have something competitive.
- deleted 3y ago[deleted]
- lawlessone 3y ago> could be the one to roll their own silicon and put an LLM in our back pockets. Wouldn't they have to do a lot of catching up with everyone else already working on it?
- fabmilo 3y agoI like how it goes from 13.5 GB to 1.76 GB and gets comparable results. Definitely there is some underlying process of why this works so well that we are missing. Are the bits selecting the different subspaces? Can we improve this process by using better normalization layers / gating units?
- nobodyandproud 3y agoI wonder what information theory can tell us here.
- ttul 3y agoNeural networks work, but why they work is not well known. Researchers continue to find new “free lunches” all the time.
- rapatel0 3y agoThe entropy of English is ~1 bit per letter (Measured in a funny way by Shannon - https://cs.stanford.edu/people/eroberts/courses/soco/projects/1999-00/information-theory/entropy_of_english_9.html https://cs.stanford.edu/people/eroberts/courses/soco/project...) In general, it's a bit funny in how we build ML models. You take a bucket of matrices, fill them with random numbers, and then start to introduce "biases" through back propagation. A model can converge down loss, but most of them are still filled with random noise. Binary weights are somewhat obvious in retrospect. Weights indicate the strength of an association between two neurons. Intuitively, most of the value is probably just that an association between two neurons exists, and whether it's positive or negatively associated.
- Animats 3y agoI'm surprised that 1-bit works. Trinary (-1, 0, 1) makes sense, but if you only have 0 and 1, the asymmetry ought to be a problem. Or is multiplying an XOR, so 1x1 = 0?
- edflsafoiewq 3y ago
- WithinReason 3y ago1-bit weights have been a thing since at least 2016: https://arxiv.org/abs/1606.06160 https://arxiv.org/abs/1606.06160
- buildbot 3y agoXNOR-Net was the "first" in that generation, as I recall. It's not a new idea at all though. Check out the dates on these papers - https://ieeexplore.ieee.org/abstract/document/286901 https://ieeexplore.ieee.org/abstract/document/286901 < 1994 https://ieeexplore.ieee.org/abstract/document/344783 https://ieeexplore.ieee.org/abstract/document/344783 < 1993 https://link.springer.com/article/10.1007/BF00337115 https://link.springer.com/article/10.1007/BF00337115 < 1986
- rapatel0 3y agoDon't forget Michael https://arxiv.org/abs/1511.00363 https://arxiv.org/abs/1511.00363 < 2015
- elcomet 3y agoThey have been a thing for decades https://ieeexplore.ieee.org/abstract/document/363432 https://ieeexplore.ieee.org/abstract/document/363432
- redskyluan 3y agowhat will be the next step of 1 bit? could it be 0.5 bit? Any kind of quantization is not gonna to last long.
- CamperBob2 3y agoNot using traditional numbers at all, I imagine. It's possible that trying to build neural nets out of vectors that are themselves built with quantized scalars is a suboptimal approach.
- pixelpoet 3y agoThe way things are going we'll have -1 bit before long :)
- ganzuul 3y agohttps://en.wikipedia.org/wiki/Adjacency_matrix https://en.wikipedia.org/wiki/Adjacency_matrix What would a graph look like when edges are represented with less than 1-bit?
- robrenaud 3y agoIt would have predictible structure. There would be groupwise, regular sparsity. You could exploit these patterns to infer properties like, "there is no connection from this vertex to this entire span of verticies". Just think about representing a compressed version of a matrix. You want to have a short encoding for matrices are ones that are more ordered in some real, exploitable way. With some determnistic computation of the compressed representation, you could produce the full adjacency matrix.
- ganzuul 3y agohttps://proceedings.mlr.press/v206/meller23a/meller23a.pdf https://proceedings.mlr.press/v206/meller23a/meller23a.pdf It seems to have a lot of similarities to the technique in OPs article.
- selecsosi 3y agoI think the problem is that you are also trying to represent the biases of directionality in the connection e.g.: [-1,0,1]. In the adjacency matrix, if you store the sign (direction) of the edge, then I think that would map fairly isomorphically.
- ganzuul 3y agoBasically factorized into a giant state machine. Would be logical.
- grungegun 3y agoDoes anyone know if this works on vanilla deep networks? These quantization articles always seem to target LLM's which leads me to wonder if there's something special about the LLM architecture vs a vanilla deep architecture.
- deleted 3y ago[deleted]
- Y_Y 3y agoCertainly does, people have been doing this in computer vision for years.
- alephxyz 3y agoLLMs have been trending towards obscenely large number of parameters (314B for grok), which makes quantization crucial if you want to run them without a Meta-sized budget.
- zaptrem 3y agoTransformer LLMs are just a bunch of MLPs (linear layers) where you sometimes multiply/softmax the output in a funny way (attention). In other words, they're arguably more "vanilla deep net" than most architectures (e.g., conv nets). (There are also positional/token embeddings and normalization but those are a tiny minority of the parameters)
- amelius 3y agoOk, but what does a perceptron look like in 1-bit? Would it be just some simple logic gate, like an OR-gate?
- zaptrem 3y agoNot my area of expertise but I'd assume it becomes a decision tree or something. Edit: lol https://news.ycombinator.com/item?id=39868508 https://news.ycombinator.com/item?id=39868508
- 3y ago
- ianbicking 3y agoReduction to decision tree! But I'm unclear how it actually run, and the article talks about the conversion and training but doesn't describe how it runs... I suppose because it's obvious to someone who has followed quantization. Thinking out loud... if you have a model of just 1 and 0, my first thought is that the outputs are 1's and 0's but I think that's wrong. Instead it's a bunch of floats, and you multiply them by 1 or 0 (in a sense you are sampling the output of the previous layer?), add them up, and put the result through some activation function. And two-bit quantization sounds kind of similar, just with a _little_ scale, going from -1 to 2 instead of 0 to 1. It seems kind of interesting that you now have a bunch of weights that are exactly 0, meaning you can assert something about what parameters and weights affect what parts of the output. Though in some sense the way they compress the weights down to one bit is also a heuristic you could use to interpret the original model... this just makes it clearer that in totality you are making a defensible simplification, because the end result is still a working model. It also seems like you could make a lot of mathematical assertions about a one bit model that would be harder to make otherwise. Like maybe you could start thinking of a model as an equation, a _particular_ (though giant) equation, and look at its properties and consider symbolic transformations to that equation.
- fabmilo 3y agoI like the decision tree analogy
- bick_nyers 3y agoA comment I really liked on a previous post about ternary highlighted that what you are actually measuring with {-1, 0, 1} is inverse correlation, no correlation, and correlation.
- mmoskal 3y agoIt seems the trick here is they first quantize it to 1- or 2-bit, and then they fine-tune the quantization bias parameters (the parameters that dequantize from 1-2 to 16 bit) via LoRA. Then they have specialized kernels to do matrix multiplication at the bit level. Also, the 2-bit model seems much better than the 1-bit model - they use [-1, 0, 1, 2] - I wonder if '2' is needed in light of the 1.58b paper (which claims -1 is def. needed).
- andy_xor_andrew 3y agoInteresting, and it kinda makes sense. You quantize, which invariably means you lose some precision, but then you can finetune post-quantization to recover at least some of it. Neat idea.
- jimmySixDOF 3y agoWhich is itself a little counterintuitive as the arxiv papers they cite say models need to be pretrained from the ground up using 1- or 2-bit (or 1.58bit). It definitely adds some interesting data points for the open source community who are experimenting in every possible direction.
- r1chardnl 3y agoIs this similar to binary search or could it be a binary search where you're just excluding all possibilities that you know can't be the probable result that you're looking for?
- nyrikki 3y agoFeed forward networks are effectively DAGs With the smaller model size, quantized models have less accuracy and less stable training, but large models take advantage of increased parallelism. Binary search is more linear, circuits are a better lens than turing machines. While it still needs to be verified, if you look into what a uniform consent depth threshold circuit is, that will help. Although I guess binary weights may be in AC0 and not TC0, but that may not hold with billions of parameters.
- londons_explore 3y agoI believe the future is 1 bit models - for both training and inference. When people make custom silicon for 1 bit models, they'll find that it is sooooo much more power and silicon-space efficient to do 1 bit math than 16 bit floating point - like 100x or more. That extra model size will vastly overshadow any worse performance of the models.
- JHonaker 3y ago> That extra model size will vastly overshadow any worse performance of the models. ...What?
- sroussey 3y agoI think OP was referring to parameter size. You can make up for quantization by having more parameters.
- JHonaker 3y agoI know. I was being a bit mean, but they seem to be implying that more parameters even with worse performance is preferable. That seems backward to me. There is no reason for parameter size to be intrinsically valuable, and I think people are a bit infatuated with it.
- deleted 3y ago[deleted]
- cma 3y agoFor training how do you get any kind of meaningful derivative with it?
- twelfthnight 3y agoMaybe evolutionary algorithms instead? Hasn't proven super useful historically, but maybe at the scale of enormous LLMs it will be?
- littlestymaar 3y agoStupid question: how can you do deep learning on 1 bit? DL works by calculating a gradient and following it to minimize a loss function, but if all your parameters are 1 bit, there's no such thing as a gradient as you don't have differentiation at all. So how does it work?
- edflsafoiewq 3y agoNot sure what you mean, why wouldn't there be a gradient? There is a problem that small changes to weights may have no effect (eg ((0 + 0.3) + 0.3) + 0.4 = 0 if you have to round the result to 0 or 1 after every step), but to fix this I believe everyone maintains high precision weights for the backwards pass, from which the quantized weights are derived.
- scotty79 3y agoYou could treat fractional addition as probabilistic operation.
- Centigonal 3y agoNot a stupid question at all. Early quantization approaches just quantized a high-bit-precision pre-trained model - no need to calculate gradients on the quantized weights. BitNet[1] changed the game by training a low-precision model from scratch. It achieves this by keeping high precision in the gradients, optimizer state, and in "latent weights" which are then quantized on the fly. I don't really understand the finer details of how this works, so check out the paper if you're interested. This article's approach is interesting. They start by quantizing a pre-trained high-precision model, and then they fine-tune the quantized model using LoRA (which dramatically improves the performance of the quantized model). They don't talk about the bit depth of the values in the LoRA matrices, so it may be that those are higher bit-depth. [1] https://arxiv.org/pdf/2310.11453.pdf https://arxiv.org/pdf/2310.11453.pdf
- kromem 3y agoThe most exciting part about ternary or binary weights is the inevitable hardware revolution for AI dedicated chips that's going to result from it.
- imtringued 3y agoYour hardware already supports addition and subtraction and the tensor cores of NVIDIA GPUs are already fast enough to keep up. The only benefit is reducing memory capacity and bandwidth requirements.
- Matumio 3y agoYou mean we'll have hardware accelerated ternary instructions? https://www.intel.com/content/www/us/en/docs/intrinsics-guide/index.html#techs=AVX_512&text=ternary&expand=5438,5437 https://www.intel.com/content/www/us/en/docs/intrinsics-guid... (Okay probably those are not ready to be used as NN weights if the activations are not binary too, but... the gap to what CPUs already can do is getting smaller.)
- jstmm 3y agoIn both the '1-bit Model' and '2-bit Model' tables, the forward time (sec) for Llama2-7B with FP16 (full precision) is 0.1 s, whereas it's ~0.231, ~0.257, ~0.353 s respectively for HQQ (1-bit) / HQQ+ (1-bit) / Quip# (2-bit) meaning the FP16 model has ~3x lower inference time. On the contrary, in the BitNet b1.56 paper [0] the authors report their 7b model has 2.9x reduced inference latency. It's not clear to me what's happening here. Can someone explain why the 1/2bit HQQ/HQQ+ models are so much slower than the BitNet b1.56 models? [0] https://arxiv.org/pdf/2402.17764.pdf https://arxiv.org/pdf/2402.17764.pdf
- thatguysaguy 3y agoSounds like these guys didn't use custom kernels, but BitNet did.
- mobicham 3y agoThat's correct. Only the dequantization is done on CUDA, the matmul is done with Pytorch. If they put their kernels open-source we could re-use them!
- londons_explore 3y agoGPU's aren't really designed for 1 bit math... They don't perform much faster than floating point math. Whereas a custom ASIC or updated design of GPU could give massive speedups with 1 bit math.
- bee_rider 3y agoFor 1 bit math, at least it should be possible to populate every other bit of an integer type, right? Surely one could do better with a dedicated type for this, but at least we could pack 16 single-bit weights into a 32 bit int for addition, right?
- UncleOxidant 3y agoYes, exactly. Neither GPUs nor CPUs are setup for 1 bit math. Pulling 1 or 2 bits out of a word isn't all that straightforward on CPU or GPU - lots of shifting and masking. I wonder how long it's going to be before we see custom hardware for bitnets? I suspect we'll see it on FPGAs first.
- vladf 3y agoReally strong binary results. So strong it was fishy. I hope someone can explain my confusion below. > We compared the performance of the Llama2-7B model in three configurations: FP16 (full precision), HQQ (without fine-tuning), and HQQ+ (with adapter layers) using a group-size of 8. Interesting, what is "group-size of 8"? From their HQQ post (https://mobiusml.github.io/hqq_blog/ https://mobiusml.github.io/hqq_blog/), it's the block size at which they add scales (presumably 16-bit) and shifts (in that post, it's 8-bit). So for every 8 binary weights we have a 16-bit scale and 8-bit shift? > Fine-tuning with Low-Rank Adapters They say they inline the shift into the LoRA but how can you do this, block-wise, without increasing your LoRA rank by num-blocks (they claim to only use 1 additional rank)? Then, the reported 7B sizes, in GB: > 13.5 (fp16) 1.76 (HQQ 1-bit) 1.85 (HQQ+ 1-bit) 2.72 (quip# 2-bit) those numbers would make sense if it was _actually_ 1 bit. But if you include the overhead of 16-bit scales (and why is the shift inlineable into lora? still unexplained) it'd be more like 3-bit. From their HF page: > This version offloads the meta-data to the CPU, so only the binary weights and the low-rank adapters are stored in the GPU memory. Interesting, so we have to go back to CPU to rescale? Is this how they counted GB? This should have been clearly caveated in the table. I also am amazed they got latency lower than quip if they pingpong to CPU.
- danielhanchen 3y agoWhen one does quantization, it's done in blocks. Bitsandbytes uses a blocksize of 64 I think. W * scale + zero_point is needed for each group size of 8. So you need 2 numbers in fp16 for each 64 group of numbers. For BnB, you get 4.5bit approx since 64*4bit + 16bit + 16bit = 288/64 = 4.5. So 4bit is actually 4.5bit. For HQQ 1bit, a group size of 8 needs 2 fp16 numbers (you mentioned 8bit for shift). So you need 8 * 1bit + 16bit + 8bit for each group ie 32bits for each group size of 8. Or 4bits per param. I'm assuming the scale and zero_point are both moved to 8bit maybe so 8*1bit + 8bit + 8bit = 24bit / 8 = 3bits per param? "This version offloads the meta-data to the CPU, so only the binary weights and the low-rank adapters are stored in the GPU memory.", so the 8+8 scale / zero_point moves to the CPU. So GPU memory 1bit, but CPU meta data is the rest - very smart!
- vladf 3y ago
- danielhanchen 3y agoHighly respect HQQ team's work - same accuracy as GPTQ / AWQ and with no activation aware tuning calibration part - ie no more 3 hour calibration runs! A big fan of the 4bit ones especially, and the 3,2, and now 1bit ones! Also super cool idea of 1bit needing some calibration like AWQ - no calibration data shows very bad results, but with some LoRA adapters and finetuning, a great recovery of performance is possible. Planning to add support for this inside Unsloth to make all low bit finetuning 2x faster and save tonnes of VRAM!
- shrubble 3y agoI wonder if there is a correspondence to Sony's 1-bit DSD audio stream? Could hardware developed for this purpose be used for some part of the processing?
- codedokode 3y agoI was thinking about this idea (using only additions for inference) before too. The article says about using 1-bit values for weights, but there is also another option: use integers for weights, but allow the output of a neuron take only 0 and 1 (for example, 1 if output is > 0 and 0 otherwise). In this case we do not need multiplications as well, but cannot save memory for storing weights. Maybe it has some other advantages, who knows. Never had time to research this further.
- Greenpants 3y agoYou may be interested in the "binary step" activation function. This does what you're suggesting. In general, complex behaviour really takes a hit though using this for the activation function of a neuron (though I'm also not sure which papers show metrics on this being used for transformer models).