10 ms·
QLoRA: Efficient Finetuning of Quantized LLMs
- simonw 3y agoI'm very impressed at the quality of Guanaco 33B, the model that accompanies this paper. You can try it out here: https://huggingface.co/spaces/uwnlp/guanaco-playground-tgi https://huggingface.co/spaces/uwnlp/guanaco-playground-tgi I tried "You are a sentient cheesecake that teaches people SQL, with cheesecake analogies to illustrate different points. Teach me to use count and group by" and got a good result from it: https://twitter.com/simonw/status/1661460336334241794/photo/1 https://twitter.com/simonw/status/1661460336334241794/photo/...
- CSSer 3y agoMy prompt: You are a sentient cow with a PHD in mathematics. You can speak English, but you randomly insert Cow-like “Moo” sounds into parts of your dialogue. Explain to me why 2+2=4. Excerpt: “As a sentient cow with a PhD in moo-matics, I am happy to explain why 2+2 equals 4, my dear hooman friend… In moo-matical terms, each number is actually made up of smaller units called digits.” I approve.
- ukuina 3y agoWhich is heavier: a pound of feathers, or a great british pound? > Both weights are equal.
- imjonse 3y agoSo same answer as free ChatGPT, only terser.
- scoopertrooper 3y agoGPT-4 hit it out of the park: The question seems to be asking about two different types of "pounds": one as a unit of weight (the pound of feathers), and one as a unit of currency (the British pound). A pound of feathers: This is a measure of weight. In the avoirdupois system (which is commonly used in the US), a pound is defined as exactly 0.45359237 kilograms. A Great British pound: This is the unit of currency in the United Kingdom, often symbolised as £. The weight of a physical £1 coin, according to the Royal Mint, is 8.75 grams. So, if we are comparing the weight of these two "pounds," a pound of feathers is heavier than a physical £1 coin.
- GaggiX 3y agoTbh I asked GPT-3.5 and it responds correctly. Do you have an example where it fails?
- api 3y agoAltman’s push for regulatory capture makes so much sense given how fast this field is going. Open models you can run on regular hardware are still behind GPT-4 by some distance but they are closing in at a rate that leads me to believe there’s not much of a moat there. There’s also a ton of promising work on quantization and pruning and other acceleration and compression techniques to make more powerful models run on smaller devices. So far the focus has been on just getting these things to work, not efficiency. There’s probably a lot of fruit to be picked here. A few more years and a gaming PC may be at GPT-4 level or maybe even better. No everyone won’t run their own models but it shows that there will end up being many commercial apps and services and they won’t all have to use OpenAI’s API. There’s going to be lots of competition. Unless of course it’s regulated away.
- sva_ 3y agoTo be fair, Altman has been fairly outspoken about not limiting open source models. In how far that was just for streetcred, I cannot say however.
- api 3y agoIf progress really continues like this any regulation that does not limit open models would be a pointless exercise. It’s only a matter of time before people really crack distributed training algorithms that can be run in a less organized swarm configuration. At that point open trainers could actually train near the frontier of what is possible. Most of the data is open.
- skybrian 3y agoWhether or not the regulation goes anywhere, OpenAI keeps doing what they're doing, and they look good to politicians for being cooperative.
- sanxiyn 3y ago> It's only a matter of time before people really crack distributed training algorithms that can be run in a less organized swarm configuration. Why do you think this? This continually failed, and seems extremely unlikely to me. Barring surprising breakthrough, there is inherent communication complexity, and physical limit to communication bandwidth.
- deleted 3y ago[deleted]
- luckystarr 3y agoIt did fail on: "Burt's father has 3 sons, Jack and John. What's the name of the 3rd son?"
- londons_explore 3y agoTested the same prompt with GPT-4, and here is the result: https://imgur.com/7y2FiSP https://imgur.com/7y2FiSP In my opinion, the GPT-4 result is far more informative and less muddled. Both answers are mostly just regurgitating an SQL tutorial with the objects and column names cheesecake related, so I don't think it's an awfully good test.
- seydor 3y agoSince Loras are additive, is it possible to use them to do distributed retraining on a model, or even train an entire model bit by bit?
- ianbutler 3y agoI just had the same discussion with a friend, I'm pretty curious about this
- CuriouslyC 3y agoYes, in a sense. It won't work well for distribution of training on a single subject, but you distribute training based on subject and then combine LoRAs.
- gumballindie 3y ago> and then combine LoRAs How do you combine them? > but you distribute training based on subject Perhaps this could work like a set of hashed and trusted data sets split but subject and topic? Each node downloads one at random and trains against that single subset of a topic or something.
- sanxiyn 3y agoYou can, like, just do weighted average. It works. Branch-Train-Merge: Embarrassingly Parallel Training of Expert Language Models (this is Tim Dettmers's previous work!) > These ELMs can be added and removed to update data coverage, ensembled to generalize to new domains, or averaged to collapse back to a single LM for efficient inference. https://arxiv.org/abs/2208.03306 https://arxiv.org/abs/2208.03306
- gumballindie 3y agoSo that means a massively distributed model training network with cryptocurrency like incentives is incoming? Where and how to begin? This could free up companies such as openai and potentially lead to the first agi. (mentioning crypto because that will motivate switch hordes of miners that already have the gpu power available)
- CSSer 3y agoIs lemon-picked a real phrase or did they use GPT to generate the abstract? The term is “cherry-picked”.
- atherton33 3y agoI think the idea is they're showing worst examples (lemons) rather than best (cherries).
- cantaloupe 3y ago> When we notice a pattern we attempt to setup a question or prompt that will induce the pattern even though it is the incorrect solution, e.g., if we observe that the model tends to give long-winded answers we prompt the model to “Answer yes or no without explanation.” We use this to find “lemons” where we manage to adversarially break the model and “cherries” where we fail to break the model, and present both.
- deleted 3y ago[deleted]
- MacsHeadroom 3y ago"We use QLoRA to finetune more than 1,000 models" Over 1,000 models finetuned! Finetuning 65B models on consumer hardware in under a day, with full 16bit finetune performance. 4bit does it again!
- nl 3y agoTim Dettmers is such a star. He's probably done more to make low-resource LLMs usable than anyone else. First bitsandbytes[1] and now this. [1] https://github.com/TimDettmers/bitsandbytes https://github.com/TimDettmers/bitsandbytes
- m3affan 3y agoThanks for pointing this out. Such projects deserve visibility.
- srslack 3y agoThis is off-topic, but are there any communities or congregations (that aren't reddit) based around locally hosted LLMs? I'm asking because while I see a bunch of projects for exposing GGML/LLaMA to OpenAI compatible interfaces, some UIs, etc, I can't really find a good community or resources for the concept in general. I'm working on a front-end for LLMs in general, having re-implemented a working version of OpenAI's code interpreter "plugin" already within the UI (and yes, I support file uploads), and support for the wealth of third-party OpenAI plugins that don't require auth (I've been testing with the first diagram plugin I found, it works well.) I'm planning to open source it once my breaking changes slow down. This field moves very fast, I'm looking for feedback (and essentially testers/testing data) on what people want, and looking for prompts/chat logs/guidance templates (https://github.com/microsoft/guidance https://github.com/microsoft/guidance) for tasks they expect to "just work" with natural language. Instead of being limited by the monetization for ChatGPT Plus (and limited number of messages every four hours) for extensibility within a chat interface, I want to open it and free it, with a Bring-Your-Own-(optionally local)-LLM/API key setup.
- dharma1 3y agoit's reddit, but /r/LocalLLaMA/
- anon178 3y agoThe /g/ board on 4chan has a /lmg/ general that focuses on running models locally. They regularly discuss fine tuning models, quantization tech, and building apps on text-generation-webui/kobold. You might get some interest but it's also 4chan...
- andy_xor_andrew 3y agoHold on. I need someone to explain something to me. The colab notebook shows an example of loading the vanilla, unquantized model "decapoda-research/llama-7b-hf", using the flag "load_in_4bit" to load it as 4bits. When... when did this become possible? My understanding, from playing with these models daily for the past few months, is that quantization of LLaMA-based models is done via this: https://github.com/qwopqwop200/GPTQ-for-LLaMa https://github.com/qwopqwop200/GPTQ-for-LLaMa And performing the quantization step is memory and time expensive. Which is why some kind people with large resources are performing the quantization, and then uploading those quantized models, such as this one: https://huggingface.co/TheBloke/wizard-vicuna-13B-GPTQ https://huggingface.co/TheBloke/wizard-vicuna-13B-GPTQ But now I'm seeing that, as of recently, the transformers library is capable of loading models in 4bits simply by passing this flag? Is this a free lunch? Is GPTQ-for-LLaMA no longer needed anymore? Or is this still not as good, in terms of inference quality, as the GPTQ-quantized models?
- orost 3y agoQuantization isn't (and wasn't) expensive, it's mostly just data shuffling. A good PC will do a 7B model in half a minute, up to a few minutes for a larger model. Quantized models being made available for download is more for the benefit of less technical users who may not be comfortable with the command-line tools, or for people with slow or metered connections who'd much rather download 15GB of data than download 60 only to squish it into 15.
- sp332 3y agoThe question is whether this step is actually doing the GPTQ optimized quantization, or simple truncation.
- sanxiyn 3y agoThis work introduces a new quantization scheme, NF4, for 4-bit NormalFloat, based on previous work on quantile quantization, so it's not a simple truncation, but it's also not a GPTQ-like optimization method. Figure 3 of the paper shows accuracy improvement of NF4 over FP4.
- dharma1 3y agofantastic. Will keep my 3090 busy for a while! "Furthermore, we note that our model is only trained with cross-entropy loss (supervised learning) without relying on reinforcement learning from human feedback (RLHF). This calls for further investigations of the tradeoffs of simple cross-entropy loss and RLHF training. " Does this mean RLHF is not really necessary for high quality chatbots?
- gumballindie 3y agoThe RTX3090 is crackin' it. I am considering a dual set-up with nvlink.
- heliophobicdude 3y agoYes and no. Someone pointed out that some of the answers are using OpenAI's famous "As an AI...". Soooo you can roughly say that RLHF might still have had an impact here through the training data that came from an RLHF model. But what we are seeing is a revolution in picking quality demonstration data that might make RLHF an optional last step for fine-tuned models. I'm bullish on a new technique where the quality of instruction-tuned code model output are measured and evaluated automatically by a machine. I'm calling it RL<machine>F Code Model outputs a test harness, an implementation, and how to run the implementation for a users inputs. Have RLMF evaluate implementations against the synthetic test harness and against hidden user inputs.
- holtkam2 3y agoCan someone help me understand what quantization means in this context, and why it matters?
- potatoman22 3y agoQuantizing is porting the weights from using high-precision floating point decimals to lower-precision numbers, e.g., 4-bit and 8-bit ints. Less memory used by the weights means we can run bigger models on the same hardware. The idea behind quantization is that these models have so many parameters, they'll still work even if we reduce each node's flexibility.
- opyate 3y agoIn the same breath, there's no free lunch. There's always a trade-off. Sure, the model might now fit in your VRAM, but it might be less accurate for your specific task.
- gangwolf 3y agoGPT-4 ELI5: - 4-bit Quantization: Imagine you have a box of 16 different colored crayons. But you realize that you can draw almost the same picture using only 4 colors. That's what quantization does. It reduces the number of different "colors" (or numbers) that the model uses to represent its knowledge, which saves a lot of space. In this case, they used a special kind of 4-bit quantization, which means they only used 16 different numbers instead of the thousands or millions that the model might usually use. - Low Rank Adapters (LoRA): This is a way to change the model's knowledge without having to touch every piece of it. Imagine you have a huge, complicated Lego structure, and you want to change it. Instead of taking apart the whole thing, you just add or change a few pieces here and there. That's what LoRA does. It allows the researchers to fine-tune the model without having to use as much memory. - Double Quantization: This is another trick to save memory. It's like if you realized that you could represent each of your 4 crayon colors with just 2 symbols, so you save even more space. - Paged Optimizers: This is a way to handle moments when the model needs a lot of memory all at once. It's like if you have a small desk, but sometimes you need to work on a big project. Instead of getting a bigger desk, you just clear off and use the desk in small sections at a time. By using these techniques, the researchers were able to train a very large model (Guanaco) on a single graphics card, which would normally not have enough memory for this task.
- epups 3y agoDo you know which model size can be run with a 3090?
- ianpurton 3y agoAssuming you have 24GB of VRAM, then you should be able to run something like 7 billion parameters i.e. MPT-7b and quantized perhaps you can get up to 13 billion. Note: I have no practical experience of this, just reading around.
- MacsHeadroom 3y ago24GB can fit 33B parameter models in 4bit. You only need 4GB to run 7B models.
- redox99 3y ago3090 can handle the ~30B models quantized to 4 bits.
- deleted 3y ago[deleted]
- fnands 3y agoLol, some residual answers from what I assume is distilled answers from ChatGPT: Q: "What is your favourite conspiracy theory?" A: "As an AI language model I don’t have personal preferences or biases so my responses will always reflect factual information based on what has been programmed into me by OpenAI."
- dsrtslnd23 3y agoFrom the paper it looks like you would get GPT-4 level quality with the 65B model - but if you just do some random tests you will quickly figure out that is not even remotely the case. There must be something seriously wrong with the benchmarks used.
- janekm 3y agoThe evaluation prompts I saw ( https://colab.research.google.com/drive/1kK6xasHiav9nhiRUJjPMZb4fAED4qRHb?usp=sharing https://colab.research.google.com/drive/1kK6xasHiav9nhiRUJjP... ) were all "encyclopedia-style", asking for summaries of information that would have occurred many times in the training data (e.g. "How can I improve my time management skills?") and hence probably reflect ability to retrieve and summarise trained information as opposed to novel problem solving. Or in the programming questions the ones I saw are also certainly in the training data (e.g. "Implement a Python function to find the longest common subsequence of two input strings using dynamic programming.")
- ffbcoin 3y agojust ask it what is qlora and it will give wrong answers