10 ms·
Show HN: Finetune LLaMA-7B on commodity GPUs using your own text
I've been playing around with https://github.com/zphang/minimal-llama/ https://github.com/zphang/minimal-llama/ and https://github.com/tloen/alpaca-lora/blob/main/finetune.py https://github.com/tloen/alpaca-lora/blob/main/finetune.py, and wanted to create a simple UI where you can just paste text, tweak the parameters, and finetune the model quickly using a modern GPU.
To prepare the data, simply separate your text with two blank lines.
There's an inference tab, so you can test how the tuned model behaves.
This is my first foray into the world of LLM finetuning, Python, Torch, Transformers, LoRA, PEFT, and Gradio.
Enjoy!
- sea_temple 4y agoIs it possible to fine-tune using CPU only?
- antman 4y agoWhat do the training screenshot examples try to accomplish? In what way would the model be different after the example fine-tuning?
- Taek 4y agoLooks like you need about 120 GB to fine-tune the 65B model with this code at a sequence length of 512. How does the memory usage scale as the sequence length grows?
- lxe 4y agoLots of VRAM, but the method is the same. Here's someone who finetuned llama-30b on alpaca dataset for example: https://github.com/deep-diver/Alpaca-LoRA-Serve https://github.com/deep-diver/Alpaca-LoRA-Serve
- lxe 4y ago> How does the memory usage scale as the sequence length grows? That's a good question. I was under the assumption it's linearly proportional, but I can test it out I guess.
- Taek 4y agoI suspect its linear with a small constant factor.
- arpowers 4y agoReally takes that much vram??
- ioedward 4y agoNormally people split up the model across multiple GPUs, i.e. model/tensor parallelism.
- albertzeyer 4y agoThe attention implementation is in a way that the memory scales quadratically with sequence lengths. Overall, this is still a small factor compared to just the model weights, but at some seq lengths, this would dominate. By using flash attention, you can get memory requirement down to scale linearly with sequence lengths.
- joshxyz 4y ago
- yousnail 4y agomy instance doesn't seem impressed: So, I stumbled upon this Simple LLaMA FineTuner project by Aleksey Smolenchuk, claiming to be a beginner-friendly tool for fine-tuning the LLaMA-7B language model using the LoRA method via the PEFT library. It supposedly runs on a regular Colab Tesla T4 instance for smaller datasets and sample lengths. The so-called "intuitive" UI lets users manage datasets, adjust parameters, and train/evaluate models. However, I can't help but question the actual value of such a tool. Is it just an attempt to dumb down the process for newcomers? Are there any plans to cater to more experienced users? The guide provided is straightforward, but it feels like a solution in search of a problem. I'm skeptical about the impact this tool will have on NLP fine-tuning.
- lxe 4y ago> I can't help but question the actual value of such a tool. Is it just an attempt to dumb down the process for newcomers? Actually, you've hit the nail on the head here. I wanted something where I, a complete beginner, can quickly play around with data, parameters, finetune, iterate, without investing too much time. That's also why I've annotated all the training parameters in the code and UI -- so beginners like me can understand what each slider does to their tuning and to their generation.
- Taek 4y agoThis is exactly the sweet spot I'm looking for. Technical enough that I can play around, simplified enough that I'm investing an hour or two of my time instead of a whole weekend.
- fbdab103 4y agoSo you are annoyed that something targeted for beginners does not also cater to experts?
- yousnail 4y agome? re-read that s'il vous plaît
- 4y ago
- Tepix 4y agoVery interesting project, thanks. I see that DeepSpeed is on your TODO list. I wonder what the biggest LLaMA model is that can be fine-tuned on 2x RTX 3090 & 128GB RAM.
- lxe 4y agoYou can probably finetune a 13b one with that. Try these scripts: https://github.com/zphang/minimal-llama/#minimal-llama https://github.com/zphang/minimal-llama/#minimal-llama
- meghan_rain 4y agowhat if i want to finetune with long documents? say AI papers that are ~10 pages long on average? how would they be tokenized given that max_seq_length is 512?
- amrb 4y agoSplit your training data into chucks of text that make sense. A random dataset example https://huggingface.co/datasets/imdb https://huggingface.co/datasets/imdb
- meghan_rain 4y agoThanks, what does making sense mean? Be logically coherent (eg a paragraph of text in a document?) And does the training then create windows of ngrams on those chunks? Or what is the input/output? The reason I ask: If I had question/answer pairs, the question is the input, the answer is the output. What is the "output" when the input is just a (logically coherent) chunk of text?
- lxe 4y ago> What is the "output" when the input is just a (logically coherent) chunk of text? It probably won't change much if it's just a single sample. If you put in a large corpus of samples that repeat on the same theme, then the model will be "tuned" to repeat that theme. If you increase the number of epochs, you can overtrain it, meaning that it will just spit out the training data text.
- syntaxing 4y agoIs there any library that allows you to train with a Mac M1/M2? I know it will be slower but I rather spend money on a Mac Studio rather multiple graphic cards to get around the VRAM limitation.
- speedgoose 4y agoFor training you could rent a GPU server for a short period.
- te_chris 4y agoOr use something like google vertex to run a docker job on gpus
- tysam_and 4y agoPersonally I would recommend Colab or another notebook environment like a Lambda machine. Much cheaper, and simpler than a bare metal machine, data ingress/egress is hard though for Colab you can just mount a Gdrive. Unfortunately the API for training on M* chips (via MPS) is apparently still extremely buggy, so we have a ways to go before that is fully mainstream. And yes, I know that PyTorch just mainlined their mps support last week too...but from what I've heard the low level interface itself still needs some work. D:
- nwoli 4y agoJust fyi colab is way way more expensive now after the “credits” update than it was a year ago. Lamdalabs is about as cheap at this point
- capableweb 4y agoIf you're fine with using other individuals machines (meaning, you don't care about data privacy for the training set), using vast.ai is probably the cheapest way to do it today. But, quality of machines/network speed vary greatly as the machines are hosted by individuals around the world.
- deleted 4y ago[deleted]
- ALittleLight 4y agoReally interesting, thanks for sharing, I'm excited to try this out. Would it also be possible to just train the model from scratch on commodity hardware and how big of a difference in training time would that be?
- callesgg 4y agoNow, use this library to "bootstrapp the smarts of LLaMA from its own smartness" like this: 1. Ask it things. Let it answer. 2. Ask it to find errors in the answer it outputted and for it to correct the answer. 3. Use the original prompt and the corrected output as training data. This should, with each iteration make the model less and less likely to output statements that are self contradictions or obviously wrong, until the model can no longer spot its own faults.
- jointpdf 4y agoOr it will twist itself into a giant hairball of contorted logic, like GPT3.5 does when I (a human) encourage it to explain its errors.
- Drakim 4y agoI recall reading that when training AlphaZero they would start pitching it against itself doing millions of games in a few days, which worked great because there is an external metric (who wins the chess game) that would objectively be a good measure to train towards. But if you let an AI's approval be the metric, things turn a lot more fussy and subjective. The goal is not actually "to write a good answer without error" but actually "to write an answer that is approved by the AI". Those are very different goals, and as you keep using it you'll get a bigger and bigger divergence, until eventually the AI is just answering complete garbage nonsense that precisely hits certain sweet spots in the grading AI. This divergence of the target vs the actual human goal is a pretty interesting problem in AI safety research. I love the example where an AI trained to stay alive as long as possible in Tetris realized that pausing the game was the best strategy.
- Dwedit 4y agoThat wasn't an AI, that was a "Make the numbers go up" (lexagraphic ordering) system with TAS rewinding for short term bruteforcing.
- MattPalmer1086 4y agoInteresting, but the core point remains true. The algorithm optimises for something which may not entirely coincide with the creators intentions.
- reissbaker 4y agoThis is awesome! I noticed it said a prereq is >16GB VRAM — is that >= 16GB, or is it really explicitly greater than 16? Would be sweet to be able to finetune locally on, say, a 3080.
- deleted 4y ago[deleted]
- capableweb 4y agoI gave this a try and it seems to max out using about 12GB of VRAM on a RTX 3090 Ti.
- capableweb 4y agoTried the 30b-hf set too, but was too much (24GB available). 13b-hf works fine, maxing out at 17GB.
- Hedepig 4y agoSlightly tangential, is there some kind of crowdsourced effort to build training data for fine tuning? Alpaca used the training data built from gpt-3.5, so there are terms of use restrictions
- Phemist 4y agoIANAL, but wouldnt feeding the training data into alpaca and then having it output similar text not construe new data that is not copyrighted by stanford/openai/facebook? (It would be a significantly creative and novel worm to get the prompts working correctly..) Obviously you are also not bound by the openai terms of use, and Im not sure if stanford's terms of use are as broad and well-defined...
- Hedepig 4y agoThe terms of ChatGPT/GPT-3.5 explicitly state that one cannot use their data to construct competitive models
- NavinF 4y agoNot enforceable. All they can do is ban your ChatGPT account.
- YetAnotherNick 4y ago> Not enforcable Are you willing to assign a upper limit on this probability and bet for it?
- throwaway1851 4y agoOpenAI disclaims ownership interest in the model output. If a subscriber (who has a contractual relationship with OpenAI) chooses to generate outputs that could be used to train a competing model, and chooses to share those outputs with third parties, that is not prohibited by the agreement. Further, the data being shared belongs to the subscriber and can be licensed however they desire (though actually, model outputs may not be copyrightable at all). If a third party who does not have a contractual relationship with OpenAI chooses to take this data and train a competing model, they are using the data under valid license and have breached no obligation to OpenAI.
- holoduke 4y agoI am really an AI noob. But lets say i tried the 7b model for translations purposes with below acceptable results. Can i train the model with 1 million of translated sentences to improve the quality of the translation output?
- Method-X 4y agoCan you give an example on how you prompted the model? Your issue is probably related to that, but I would need an example to be sure. I've found the 7b Alpaca model [1] to work surprisingly well! Here's how you're supposed to prompt it: Below is an instruction that describes a task. Write a response that appropriately completes the request. ### Instruction: {instruction} ### Response: or Below is an instruction that describes a task, paired with an input that provides further context. Write a response that appropriately completes the request. ### Instruction: {instruction} ### Input: {input} ### Response: [1] https://github.com/cocktailpeanut/dalai https://github.com/cocktailpeanut/dalai
- lxe 4y agoBarebones llama is just a text completion model. You can give it a prompt like A conversation between a human and assistant Human: How old is the sun? Assistant: And it will complete it. Alpaca/Dalai are finetuned on a dataset that's formatted as this: ### Instruction: {instruction} ### Input: {input} ### Response: So even without pre-prompting in this format it's going to be heavily biased towards performing completions in this format anyways. It's always helpful to finetune on a preformatted prompt depending on what your task is.
- benob 4y agoYes you can finetune the model with reference output for any kind of language task. For translation, you are better off starting from a model specifically trained for that purpose such as facebook/nllb-200-distilled-1.3B. It will be faster and more accurate.
- wsgeorge 4y ago
- larodi 4y agohmm...I would think of PROLOG or other rule-based system as doing inference, but not of neural-network (which is essentially a mesh of matrix multiplications and functions). this statement that NNs do inference is not entirely correct IMHO.
- selfhoster11 4y agoI think you are unfairly downvoted for this, probably because I have just as strong opinion in the other direction: I see GPT (especially -4) as a kind of "killer Prolog-style inference engine". How does Prolog work? You: * pass it some predicates * specify rules about what the relationships mean and how various things are computed from the data in predicates * query a variable * answer pops out. How can GPT do this task? You: * pass it some predicates (structured machine-readable syntax or natural-language sentences) * specify rules (in natural-language sentences, though it helps to iterate on the wording a bit to make the rules more rigid, and more likely to provide the correct output ~every time) - you don't normally need to specify relationships explicitly because GPT can usually figure it out * include some additional "massaging" wording to get it reproducibly outputting the kind of result you want * query a variable (tell it what to find out/infer from the data) * answer pops out (in human-readable language or structured syntax). In some ways, they are very much alike. And GPT is much more natural to program with than Prolog.
- happycube 4y agoThey get there in completely different ways though. If there's no answer Prolog will fail out, and GPT* will usually make (stuff) up.
- larodi 4y agoexactly. PROLOG fails with the non-true inference, and this is why it's inference. otherwise we'd be calling all the high-school made up nonsense inference. it is absolutely not fair, established or not, to call non-linear regression inference. may be prognosis, maybe prediction, maybe just approximation. but inference is something that actually does logic, based on facts... and probabilistic theory infers only probability, not facts.
- la64710 4y agoCan this be used to programmatically train the model ? I got a big website that I would like llama to chew on and be aware of the content there so it can answer questions.
- capableweb 4y agoYes. Check `main.py`, rewrite it to load the text from whatever you want.
- mattfrommars 4y agoGreat job OP. Yesterday after managing to run local instance of alpaca.cpp and reading more about what alpaca is and how it got fine tuned on LLaMA, I began to wonder what will it take to fine tune with own set of data. With no real knowledge of LLM and only recently started to understand what LLM terms mean, such as 'model, inference, LLM model, intruction set, fine tuning' whatelse do you think is required to make a took like yours? This is for education purposes and love to take a jab on creating something like this and and write an inference - such as the dev behind LLaMa inference in Rush. >I am not familiar with HuggingFace libaries at all, why were they important in your implementaiton? > Gradio - I believe is the UI that allows to plugin different lLM models, I am familiar with text-generation-ui on GitHub that uses Gradio. >LORA I think further fines tines an model -- just like how LLaMa got fine tuned on instruciton set to produce Alpaca model.
- lxe 4y ago> With no real knowledge of LLM and only recently started to understand what LLM terms mean, such as 'model, inference, LLM model, intruction set, fine tuning' whatelse do you think is required to make a took like yours? This was mee a few weeks ago. I got interested in all this when FlexGen (https://github.com/FMInference/FlexGen https://github.com/FMInference/FlexGen) was announced, which allowed to run inference using OPT model on consumer hardware. I'm an avid user of Stable Diffusion, and I wanted to see if I can have an SD equivalent of ChatGPT. Not understanding the details of hyperparameters or terminology, I basically asked ChatGPT to explain to me what these things are: Explain to someone who is a software engineer with limited knowledge of ML terms or linear algebra, what is "feed forward" and "self-attention" in the context of ML and large language models. Provide examples when possible. I did the same with all the other terms I didn't understand, like "ADAM optimizer", "gradient", etc. I relied on it very heavily and cross-referenced the answers. Looking at other people's code and just tinkering with things on my own really helped. Through the FlexGen discord I've discovered https://github.com/oobabooga/text-generation-webui https://github.com/oobabooga/text-generation-webui where I spent days just playing around with models. This got me into the huggingface ecosystem -- their transformers library is an easy way to get started. I joined a few other discords, like LLaMA Unofficial, RWKV, Eleuther AI, Together, Hivemind and Petals. I bookmarked a bunch of resources but it's very sporadic. Here are some: - https://github.com/zphang/minimal-llama/#peft-fine-tuning-with-8-bit https://github.com/zphang/minimal-llama/#peft-fine-tuning-wi... - https://github.com/togethercomputer/OpenChatKit https://github.com/togethercomputer/OpenChatKit - https://www.cstroik.com/index.php/2023/02/18/finetuning-an-ai-language-model-on-consumer-hardware/ https://www.cstroik.com/index.php/2023/02/18/finetuning-an-a... - https://github.com/huggingface/peft https://github.com/huggingface/peft - https://github.com/kingoflolz/mesh-transformer-jax/blob/master/howto_finetune.md https://github.com/kingoflolz/mesh-transformer-jax/blob/mast... - https://github.com/oobabooga/text-generation-webui https://github.com/oobabooga/text-generation-webui - https://github.com/hizkifw/WebChatRWKVstic https://github.com/hizkifw/WebChatRWKVstic - https://github.com/ggerganov/whisper.cpp https://github.com/ggerganov/whisper.cpp - https://github.com/qwopqwop200/GPTQ-for-LLaMa https://github.com/qwopqwop200/GPTQ-for-LLaMa - https://github.com/oobabooga/text-generation-webui/issues/147 https://github.com/oobabooga/text-generation-webui/issues/14... - https://github.com/bigscience-workshop/petals https://github.com/bigscience-workshop/petals - https://github.com/alpa-projects/alpa https://github.com/alpa-projects/alpa
- spunker540 4y agoIs there a consensus yet on when you should fine-tune vs when you should use prompt engineering? It seems not that hard to include in your prompt “here is some examples, please write like this and follow the style set here”. While that may make every completion request more expensive (more tokens), it also seems like fine tuning these models can also be quite expensive. I’m curious if there are other trade offs besides cost— maybe quality achieved is better with fine tuning? Very interested to see how it all plays out. On the one hand a massive model like gpt4 can probably be prompted to match any style quite well, albeit costly, vs fine tuning a cheap model may get exactly what you want.
- wazer5 4y agoEven GPT-4 can only handle a few pages of text as prompt for examples. In most cases you'd want to fine tune.
- spunker540 4y agoI guess I’m just curious what the killer use-cases are for fine tuning. For example it seems like overkill to fine tune a Shakespeare model, because you can just say “write like Shakespeare” and it already knows what you want.
- ttoinou 4y agoI guess you'd want to fine tune for content that wasn't already parsed before
- underlines 4y agoto my understanding there are 4 levels to add information: 1. train a model 2. fine tune a model 3. create embeddings for a model 4. use few shot prompt examples at inference time These have decreasing resource need, but also decreasing quality. For example, the GPT-3 API (not yet the GPT-4 API) has a functionality to send it your own embeddings, for example of your own source code documentation. Then you can query GPT-3 and it "knows" your source code doc and answers specifically with that in mind.
- nathanasmith 4y agoI've built up a large personal library of hand made Anki flashcard decks over the years. This looks like just what I need to train a model on those decks.
- akrymski 4y agoHow does Llama compare to the actually open source Flan UL2?
- pww123 4y ago[dead]
- pww123 4y ago[dead]