10 ms·
How to Finetune GPT-Like Large Language Models on a Custom Dataset
- artembugara 3y agoHave a question to the Generative AI experts here. So, I can use smthg like GPT-4 to label data and then use that as a train set for my own LLM, right? EDIT: adding this from OpenAI Restriction TOS: "(iii) use output from the Services to develop models that compete with OpenAI;"
- montenegrohugo 3y agoYup, totally. This is a form of knowledge distillation. Openai, or other foundational model providers, can't really do anything about it.
- cookieperson 3y agoWell they can sue you and bankrupt you by delaying trial for a decade. That's how the US patent system works anyways...
- sanxiyn 3y agoSue on what grounds? It will be quickly dismissed.
- mind-blight 3y agoThat is not how the US legal system works. You can sue someone for and regardless of merit, and they will have to defend themselves. That costs time and legal fees. If they lose, they can appeal, and continue appealing. If it's baseless, they'll lose, but you still spend a lot of money and time dealing with the lawsuits.
- cookieperson 3y agoIf by quickly you mean 5 to 10 years of paying a retainer on a lawyer sure. Even if you win the case you lose in life. Most individuals can't afford 500k in legal fees to have them be reimbursed years later. Big companies have lawyers on staff at a discount and they play these games every day. This happens with illegal things all the time. IE manager sexually harasses someone on video or something, it's some CEOs nephew who did it, so they fire the person who got harassed. The person who got harassed now has to aquire legal counsel on top of paying relocation claw backs etc. Few years ago by and the person who was in the right is trying to hold down a job, a family, and the stress of the legal battle. The company offers to settle two years in for 50k and 99% of people take it, sometimes at a loss. Also, getting employed is a lot harder when a background check reveals suing a previous employer or really any company, because shocker, most companies do illegal shit regularly... So it's almost always best to settle I realize I painted a picture pretty far from my previous statement but I figured you were new in your career and could benefit from an allegory of how stuff like this goes down.
- wodenokoto 3y agoIt is my understanding that this is how “alignment” works. That is, openAI paid people to chat with their LLM to fine tune it and then other LLMs use chatgpt to generate training data to align their models.
- visarga 3y agoThere are three ways 1. make your own RLHF dataset - like OpenAI and Open Assistant 2. exfiltrate data from a bigger/better LLM - Vicuna & family 3. use your pre-trained LLM to generate RLAIF data, no leeching - ConstitutionalAI, based on a set of rules instead of labelling examples
- cubefox 3y agoI wonder whether these approaches fit into the above categories: https://arxiv.org/abs/2305.13735 https://arxiv.org/abs/2305.13735 https://arxiv.org/abs/2305.11206 https://arxiv.org/abs/2305.11206
- snickmy 3y agoIndeed, fine tuning with either synthetic data (as you are proposing) or human review works like that. you can read more here: https://huggingface.co/blog/rlhf https://huggingface.co/blog/rlhf
- fallingmeat 3y agoThat is against their ToS though if you use your new LLM commercially.
- pmoriarty 3y agoSo what are they going to do about it?
- jstummbillig 3y agoThat escalated quickly.
- fallingmeat 3y agoGreat question! I don’t know the end game there. Maybe if they suspected their model was used they would sue, and in discovery find you used their model for training?
- visarga 3y agoMaybe we don't need to worry, OpenLLaMA is under training right now. It will be the commercial version of LLaMA. > Update 05/22/2023 > We are happy to release our 700B token checkpoint for the OpenLLaMA 7B model and 600B token checkpoint for the 3B model. We’ve also updated the evaluation results. We expect the full 1T token training run to finish at the end of this week. https://github.com/openlm-research/open_llama https://github.com/openlm-research/open_llama So we could develop on LLaMA for now and switch to OpenLLaMA later.
- bottled_poe 3y agoNothing until it’s worth their while.
- postsantum 3y agoMS lawyers have a good track record at sending out those scary cease&desist letters
- 3y ago
- foobarbecue 3y agoIs "ca" "can" or "can't"?
- artembugara 3y agocan
- notpublic 3y agonot an AI expert but from a talk I recently heard... if there is a mismatch in training data between the "teacher" LLM and "student" LLM, you risk teaching the student to hallucinate or to ignore information
- moffkalast 3y ago> I can use smthg like GPT-4 to label data and then use that as a train set for my own LLM, right? Yes, almost all improved LLama models are tuned exactly that way (trained on examples of questions and answers from say GPT 4). If OpenAI stole copyrighted works to train their models it is morally fair game to do the same to them regardless of their TOS. It's not like they can prove it anyway. Plus there's the other point where they also say that everything generated by their models is public domain, so which one is it eh?
- sirsinsalot 3y agoThis ... but we all know business is corrupt. The current attempts to spur on regulation by OpenAI is moat building
- ehnto 3y agoWe were complacent while it happened because OpenAI wasn't a business, it wasn't seen as unethical to use community work to contribute to community research. Now they're entrenched and pulled the rug out from the community, whilst also trying to shut the door on anyone else. Just a really disappointing series of events, the money and profit were never the big issue.
- jrm4 3y agoI'm a lawyer so one should never break the law. Nonethless, I can observe and predict that non-consensual "open sourcing" of these models would likely end up probably the best and safest way to do all of this stuff.
- Fgehono 3y agoBecause by training it they created something new. I don't mind just making a point. But I don't think they mind. I don't believe that this type of model training is able to be bleeding edge which should guarantee that openai has enough motivation to continue the development and having a healthy competition
- sp332 3y agoIt's against the terms of service to do the generation, but the generated text is not copyrighted. Those are different things.
- chaxor 3y agoYes, and in fact that's the best method available if you want good performance. I would suggest using a local open source model to do this however, to cut down on costs and make it far simpler to deal with than the unwieldy OpenAI systems. https://arxiv.org/pdf/2305.02301.pdf https://arxiv.org/pdf/2305.02301.pdf
- Obscurity4340 3y agoThis looks like the Orion broswer logo
- quickthrower2 3y agoWhen is fine tuning worth it, rather than just prompt engineering?
- messe 3y agoWhen you're starting to run into context limits.
- tstrimple 3y agoFrom what I've seen, it's when embeddings get too large for the token limit or the embeddings drive the cost up too much because you're always operating near the max token limit. In those cases, it may be worth the up front training cost and slightly higher per-token cost to dramatically reduce the amount of tokens in the average request. If you're building a higher throughput solution, the difference in cost can be quite large.
- snovv_crash 3y agoIf you want to teach it eg. all of the text in your private training manuals and internal documentation, which wouldn't fit in the input token size.
- heliophobicdude 3y agoI think these are two very separate concepts. What we are mostly seeing when it comes to fine-tuning is making a model promptable. Models like LLaMA or the original GPT3 weren't promptable. They were fine-tuned with demonstration data that looks like a prompt input, prompt output. See below: { "instruction": "What would be the output of the following JavaScript snippet?", "input": "let area = 6 * 5;\nlet radius = area / 3.14;", "output": "The output of the JavaScript snippet is the radius, which is 1.91." }, [1] Prompt engineering is really just carefully designing what inputs and outputs on a prompt-ready model work best. I highly recommend skimming this RLHF article and looking for the parts where it talks about demonstration data [2] 1: https://github.com/sahil280114/codealpaca/blob/master/data/code_alpaca_2k.json#LL22C7-L26C9 https://github.com/sahil280114/codealpaca/blob/master/data/c... 2: https://huyenchip.com/2023/05/02/rlhf.html https://huyenchip.com/2023/05/02/rlhf.html
- swalsh 3y agoHow does this compare to fine tuning something like BERT?
- theaniketmaurya 3y agoI would say similar since the building block is the transformer for both. In this blog post, the fine-tuning strategy used is Adapter. It basically adds a learnable layer to the Transformer block.
- slenocchio 3y agoCan someone explain why I'd want to use fine-tuning instead of a vector database (or some other way of storing data/context)?
- mgfist 3y agoFirst reason that comes to mind is you can make much smaller models, which helps with latency, cost and may enable you to run the model locally.
- pid-1 3y agoI've been playing with using documents as OpenAI embeddings for the past weeks and, at least for my use case, the results are meh. It seems sometimes just using context is not enough. My next step is to play with fine tunning, but I have no results to report yet.
- santiagobasulto 3y agoI'd be very interested in knowing the outcome. Do you blog anywhere (or post on social)?
- deforciant 3y agohave you tried other models to generate embeddings? I am going to that direction too to create an additional layer of helpers for search. Also, thinking if the document is not too big, it might fit into the initial context with the prompt
- akiselev 3y agoTry using InstructXL for embeddings. It’s got a more complex prompt structure for generating embeddings which might be more useful
- potatoman22 3y agoIf the documents are large, try embedding smaller portions. If there's a heavy domain vocabulary, you might need a custom model.
- 3y ago
- mercurialsolo 3y agoWhile the fine-tuning pipeline is fairly straightforward for tuning and building custom models, the RLHF pipeline doesn't look to be as straightforward. Creating a dataset for RLHF seems like a fairly labour intensive exercise especially if your model is tuned to do work like code generation ? What about the Replit Ghostwriter? Did it have a RLHF phase?
- nico 3y agoWhat is the main difference between training and fine tuning? Can you start with a model trained only in producing the letter a, and then fine tune it to learn b, then c, then words, sentences, etc?
- worldsayshi 3y agoYeah, since fine tuning seems to be so much more cheaper than training why haven't OpenAI fine tuned ChatGPT on data past 2021?
- ajb117 3y agoMy guess is that it's because they've already done RLHF on top of the standard next token prediction. In other words, they can't cheaply fine tune ChatGPT without undoing the RLHF objective by training on next token prediction with post-2021 data, and then retraining with RLHF to make sure it still gives good human-like output. I mention the "undoing RLHF" since it's not uncommon for fine-tuned models to increase in error in the original training objective after being fine-tuned with a different one. I think people saw this happen in BERT. Also ChatGPT is almost certainly huge.
- heliophobicdude 3y agoOne argument is that it can contaminate training data from output of itself or other models. We have already documented evidence of the effect of this. In the GPT-4 technical report [1], they reported contamination of humaneval data in the training data. They did measure against a "non-contaminated" training set but no idea if that can still be trusted. Why would this matter? We can have seemingly strong benchmarks for containments but measures poorly against new and quarantined information. Classic over fitting. Another argument is that data being put out there could very much be wrong and the amounts of it amplified by other models. Take a look at this sample of demonstration data for codealpaca [2]. Not only is its output wrong but bad practices like,making up a random computation without it having access to a place to run a calculation, teaches the model these type of responses are ok. { "instruction": "What would be the output of the following JavaScript snippet?", "input": "let area = 6 * 5;\nlet radius = area / 3.14;", "output": "The output of the JavaScript snippet is the radius, which is 1.91." } 1: https://cdn.openai.com/papers/gpt-4.pdf https://cdn.openai.com/papers/gpt-4.pdf 2: https://github.com/sahil280114/codealpaca/commit/0d265112c705f88848783dfdf9bdca9df64d8e35#diff-0585315a2292ca9c3186c41e52ce9427a39b5f595d4bee6aae15e9443c95051b https://github.com/sahil280114/codealpaca/commit/0d265112c70...
- hospitalJail 3y agoHas anyone tried to use this? The guide obv didn't make usable code and the github looks nearly unrelated. I'm somewhat surprised there isnt a parameter for 'input_data' and 'output_data' and it returns a trained model. I can't figure out why there is so much boilerplate when that stuff could be contained as parameters.
- lyapunova 3y agoI got stuck. Try these: https://huggingface.co/blog/stackllama https://huggingface.co/blog/stackllama, https://huggingface.co/blog/trl-peft https://huggingface.co/blog/trl-peft, https://huggingface.co/blog/hf-bitsandbytes-integration https://huggingface.co/blog/hf-bitsandbytes-integration
- akrymski 3y agoThese NanoGPT based models are great, thank you for contributing to OS. Would love to see this ported to CPUs ala llama.cpp. Any plans in that direction?
- nomagicbullet 3y agoIs there are Dreambooth equivalent for fine-tuning ChatGPT as there is for Stable Diffusion? I have to imagine that if we can add custom data to a DL text-to-image model, we should be able to do the same with a text-to-text one. Edit to add: There are a number of Google Colabs for fine-tuning SD and I wonder if there are (or if it is technically feasible) to accomplish the same with other txt2txt models.
- SparkyMcUnicorn 3y agoThese aren't for ChatGPT, but work on LLaMA, Vicuna, etc. https://github.com/oobabooga/text-generation-webui/blob/main/docs/Training-LoRAs.md https://github.com/oobabooga/text-generation-webui/blob/main... https://github.com/zetavg/LLaMA-LoRA-Tuner https://github.com/zetavg/LLaMA-LoRA-Tuner https://github.com/h2oai/h2o-llmstudio https://github.com/h2oai/h2o-llmstudio https://github.com/rhulha/lora https://github.com/rhulha/lora
- a5huynh 3y agoIf you're running the text-generation-webui (https://github.com/oobabooga/text-generation-webui https://github.com/oobabooga/text-generation-webui) it has the ability to train LoRAs. It'll require a beefy GPU but I've seen some fun examples like someone training a LoRA on Skyrim books.
- stoptrlling 3y agoAnyone knows the computational cost of training with these LoRa designs? Given that we are talking about rates of token per seconds, it seems training a bigger dataset could be extremely expensive
- t-vi 3y agoThe adapter and LoRa have a drastically fewer parameters, so one might expect that forward + backward is roughly 2x the cost of forward. Then (as far as I know), in contrast to generation, training is done on the entire output of the transformer (so all tokens of the full input) rather than serially token-by-token (in the RNN days, this was called teacher-forcing), so that may give you a significant boost in the tokens per second rate over generation.
- jpe90 3y agoWould it be feasible to fine-tune a large, capable model (like the recent LIMA) on the source code (and maybe a few high quality libraries) of a niche language, such that it's much better at helping you write and understand it? Imagine how many doors it would open if you could fine-tune models capable of writing language bindings for you and keeping them up to date.
- tazjin 3y agoTotally. GPT-4 can already do this, untuned, on niche languages and libraries. One of the main problems is still that you don't know when it's hallucinating a function or whatever though.
- sandGorgon 3y agohas anyone here used EasyLM ? it seems the most used for the best finetuned models out there.
- winstonprivacy 3y agoSounds interesting. Curious if there is a tutorial for this.
- zhwu 3y agoIt seems training the Vicuna on custom dataset could be quite easy as well, according to the following: https://github.com/skypilot-org/skypilot/tree/master/llm/vicuna#training-vicuna-with-skypilot https://github.com/skypilot-org/skypilot/tree/master/llm/vic...
- lyapunova 3y agoI have been working in this space for quite a while and while I think the beginning of pytorch lightning meant well, it seems modern use-cases have outgrown it. These days, when I see content from Lightning AI, I prepare for a contrived approach to doing something that fits within the ecosystem. I can't help but feel they are trying to induce "vendor lock-in" where there really isn't a business case for it... Anyways, I tried to follow these steps and hit a dead-end. I have to say the content put out by huggingface is always way more straightforward and gets me to where I need to be when I want to spin up quickly.