9 ms·
LLMs Unleashed: The Power of Fine-Tuning
- mickeyfrac 3y agoThe link to your terra cotta product, which I assume is the point of the article, is broken.
- deleted 3y ago[deleted]
- lucaspauker 3y agoHi, we checked and it works on our end. Could you try again here please: https://terra-cotta.ai/ https://terra-cotta.ai/? If not, could you share the error?
- stevelini 3y agoStep one - "upload your data" and we'll do cool stuff Ok. How do I upload my data? Can't find any link. Sign up for our newsletter. Nope. And we went to Stanford? I don't care. I'm not trying to be nasty, though it probably comes of like that. Assume I want to get started with your product and invite me in...
- SpaceManNabs 3y agoI like that your article was well cited. Fun read. Nothing stands out as too inaccurate. You should try a post on parameter efficient tuning next!
- treprinum 3y agoThis is 2020-level stuff. These days with emergent abilities in LLMs trained with over 1T tokens like GPT-4 a single-shot chain-of-thought beats most fine-tunings. I did research on transformer adapters i.e. parameter-efficient fine-tuning and that stuff is now completely obsolete outside of some restricted domains where small models can still perform well.
- jph00 3y agoI haven't seen any recent papers that show that fine-tuning is obsolete - I've only seen papers showing the opposite. I'd be very interested to see any papers that have demonstrated applications where fine-tuning is not effective nowadays, if you have any links. Here's an example of a paper that shows good results from fine-tuning for instance: https://arxiv.org/abs/2305.02301 https://arxiv.org/abs/2305.02301
- treprinum 3y agoThis Stanford seminar video can provide some references: https://www.youtube.com/watch?v=tVtOevLrt5U https://www.youtube.com/watch?v=tVtOevLrt5U
- jph00 3y agoThat video shows examples of new models with one-shot performance better than old models with fine-tuning. It doesn't compare new models with fine-tuning to new models without fine-tuning.
- treprinum 3y agoRight, the page 28 of this article mentions "On some benchmarks, task-general models (not explicitly trained to perform a task) surpass prior state-of-the-art performance held by a task-specific model" but they don't provide data for fine-tuning of their very large models: https://arxiv.org/pdf/2206.07682.pdf https://arxiv.org/pdf/2206.07682.pdf
- ssivark 3y agoWhat’s your take on the argument that fine tuning might be destroying model calibration and causing it to overfit to aspects of the finetuning dataset eg: causing it to appear more confident, rather than becoming more correct? See https://twitter.com/animaanandkumar/status/1681690654060187648 https://twitter.com/animaanandkumar/status/16816906540601876...
- jph00 3y ago> "The idea of fine-tuning has a strong research pedigree. The approach can be traced back to 2018, when two influential papers were published." The article refers to the BERT and GPT papers as the source of the fine-tuning idea. However, we actually first demonstrated it for universal models in 2017 and published the ULMFiT (Howard and Ruder) paper in early 2018. Prior to that, Dai and Le demonstrated the technique for in-corpus datasets. So it would be more accurate to say the approach can be traced back to those two papers, rather than to BERT and GPT. BERT and GPT showed the effectiveness of scaling up the amount of data and compute, and switching the model architecture to Transformers (amongst other things).
- newhaus1994 3y agoYeah, my understanding is the ULMFit paper was the “genesis” Of fine-tuning in the way we mean it now.
- arugulum 3y agoAs a researcher in the field, I agree with this characterization. I think it's more accurate the say that GPT and then BERT massively popularized and simplified the idea/approach. Prior to ULMFiT/GPT/BERT, fine-tuning usually meant freezing most of a model and tuning a small layer on top of it. (ELMo also fits somewhere in here, being a kind of halfway step). ULMFiT was a relatively lesser known work but to my knowledge one of the first to do 1) LM pretraining and 2) fine-tuning all the layers, albeit with some complexity (gradual unfreezing/different learning rates for layers). GPT simplified this massively by simply tuning all weights: no nuance about it. (Also added a classifier on top). BERT took this a step further, benefiting from the larger size of BERT-large and bidirectional attention, which works really well on the NLU datasets of the time along with a classifier head. (It's not until T5 that seq2seq for general tasks became prominent again.) The idea that you tune all the weights of what was then considered a massive model was considered somewhat excessive at the time, but BERT so massively dominated every other architecture and approach at the time that everyone switched over to it. (Adapters (alongside PAL), one of the first parameter-efficient tuning methods in the Transformer era of NLP, came out shortly after.)
- bravura 3y ago
- nullc 3y ago> Fine tuning is better for complex tasks where the model’s generated output must be accurate and trusted. uhhh. I understand what was intended there but while fine tuning may reduce the rate of hallucinations and make hallucinations more plausible, it's not magic accurate and trust-worthy dust. Unfortunately many people think this stuff is magic and care should be taken to not encourage people to confuse improvements with resolving the issue. One way of characterizing the LLM accuracy problem is that it often looks very accurate and convincing even when it is emitting nonsense. If you cast the problem in those terms-- as a problem of looking more trustworthy than it actually is-- fine tuning actually exacerbates the problem.
- phas0ruk 3y agoHelpful. I was thinking today about when it makes sense to fine tune vs use embeddings to feed into the LLM prompt and this helped solidify my understanding.
- DebtDeflation 3y agoExcept that the article didn't cover that distinction at all. It looked at (manual) prompt engineering vs fine tuning. What you are describing is Retrieval Augmented Generation (RAG) which is creating embeddings from a knowledgebase, doing a similarity search using an embedding of the search query, and then programmatically generating a prompt from the search query and the returned content. IMO, this design pattern should be preferred to fine tuning in the vast majority of use cases. Fine tuning should be used to get the model to perform new tasks; RAG should be used instead to add knowledge.
- joshka 3y agoRealistically this seems like a question that would be difficult to generalize an answer to without measuring it. Intuition is unlikely to yield a better result than actually trying it.
- bugglebeetle 3y agoAre there any good tutorials on fine-tuning the quantitized versions of the LLama models anywhere? I have a few NLP tasks I’d like to test out, with plenty of training data, but everything I’ve seen doesn’t seem generalizable enough or lacks necessary details.
- autokad 3y agoI've tried text generation on gpt3 and it was very very bad. has anyone done so and got good results? care to share the code?
- marcopicentini 3y agoWhat if we fine tune a model like LLaMA on all published research papers? Would be able to product new knowledge?
- capableweb 3y agoDepends on what you mean with "new knowledge". A lot of inventions are "just" novel combinations of things we already knew.
- 6gvONxR4sf7o 3y agoThe value of research isn’t just in the ideas. It’s in the grounding of ideas in fact. It might be able to suggest interesting experiments, but those experiments still need to be carried out.
- zbyforgotp 3y agoFinetuning is not useful for teaching new facts, the current solution for that is using RAG: https://zzbbyy.substack.com/p/why-you-need-rag-not-finetuning https://zzbbyy.substack.com/p/why-you-need-rag-not-finetunin...
- ramesh31 3y agoCan anyone provide a step-by-step ELI5 guide to fine tuning Llama? I still don't quite understand.
- treprinum 3y agoLook up LoRA.
- joshka 3y agohttps://replicate.com/blog/fine-tune-llama-to-speak-like-homer-simpson https://replicate.com/blog/fine-tune-llama-to-speak-like-hom...
- popohack 3y agohttps://huggingface.co/docs/trl/sft_trainer https://huggingface.co/docs/trl/sft_trainer and https://huggingface.co/docs/trl/using_llama_models https://huggingface.co/docs/trl/using_llama_models, supervised fine-tuning and another llama example with rlhf
- LASR 3y agoWe’ve found that 1-shot or few-shot methods with 3.5Turbo or 4 are vastly simpler and exceeds the quality of fine-tuned models from the GPT-3 era. We have some 100k context models too that can ingest entire documents. So right now, I would say fine-tuning is probably only useful for a very narrow set of use cases.
- danenania 3y agoEven 100k still seems very limiting for many applications naturally suited to LLMs. What if I want an AI assistant that is specifically trained on a large codebase? Or all my product’s docs (which might easily exceed 100k characters for a big project). Or one that knows the exact details of the entire tax code? Or one that knows every line of Dostoyevsky’s novels so I can have angsty existential conversations with it? Or that can fully remember conversations with me that stretch on for years? It seems like you’d need fine tuning for these kinds of use cases? Or am I missing something?
- treprinum 3y agoYou use a vector database with a query similarity search to find portions of documents that might be relevant to your prompt, rank them, take the top n, put them into the context and run through your LLM to generate an answer to your question.
- fzliu 3y agoThis paradigm applies for much more than just answering questions - I've found the in-context learning capabilities to be very relevant as well. For example, you can query a vector database and input that information into an LLM and ask it to make a prediction across the input data. https://zilliz.com/blog/ChatGPT-VectorDB-Prompt-as-code https://zilliz.com/blog/ChatGPT-VectorDB-Prompt-as-code
- treprinum 3y agoThat's pretty cool, thanks for sharing!
- Animats 3y agoThis is basically an ad for fine-tuning as a service. Can anyone offer an example of a free public-facing LLM which has been fine-tuned by adding much specific info about some narrow area? Say, one that knows all the PR about some car brand or fandom? Somebody must have tried that by now.
- treprinum 3y agoThe only known to me is LLaMA 2 to remove censorship using LoRA.
- Animats 3y agoHm. Why aren't those things all over the place?
- treprinum 3y agoThey kinda are, HuggingFace hub is already full of them. Most of them were released like 3-4 days ago so that might be a reason why you don't see them. I already put one of them into production.
- nickpsecurity 3y agoThe people that make them often post them or comment in the sub-Reddit below. I've linked to a search for uncensored models in there. Usernames of main people involved are faldore and The-Bloke. Most had Wizard Uncensored in their names. https://www.reddit.com/r/LocalLLaMA/search/?q=uncensored&restrict_sr=1 https://www.reddit.com/r/LocalLLaMA/search/?q=uncensored&res... People have posted comparisons showing it works pretty well. Some foundational models have enough censorship built in that they still try to resist it. You have to prompt those cleverly to do what you want them to do.
- fnordpiglet 3y agoThere are tons of them for other models, often Alpaca or LLaMA. This is a popular tool for finetuning Alpaca: https://github.com/tloen/alpaca-lora https://github.com/tloen/alpaca-lora
- zmmmmm 3y agoFine tuning seems to me to be dangerously close to a new snake oil of AI these days. The narrative goes, "look how awesome ChatGPT is, imagine how good it would be trained on just your company's documents". Which 1000% misses the point. ChatGPT is because (a) it is trained on almost nothing short of the entire corpus of human language ever created. At > 1 trillion parameters, it can have ~1000 parameters for every human on the planet. Let that sink in. And then (b) because it has been subjected to an unknown but likely massive amount of human reinforcement feedback. The idea that you can meaningfully impact the output of the model towards factual accuracy or logical correctness just by doing a small amount of fully automated training using a tiny corpus of company documents is seductive, but super far from robustly demonstrated as far as I'm aware. Yet this is the pitch being sold very often.
- janalsncm 3y agoWhat? Fine tuning has been a common technique for years now. Fine tuned BERT models were behind a lot of retrieval-based systems and they work well. A more recent example is stable diffusion fine tuned on specific subjects. Whether fine tuning can reduce hallucination is first of all a question which only pertains to decoders and which is highly dependent on how the model has been fine tuned.
- coffee_am 3y agoNoob question: when folks talk about fine-tuning LLM, do they usually fine-tune the encoder (of the prompt), the decoder (that generates the text) or both ?
- drdirk 3y agoBoth can be done. Fine-Tuning the prompt is cheaper and can be done on consumer hardware. Fine-tuning the LLM model weights is expensive and needs cloud support.
- coffee_am 3y agoThanks for the reply, but when you mean "fine-tuning" for the prompt, do you mean fine-tuning of the LLM encoder of the prompt right ? (The thing that transforms the prompt into a sequence of embeddings?) But that is not cheap/easy to train ... I know some systems also allow an extra fixed embedding parameter to the prompt, that can also be fine-tuned. But that is yet another thing that can be fine-tuned very cheaply.