4 ms·
They test two fine tuning tasks in the article - reliable output formatting and custom tone. These are two tasks (reliable output formatting in particular) that
by joewferrara 3y ago
They test two fine tuning tasks in the article - reliable output formatting and custom tone. These are two tasks (reliable output formatting in particular) that are advertised regularly as areas where fine tuning an LLM should work. The goal is not to change what the LLM knows, but to change how the LLM communicates what it knows. In theory the user wants to leverage the LLMs knowledge base and using the different output format is more useful to the user.
The hard question IMO is the question of when does it make sense to fine tune an LLM to update it's knowledge and how much data is needed in this case? I have not seen anyone show a real example of succeeded in this case and wonder if it's close to as difficult as training the LLM from scratch or if it's a feasible fine tuning use case.
- KRAKRISMOTT 3y agoWhy not use the finite state machine logits filter trick for controlling output format? It seems to work pretty well for JSON
- antman 3y agoWhat is that? Sounds useful
- evrimoztamur 3y agoI think he's referring to https://news.ycombinator.com/item?id=37125118 https://news.ycombinator.com/item?id=37125118 Controlling token outputs to match a regex generates valid JSON 100% of the time!
- jawerty 3y agoA lot of it is based on what you’re aiming to generalize. Typically for data that is fairly “conceptual” I find a DPR type solution to be better for most people with a similarity search. But if you want to generalize on style/data extraction/basic pattern matching fine tuning is incredible. I talk through some of these concepts on a fine tuning llama 2 live stream I did a couple months ago https://www.youtube.com/live/TYgtG2Th6fI?si=7VVv0Qooe2o1wuzD https://www.youtube.com/live/TYgtG2Th6fI?si=7VVv0Qooe2o1wuzD
- ozr 3y agoFwiw, unpublished testing on LLaMA-1 13B showed that it was able to learn a new word and it's meaning via PEFT with <50 examples. Finetuning can unquestionably add new data to a model. Jeremy Howard has written a bit about how quickly LLMs can pick up new concepts as well: https://www.fast.ai/posts/2023-09-04-learning-jumps/ https://www.fast.ai/posts/2023-09-04-learning-jumps/
- mikeagb 3y agoThe question of how to fine-tune to teach LLMs facts/knowledge is definitely something we're interested in exploring more in future work. The common opinion seems to me at least to be that fine-tuning is more to teach the model how to use the knowledge it already has to complete a specific task rather than to instill new knowledge, and that RAG should be sued to provide more specific context. However, I personally believe there is potential in fine-tuning for "memorization" or learning, and am excited to see new developments in the field.
- dnnssl2 3y agoKnowledge instillation is probably the holy grail of fine tuning. The hard part is: 1. Generalizing new facts. You can create a question answer pair of: “what is the population of the world in 2023?” “8 billion”, but it may not be able to pick up alternate phrasing or “does the world have 8 billion people on it?” 2. Catastrophic and behavioral forgetting. Continued fine tuning after RLHF and instruction fine tuning may result in the loss of the alignment and instruction following capabilities trained by OpenAI. At worst, it will start spewing random tokens like the example in the post. I have not yet seen it successfully done, and I suspect that updating fractions (~.1%) of the original weights with PEFT methods won’t help.
- redox99 3y ago> I have not yet seen it successfully done, and I suspect that updating fractions (~.1%) of the original weights with PEFT methods won’t help. Nitpick, but although when training LoRAs you're only training 1% or less (depending on rank) of the number of parameters of the entire model, the adapters affect the entire model and after merging the LoRA all of the weights of the model are updated.
- BoorishBears 3y agoYour answer is not really answering and is liable to confuse someone asking the question this person asked... the answer to their question is a simple: No. Current fine tuning techniques can only contribute to knowledge indirectly (getting better queries for an external data source for example), you cannot directly embed new facts in the model is any generally efficient/effective manner. There are toy examples of fine tuning in facts that are not of use outside of academic considerations at this point, and I sense it's contributing to the widespread confusion about fine-tuning's value proposition
- dnnssl2 3y agoThere are a few reputable academic examples of factual editing, such as: https://rome.baulab.info/ https://rome.baulab.info/ I don’t believe that the answer is strictly no. There are still many questions around the fine tuning method and the scale of data, as well as expectations of task accuracy from the perspective of an end user.