3 ms·
Care to elaborate on the finetune? It's surprisingly very hard to come across a useful finetuning examples.
by eurekin 3y ago
Care to elaborate on the finetune? It's surprisingly very hard to come across a useful finetuning examples.
- mmcwilliams 3y agoSure, very generally we're doing PEFT starting with insights from examples very much like this one [0] and have gradually built our own tooling and customized the approach a lot as the underlying Huggingface libraries have progressed even in the last 6 months. I will say that one of the most important parts of the process that I've found is in the prompt structuring, the use of special tokens based on how the base models were trained and customizing the tokenizer where necessary. That work in particular is not covered adequately by the examples I was able to find when I started, in my opinion. [0] https://medium.com/@kshitiz.sahay26/fine-tuning-llama-2-for-news-category-prediction-a-step-by-step-comprehensive-guide-to-fine-tuning-48c06dee28a9 https://medium.com/@kshitiz.sahay26/fine-tuning-llama-2-for-...