2 ms·
What claud said. And QLora makes 70B training relatively affordable. But as a random aside, consider starting with an existing finetune instead of base llama
by brucethemoose2 3y ago
What claud said.
And QLora makes 70B training relatively affordable.
But as a random aside, consider starting with an existing finetune instead of base llama 70B, and match the formatting in your dataset.
- brianjking 3y agoIs there a particular fine tune you would suggest starting from? I really wish there was (maybe there is?) a 7 or 13B (or 70B) version with extended context (at least 16k) and function calling support ala OpenAI. Both exist on their own, I don't know of a combination.
- brucethemoose2 3y agoTBH I am out of the loop on finetunes, but you can search for "70B" on huggingface and sort by date.