Honestly, I dunno. I think most people are using lit-llama or EasyLM (on TPUs) for finetuning? QLORA is the gold standard for more affordable training. As for

Honestly, I dunno. I think most people are using lit-llama or EasyLM (on TPUs) for finetuning? QLORA is the gold standard for more affordable training. As for datasets, just look at the open datasets the best-in-class models are using, like Vicuna or https://huggingface.co/NousResearch/Nous-Hermes-13b https://huggingface.co/NousResearch/Nous-Hermes-13b Some model datasets like Manticore, Chronos or the infamous Pygmalion are more "secretive," but you can find the dataset gathering scripts on Github or in community chats.