3 ms·
Very interesting project, thanks. I see that DeepSpeed is on your TODO list. I wonder what the biggest LLaMA model is that can be fine-tuned on 2x RTX 3090 & 12
by Tepix 4y ago
Very interesting project, thanks.
I see that DeepSpeed is on your TODO list.
I wonder what the biggest LLaMA model is that can be fine-tuned on 2x RTX 3090 & 128GB RAM.
- lxe 4y agoYou can probably finetune a 13b one with that. Try these scripts: https://github.com/zphang/minimal-llama/#minimal-llama https://github.com/zphang/minimal-llama/#minimal-llama
- meghan_rain 4y agowhat if i want to finetune with long documents? say AI papers that are ~10 pages long on average? how would they be tokenized given that max_seq_length is 512?
- amrb 4y agoSplit your training data into chucks of text that make sense. A random dataset example https://huggingface.co/datasets/imdb https://huggingface.co/datasets/imdb
- meghan_rain 4y agoThanks, what does making sense mean? Be logically coherent (eg a paragraph of text in a document?) And does the training then create windows of ngrams on those chunks? Or what is the input/output? The reason I ask: If I had question/answer pairs, the question is the input, the answer is the output. What is the "output" when the input is just a (logically coherent) chunk of text?
- lxe 4y ago> What is the "output" when the input is just a (logically coherent) chunk of text? It probably won't change much if it's just a single sample. If you put in a large corpus of samples that repeat on the same theme, then the model will be "tuned" to repeat that theme. If you increase the number of epochs, you can overtrain it, meaning that it will just spit out the training data text.