3 ms·

Have any references on how you fine tuned?

• Tostino 3y ago
  Sure thing, I used axolotl, and my training parameters were: sequence_len: 6144 lora_r: 128 lora_alpha: 48 learning_rate: 0.00006 warmup_steps: 600 lr_scheduler: cosine gradient_accumulation_steps: 4 micro_batch_size: 1 num_epochs: 4 optimizer: paged_adamw_32bit flash_attention: true sample_packing: true
  • kaycebasques 3y ago
    Thanks for sharing all these details throughout the thread, Tostino. True open source spirit.