3 ms·

Glad to see the NER-like task performed the best, as I was just about to test something like this for comparison with a fine-tuned BERT model. Any idea about th

• binarymax 3y ago
  Great question. I wish they said how long the 10 epochs took, so we could figure out the cost (or better, just posted the time and cost together): "For the 7B and 13B models, we used 16xA10Gs, and for the 70B model, we used 32xA10Gs (across 4x g5.48xlarge instances). When using Ray, there's no need to secure A100s to perform full-parameter fine-tuning on these models! The process is simply repeated for each task. Figures below show an example run based on a context length of 512, with a total of 3.7M effective tokens per epoch on GSM8k dataset. We ran the training for a maximum of 10 epochs and selected the best checkpoint according to the minimum perplexity score on the validation set."
  • kouroshh 3y ago
    Training times for GSM8k are mentioned here: https://github.com/ray-project/ray/tree/master/doc/source/templates/04_finetuning_llms_with_deepspeed#launching-fine-tuning https://github.com/ray-project/ray/tree/master/doc/source/te...
• kouroshh 3y ago
  Hey, I am one of the co-authors of the post. So the training data for ViGGO has about 5.1k rows which we trained with a block size of 512 (you can lower the block size if you want but we didn't do so because it was just easier to not change code :)). On 16xA10Gs for 7B it took ~15 min per epoch and on 13B it took ~25 min per epoch. So the on-demand cost per epoch is ~$7.2 for 7B and ~$12 for 13B. This is based on the time only spent on the training part and does not take into account the cluster startup time and shutdown time.
  • bugglebeetle 3y ago
    Great! Thank you!