4 ms·
> ~14 min. for 7B for 1 epoch on 3.5M tokens. ~26 min for 13B for 1 epoch. > At least 1xg5.16xlarge for head-node and 15xg5.4xlarge for worker nodes for both 7
by rising-sky 3y ago
> ~14 min. for 7B for 1 epoch on 3.5M tokens. ~26 min for 13B for 1 epoch.
> At least 1xg5.16xlarge for head-node and 15xg5.4xlarge for worker nodes for both 7B and 13B
For the uninitiated, anyone have an idea how much this would cost on AWS?
- grandpayeti 3y agog5.16xlarge - $4.0960/hour g5.4xlarge - $1.6240/hour You're looking at about $30/hour to run this in us-east-1. https://instances.vantage.sh/?selected=g5.16xlarge,g5.4xlarge https://instances.vantage.sh/?selected=g5.16xlarge,g5.4xlarg...
- rising-sky 3y agothanks