The speed of training is really interesting to me too. Llambda labs rent out 8 A100s for ~$9-12/hour. Total training cost there would be ~$850-1200. Another ~$2

The speed of training is really interesting to me too. Llambda labs rent out 8 A100s for ~$9-12/hour. Total training cost there would be ~$850-1200. Another ~$2000 for the GPT training data, some more given there would be input tokens too. Very approachable prices. > Now this wouldn't be possible without the high quality synthetic dataset produced by GPT(1B tokens) Just a note that this is GPT3.5 (I assume turbo?).