3 ms·
OpenAI offers a fine-tuning api. How did come up the one minute estimate? According to a quick google search I did, the fastest SSDs these days have a bandwidt
by ma2rten 6y ago
OpenAI offers a fine-tuning api.
How did come up the one minute estimate? According to a quick google search I did, the fastest SSDs these days have a bandwidth of 3100 MB/s. So it would take 112s just to read the weights.
- liuliu 6y agoI don't have access to see whether they have fine-tuning API. Do you have any links explain the said fine-tuning? It is certainly surprising given there is no fine-tuning experiment mentioned in the GPT-3 paper. Weights loading is embarrassingly simple to parallelize. Just use madam with 3 or 4 NVMe SSD sticks are sufficiently enough. You are more likely bounded by PCIe bandwidth than the SSD bandwidth. Newer NVIDIA cards with PCIe-4 support helps.