4 ms·
11B is a chonky model! Curious to hear what kind of hardware you used to train it. GPT-J (a 6B model) was trained on a v3-128, I think, so this is about 2x that
by sillysaurusx 4y ago
11B is a chonky model! Curious to hear what kind of hardware you used to train it. GPT-J (a 6B model) was trained on a v3-128, I think, so this is about 2x that.
- rushingcreek 4y agoThis latest model was trained on a distributed multi-node cluster of 8 servers each with 8 NVIDIA A100 GPUs. Need a lot of GPU memory when training with long sequence length.
- alchemist1e9 4y agoFor how long do you know?
- EastSmith 4y agoLooks like a single NVIDIA A100 costs $23,667.29 [1]? So 8 servers X 8 cards X $23,667.29 means the whole setup costs $1,514,706.56? https://www.dell.com/en-us/shop/nvidia-ampere-a100-pcie-300w-80gb-passive-double-wide-full-height-gpu-customer-install/apd/490-bhbk/graphic-video-cards https://www.dell.com/en-us/shop/nvidia-ampere-a100-pcie-300w...
- rushingcreek 4y agoAt list prices, perhaps. Importantly, we don't own the hardware. H100s are coming soon and they are much faster.
- Jayakumark 4y agoWas it trained on Common Crawl Dataset or custom scraping ?
- alchemist1e9 4y agoAs a follow up to that question, which I’m also very curious about. Do we have rough estimates of the training GPU compute needed to train GPT-3 and has OpenAI said anything about compute resources used to train ChatGPT?