5 ms·
Fortunately, costs for training superlarge models are coming down rapidly thanks to TPUs (which was the approach used to train GPT-J 6B) and DeepSpeed improveme
by minimaxir 5y ago
Fortunately, costs for training superlarge models are coming down rapidly thanks to TPUs (which was the approach used to train GPT-J 6B) and DeepSpeed improvements.
- Nextgrid 5y agoAre there any TPUs that can be purchased off-the-shelf and then owned, like you can do with a CPU or GPU? Or are you just limited to paying rent to cloud providers and ultimately being at their mercy when it comes to pricing, ToS, etc?
- 6gvONxR4sf7o 5y agoNo, but you probably aren't going to buy an A100 either, so it's a moot point.
- fragmede 5y agoAn A100 looks to be about $12k or so. A bit out of reach for individuals, but not so bad as a business expense, but maybe you can use it to mine Bitcoin when you're not using it to train models to help pay for itself or something.
- gjs278 5y agowhy not? people are spending 6k on cloud costs to train models in this thread. it pays for itself after two models.
- timschmidt 5y agohttps://aiyprojects.withgoogle.com/edge-tpu/ https://aiyprojects.withgoogle.com/edge-tpu/
- ericye16 5y agoI don't think these are good for training though, unfortunately.