5 ms·
> Is there a reason why the pretrained models are not included as well? Because every one else, too, lacks the time and hardware required to replicate the trai
by sailingparrot 4y ago
> Is there a reason why the pretrained models are not included as well?
Because every one else, too, lacks the time and hardware required to replicate the training steps.
There is just not that many organizations that can pay for 100-200 thousands hours of training on A100 GPUs.
So the reason why the pretained model is not included, is that it does not exist (yet).
- newswasboring 4y ago> There is just not that many organizations that can pay for 100-200 thousands hours of training on A100 GPUs. How much is this in money?
- sailingparrot 4y agoWith no discount, one A100 is about 3$/hour, so we are talking about 600K$ for a full training. Probably closer to 1M$ since in practice you never train successfully to the end on the first shot.
- isoprophlex 4y agoIIRC a K80 goes for prices between 0.3 and 3 USD per GPU-hour Pretty expensive all in all
- malka 4y agobetween a a few hundred K and a few millions IMO
- lagrange77 4y agoThere should be a system in place like Folding@Home or SETI@Home for situations like this.
- HanClinto 4y agoAgreed. This feels like the next big step in open source machine learning. I would love to be able to dedicate CPU / GPU towards chunking out open-source GPT-3 / DALL-E 2 clones so long as the trained weights are shared with the world afterwards. That feels more long-term productive to me than crunching coins.
- gauravvij137 4y agoIn a way, it is similar to our approach at Q Blocks. Decentralized GPU computing for ML.
- lacker 4y agoFor what it's worth, the architecture for seti@home doesn't make much sense any more in the modern era for most types of data processing. It's simply too slow and error-prone to transfer large amounts of data between user machines. Modern radio telescope data analysis happens at datacenters with a lot of GPUs in them, just like modern AI model training.
- lagrange77 4y agoOkay, gotcha. Just out of interest, why is it error prone? Because of some kind of merge conflicts of the results?
- lacker 4y agoNo, I just mean you design your system expecting the data from machine 1462 to get analyzed at some point, but that machine belongs to some random person who uninstalled your software without you knowing about it, so you never get back an analysis. Or you have bugs that come from 100 different versions of your software running because you can't simply upgrade the software on all the machines of the datacenter.
- lagrange77 4y agoI understand. > you can't simply upgrade the software on all the machines You can update a website. And a website can use the local gpu. The break the training task in smaller pieces, parallelize and distribute it part is indeed beyond my skills.