4 ms·
Diversify to what, though? The economics of hosting models and datasets seem to just not be great in general. For the same reasons that we can expect HF to even
by 0xcde4c3db 3y ago
Diversify to what, though? The economics of hosting models and datasets seem to just not be great in general. For the same reasons that we can expect HF to eventually crack down on the "freeloaders", I think pretty much any other entity also will unless it's explicitly set up as a public benefit. I could see this sort of thing maybe being funded by universities or their affiliated foundations, but most companies with the resources to provide this kind of hosting also have a pretty strong incentive to extract revenue from it.
- michaelt 3y agoPip, maven, npm, docker hub, apt-get, github and suchlike seem to manage to provide hefty binary downloads with no need to monetise anonymous downloads. I don't quite understand how they do so, but it seems to be possible. Everyone who downloads a 5GB stable diffusion model probably downloaded 5GB of pytorch+cuda+cudnn first.
- Lord_Zero 3y agoBack in the day, universities would host all that shit via their mirrors for free for everyone. Not sure where it all is today with the cloud being so prevalent.
- zo1 3y agoYou don't even hear the term mirror anymore.
- basilgohar 3y agoLinux distros still have large networks of mirrors for downloading their releases and also packaged binary updates. Maybe we don't "hear" the term but they are still there.
- outside1234 3y agoWe call them CDNs now :)
- brucethemoose2 3y ago> 5GB of pytorch+cuda+cudnn This itself is kinda insane. In fact, it was literally untenable for PyPi, which is why (historically) installing all that was unecessarily painful.
- 0xcde4c3db 3y agoFor its part, PyPI is apparently funded by the Python Software Foundation and run by volunteer admins, with the bulk of its data served gratis by Fastly. This is based on a blog post about its infrastructure from 2021 [1], which also says that the list price for the throughput they're doing is $1.8 million per month. [1] https://dustingram.com/articles/2021/04/14/powering-the-python-package-index-in-2021/ https://dustingram.com/articles/2021/04/14/powering-the-pyth...
- bpiche 3y agoSpacy hosts them too, of course. python -m spacy download en-core-web-lg etc
- mgalgs 3y agoWe need a P2P model hosting solution. Napster for ML models. Could just be a thin layer on top of bittorrent.