3 ms·
Out of the box no, but you can solve the scaling git problem by improving the deduplication: https://xethub.com/assets/docs/faq/how-xet-deduplication-works http
by skadamat 3y ago
Out of the box no, but you can solve the scaling git problem by improving the deduplication: https://xethub.com/assets/docs/faq/how-xet-deduplication-works https://xethub.com/assets/docs/faq/how-xet-deduplication-wor...
We're focused on this problem at XetHub.
- Llama2 git repo with ~600 GB of model files: https://xethub.com/XetHub/Llama2 https://xethub.com/XetHub/Llama2
- RedPajama git repo with 3.4 terabytes of data: https://xethub.com/XetHub/RedPajama-Data-1T https://xethub.com/XetHub/RedPajama-Data-1T