I work at XetHub and we're taking a different approach here to managing ML experiments in git. Instead of trying to store data and ML models in one place (like

I work at XetHub and we're taking a different approach here to managing ML experiments in git. Instead of trying to store data and ML models in one place (like S3) and code, models & documentation in another place (like GitHub), we are scaling Git so you can version everything in a single system. You can just use git and you don't need to learn a new tool or set of commands. This way, you can start with a simple experiment tracking approach of folders inside the same branch and then evolve gradually to multiple branches with long running experiments. We're about to release our Github integration, so data & ML teams can take advantage of this inside their existing Github repos. If anyone wants a tour or wants to chat, my email's in my HN profile. If you're curious about our tech: - Here's an example 3.3 TB Git repo: https://xethub.com/XetHub/RedPajama-Data-1T https://xethub.com/XetHub/RedPajama-Data-1T - We wrote a paper on our solution to scale git to 100 terabytes: https://about.xethub.com/blog/git-is-for-data-published-in-cidr-2023 https://about.xethub.com/blog/git-is-for-data-published-in-c... - We created a Rust library to mount large repos to machines with limited storage space: https://news.ycombinator.com/item?id=37573679 https://news.ycombinator.com/item?id=37573679