4 ms·

Hugging Face's internal storage system is more sophisticated now after they acquired XetHub. They have a clever way to chunk and content dedupe the data that ma

Hugging Face's internal storage system is more sophisticated now after they acquired XetHub. They have a clever way to chunk and content dedupe the data that makes it a lot more efficient to version control large files.

Whitespace 1mo ago
Xet's tech is very interesting and I greatly enjoyed their blog posts and other technical writeups[0] But in the era of frontier models and hardware designed for fast model loading (eg: LLM on a chip) I can't imagine that this is worth so much. There's *a lot* of competition to improve Git LFS. So while I agree with you that this plays a role I also have to believe that it's to understand model usage by competition and thus stifle it. [0] - https://huggingface.co/docs/hub/en/xet/overview https://huggingface.co/docs/hub/en/xet/overview
- aniceperson 1mo ago
  git lfs just dumps your bin on s3, and is abandonoware since ms bought github, since github historically hosts on aws s3 and improving it would immediately benefit amazon. gh, the maintainers of it, is the only forge that does not allow clean-up unreachable artifact from the server.
deleted 1mo ago
[deleted]
varispeed 1mo ago
Surely you could now vibe code such system for under $1,000.
- HeWhoLurksLate 1mo ago
  I highly doubt it, there's about a million things that only show up at scale that aren't lessons the bots can easily digest from Medium posts and bypass
  - varispeed 1mo ago
    You are assuming the operator of vibe coding tools is not experienced in the domain.
- doublerabbit 1mo ago
  There's more to than just the website. You have network bandwidth costs, security costs, infrastructure costs. You can't just spin up a VPS and throw a few brain jar's in; the network bandwidth must be a fortune. Hosting over 700GB+ for many of the same models has storage cost too.
  - varispeed 1mo ago
    That is fair point, but still you can build such system for under a grand and then pimp yourself out to VCs.

Whitespace 1mo ago

Xet's tech is very interesting and I greatly enjoyed their blog posts and other technical writeups[0] But in the era of frontier models and hardware designed for fast model loading (eg: LLM on a chip) I can't imagine that this is worth so much. There's *a lot* of competition to improve Git LFS. So while I agree with you that this plays a role I also have to believe that it's to understand model usage by competition and thus stifle it. [0] - https://huggingface.co/docs/hub/en/xet/overview https://huggingface.co/docs/hub/en/xet/overview

aniceperson 1mo ago
git lfs just dumps your bin on s3, and is abandonoware since ms bought github, since github historically hosts on aws s3 and improving it would immediately benefit amazon. gh, the maintainers of it, is the only forge that does not allow clean-up unreachable artifact from the server.

deleted 1mo ago

[deleted]

varispeed 1mo ago

Surely you could now vibe code such system for under $1,000.

HeWhoLurksLate 1mo ago
I highly doubt it, there's about a million things that only show up at scale that aren't lessons the bots can easily digest from Medium posts and bypass
- varispeed 1mo ago
  You are assuming the operator of vibe coding tools is not experienced in the domain.
doublerabbit 1mo ago
There's more to than just the website. You have network bandwidth costs, security costs, infrastructure costs. You can't just spin up a VPS and throw a few brain jar's in; the network bandwidth must be a fortune. Hosting over 700GB+ for many of the same models has storage cost too.
- varispeed 1mo ago
  That is fair point, but still you can build such system for under a grand and then pimp yourself out to VCs.