2 ms·

> There's a reason why every major tech company has developed its own distributed filesystem I haven't worked at FAANG, but is this a well-known fact? I've nev

> There's a reason why every major tech company has developed its own distributed filesystem I haven't worked at FAANG, but is this a well-known fact? I've never heard of it. Unless they're referring to things like S3? Are these large corps running literal custom filesystem implementations?

foota 1y ago
Not sure about everyone, but probably yes. Imo the killer feature is durability and replication. You can use single disks if you don't care about data loss, but once you need to start replicating data you need a distributed filesystem. Tectonic is Facebooks, Google's is Colossus. I'm not sure about the others.
- StrangeDoctor 1y ago
  Deepseek has their own and they’re relatively small https://github.com/deepseek-ai/3FS https://github.com/deepseek-ai/3FS It’s specialized knowledge, hard to do “correctly” (read posix here) but obtainable and implementable by a small team if you pick your battles right. Also supporting very specific use cases helps a lot. It’s also pretty easy to justify as the hardware and software from vanguard tech companies is outrageously expensive. I used to develop software for a blue colored distributed filesystem.
allset_ 1y ago
I can only comment about the one I work for, but yes. It's also discussed publicly to some degree. https://cloud.google.com/blog/products/storage-data-transfer/how-colossus-optimizes-data-placement-for-performance https://cloud.google.com/blog/products/storage-data-transfer...
loeg 1y ago
Yes, at least Facebook and Google have distributed file services. (Facebook's was historically based on HDFS, which was based on an old Google paper.)