3 ms·
I'm biased because I did some work on Colossus for a few years, but it's my opinion that Google's main advantage in its ability to build complex distributed sys
by pclmulqdq 3y ago
I'm biased because I did some work on Colossus for a few years, but it's my opinion that Google's main advantage in its ability to build complex distributed systems is Colossus (and GFS). The Colossus layer is the point where a lot of systems actually take care of the CAP theorem, and because Colossus does it so well they almost don't have to worry about it at all.
Ceph, Hadoop, scale-out ZFS, and others that came afterward got the idea of distributing a filesystem but sort of missed the point that GFS/Colossus's great advantage was that it's so damn good at being a distributed system, and has been tried and tested through a lot of different kinds of failure modes.
- tdba 3y agoLink to any resources you consider valuable for learning about Colossus?
- pclmulqdq 3y agoSadly, the best public material is the technical marketing for GCP, see the following for example (a few other materials can be found in this comments section): https://cloud.google.com/blog/products/storage-data-transfer/a-peek-behind-colossus-googles-file-system https://cloud.google.com/blog/products/storage-data-transfer... If you want to learn about modern exascale filesystems, the best paper today is probably from Facebook: https://www.usenix.org/conference/fast21/presentation/pan https://www.usenix.org/conference/fast21/presentation/pan