4 ms·
Colossus, the successor to GFS, is one of the coolest technologies at Google. Almost everything we have internally is built on top of it.
by nosefrog 4y ago
Colossus, the successor to GFS, is one of the coolest technologies at Google. Almost everything we have internally is built on top of it.
- wuschel 4y agoCould you point me towards some materials re Colossus?
- atombender 4y agohttps://cloud.google.com/blog/products/storage-data-transfer/a-peek-behind-colossus-googles-file-system https://cloud.google.com/blog/products/storage-data-transfer...
- deleted 4y ago[deleted]
- hppb 4y agoThis talk by Denis Serenyi is good. http://www.pdsw.org/pdsw-discs17/slides/PDSW-DISCS-Google-Keynote.pdf http://www.pdsw.org/pdsw-discs17/slides/PDSW-DISCS-Google-Ke...
- JamesSwift 4y agoWhat specifically makes it so cool in your opinion?
- dekhn 4y agoafter a couple decades of breaking every cluster filesystem I've used at scale, Colossus was the first one that managed to handle my loads effortlessly. It required far less attention from SREs when abusive users sent it a lot of load. It also made using reed solomon codes easy.
- the-rc 4y agoThe truly problematic Colossus clusters usually fell into one of two buckets: a) those with shrinking machine counts (but always increasing traffic) because the hardware was ancient and repairs/upgrades were not viable and b) those where most of the quota was held by a single project, with massive jobs. One cluster at IDI was mostly Image Search's and it wasn't fun at all.