3 ms·
> (2020) isn't quite enough to clue the reader in that GFS was extirpated at Google more than a decade ago. Historical trivia only. Do you have more info about
by tivert 3y ago
> (2020) isn't quite enough to clue the reader in that GFS was extirpated at Google more than a decade ago. Historical trivia only.
Do you have more info about that? I was just Googling and found this:
https://pdos.csail.mit.edu/6.824/papers/gfs-faq.txt https://pdos.csail.mit.edu/6.824/papers/gfs-faq.txt
> Q: Does Google still use GFS?
> A: Rumor has it that GFS has been replaced by something called
Colossus, with the same overall goals, but improvements in master
performance and fault-tolerance. In addition, many applications within
Google have switched to more database-like storage systems such as
BigTable and Spanner. However, much of the GFS design lives on in
HDFS, the storage system for the Hadoop open-source MapReduce.
> https://cloud.google.com/blog/products/storage-data-transfer/a-peek-behind-colossus-googles-file-system https://cloud.google.com/blog/products/storage-data-transfer...
- jeffbee 3y agoThere is almost nothing public about Colossus. Here's some marketing fluff, but it's clear that Colossus is the successor to GFS and they compare scale (CFS >> GFS). https://cloud.google.com/blog/products/storage-data-transfer/a-peek-behind-colossus-googles-file-system https://cloud.google.com/blog/products/storage-data-transfer... Also this slide deck https://www.pdsw.org/pdsw-discs17/slides/PDSW-DISCS-Google-Keynote.pdf https://www.pdsw.org/pdsw-discs17/slides/PDSW-DISCS-Google-K...
- nosefrog 3y ago> In addition, many applications within Google have switched to more database-like storage systems such as BigTable and Spanner. And where do BigTable and Spanner store their data? ;)
- summerlight 3y agoInterestingly, Colossus was initially designed to be the next generation BigTable storage. It turned out that other teams had a similar problem, so it has become a more general file system.
- ithkuil 3y agoTurns out that big table is also useful to implement the storage layer for big table itself. Luckily turtles can be stacked on top of each others
- dist-epoch 3y agoBigTable stores it's data in Colossus, but Colossus stores it's metadata in BigTable :)
- kristjansson 3y agoAnd Colossus stores its metametametadata in Chubby :) https://www.pdsw.org/pdsw-discs17/slides/PDSW-DISCS-Google-Keynote.pdf https://www.pdsw.org/pdsw-discs17/slides/PDSW-DISCS-Google-K...
- kccqzy 3y agoAt Google the project to migrate GFS to Colossus was called Moonshot. It was endorsed by Eric Schmidt and became a top-down mandate. This goes into details of the migration: https://sre.google/static/pdf/CaseStudiesInfrastructureChangeManagement.pdf https://sre.google/static/pdf/CaseStudiesInfrastructureChang... (scroll to Chapter 1)
- rkagerer 3y agoThanks. This stat from the section on Diskless is interesting: On shared machines, spinning disk reliability decreased compute reliability; 25%–30% of production task deaths were attributable to disk failure.
- joatmon-snoo 3y agoIf you do the math for GFS, there are some scaling limits that make it quite evident how long ago Google had to leave GFS :)