3 ms·
(2020) isn't quite enough to clue the reader in that GFS was extirpated at Google more than a decade ago. Historical trivia only.
by jeffbee 3y ago
(2020) isn't quite enough to clue the reader in that GFS was extirpated at Google more than a decade ago. Historical trivia only.
- tivert 3y ago> (2020) isn't quite enough to clue the reader in that GFS was extirpated at Google more than a decade ago. Historical trivia only. Do you have more info about that? I was just Googling and found this: https://pdos.csail.mit.edu/6.824/papers/gfs-faq.txt https://pdos.csail.mit.edu/6.824/papers/gfs-faq.txt > Q: Does Google still use GFS? > A: Rumor has it that GFS has been replaced by something called Colossus, with the same overall goals, but improvements in master performance and fault-tolerance. In addition, many applications within Google have switched to more database-like storage systems such as BigTable and Spanner. However, much of the GFS design lives on in HDFS, the storage system for the Hadoop open-source MapReduce. > https://cloud.google.com/blog/products/storage-data-transfer/a-peek-behind-colossus-googles-file-system https://cloud.google.com/blog/products/storage-data-transfer...
- jeffbee 3y agoThere is almost nothing public about Colossus. Here's some marketing fluff, but it's clear that Colossus is the successor to GFS and they compare scale (CFS >> GFS). https://cloud.google.com/blog/products/storage-data-transfer/a-peek-behind-colossus-googles-file-system https://cloud.google.com/blog/products/storage-data-transfer... Also this slide deck https://www.pdsw.org/pdsw-discs17/slides/PDSW-DISCS-Google-Keynote.pdf https://www.pdsw.org/pdsw-discs17/slides/PDSW-DISCS-Google-K...
- nosefrog 3y ago> In addition, many applications within Google have switched to more database-like storage systems such as BigTable and Spanner. And where do BigTable and Spanner store their data? ;)
- summerlight 3y agoInterestingly, Colossus was initially designed to be the next generation BigTable storage. It turned out that other teams had a similar problem, so it has become a more general file system.
- ithkuil 3y agoTurns out that big table is also useful to implement the storage layer for big table itself. Luckily turtles can be stacked on top of each others
- dist-epoch 3y agoBigTable stores it's data in Colossus, but Colossus stores it's metadata in BigTable :)
- kristjansson 3y agoAnd Colossus stores its metametametadata in Chubby :) https://www.pdsw.org/pdsw-discs17/slides/PDSW-DISCS-Google-Keynote.pdf https://www.pdsw.org/pdsw-discs17/slides/PDSW-DISCS-Google-K...
- kccqzy 3y agoAt Google the project to migrate GFS to Colossus was called Moonshot. It was endorsed by Eric Schmidt and became a top-down mandate. This goes into details of the migration: https://sre.google/static/pdf/CaseStudiesInfrastructureChangeManagement.pdf https://sre.google/static/pdf/CaseStudiesInfrastructureChang... (scroll to Chapter 1)
- rkagerer 3y agoThanks. This stat from the section on Diskless is interesting: On shared machines, spinning disk reliability decreased compute reliability; 25%–30% of production task deaths were attributable to disk failure.
- joatmon-snoo 3y agoIf you do the math for GFS, there are some scaling limits that make it quite evident how long ago Google had to leave GFS :)
- pclmulqdq 3y agoColossus retains a lot of the ideas of GFS, so I wouldn't say that this paper is completely outdated. I would recommend the Tectonic paper from Facebook for a more modern distributed filesystem example: https://www.usenix.org/conference/fast21/presentation/pan https://www.usenix.org/conference/fast21/presentation/pan