5 ms·
Understanding Google's File System (2020)
- jeffbee 3y ago(2020) isn't quite enough to clue the reader in that GFS was extirpated at Google more than a decade ago. Historical trivia only.
- tivert 3y ago> (2020) isn't quite enough to clue the reader in that GFS was extirpated at Google more than a decade ago. Historical trivia only. Do you have more info about that? I was just Googling and found this: https://pdos.csail.mit.edu/6.824/papers/gfs-faq.txt https://pdos.csail.mit.edu/6.824/papers/gfs-faq.txt > Q: Does Google still use GFS? > A: Rumor has it that GFS has been replaced by something called Colossus, with the same overall goals, but improvements in master performance and fault-tolerance. In addition, many applications within Google have switched to more database-like storage systems such as BigTable and Spanner. However, much of the GFS design lives on in HDFS, the storage system for the Hadoop open-source MapReduce. > https://cloud.google.com/blog/products/storage-data-transfer/a-peek-behind-colossus-googles-file-system https://cloud.google.com/blog/products/storage-data-transfer...
- jeffbee 3y agoThere is almost nothing public about Colossus. Here's some marketing fluff, but it's clear that Colossus is the successor to GFS and they compare scale (CFS >> GFS). https://cloud.google.com/blog/products/storage-data-transfer/a-peek-behind-colossus-googles-file-system https://cloud.google.com/blog/products/storage-data-transfer... Also this slide deck https://www.pdsw.org/pdsw-discs17/slides/PDSW-DISCS-Google-Keynote.pdf https://www.pdsw.org/pdsw-discs17/slides/PDSW-DISCS-Google-K...
- nosefrog 3y ago> In addition, many applications within Google have switched to more database-like storage systems such as BigTable and Spanner. And where do BigTable and Spanner store their data? ;)
- summerlight 3y agoInterestingly, Colossus was initially designed to be the next generation BigTable storage. It turned out that other teams had a similar problem, so it has become a more general file system.
- ithkuil 3y agoTurns out that big table is also useful to implement the storage layer for big table itself. Luckily turtles can be stacked on top of each others
- dist-epoch 3y agoBigTable stores it's data in Colossus, but Colossus stores it's metadata in BigTable :)
- kristjansson 3y agoAnd Colossus stores its metametametadata in Chubby :) https://www.pdsw.org/pdsw-discs17/slides/PDSW-DISCS-Google-Keynote.pdf https://www.pdsw.org/pdsw-discs17/slides/PDSW-DISCS-Google-K...
- kccqzy 3y agoAt Google the project to migrate GFS to Colossus was called Moonshot. It was endorsed by Eric Schmidt and became a top-down mandate. This goes into details of the migration: https://sre.google/static/pdf/CaseStudiesInfrastructureChangeManagement.pdf https://sre.google/static/pdf/CaseStudiesInfrastructureChang... (scroll to Chapter 1)
- rkagerer 3y agoThanks. This stat from the section on Diskless is interesting: On shared machines, spinning disk reliability decreased compute reliability; 25%–30% of production task deaths were attributable to disk failure.
- joatmon-snoo 3y agoIf you do the math for GFS, there are some scaling limits that make it quite evident how long ago Google had to leave GFS :)
- pclmulqdq 3y agoColossus retains a lot of the ideas of GFS, so I wouldn't say that this paper is completely outdated. I would recommend the Tectonic paper from Facebook for a more modern distributed filesystem example: https://www.usenix.org/conference/fast21/presentation/pan https://www.usenix.org/conference/fast21/presentation/pan
- dekhn 3y agoGFS was terrible. It may have been great in the early days of google when they couldn't build an index, but I recall having to bribe hwops people (with single-malt) to upgrade RAM on the GFS masters, and a few times, my team (Ads) threatened to run our own filesystems that didn't fail so badly. Colossus, on the other hand, is really good.
- deleted 3y ago[deleted]
- the-rc 3y agoAmong other things, GFS ran outside of Borg, so you couldn't just donate quota to let it run with more resources, as you found out. Conversely, given that Colossus ran on Borg, part of the migration involved an elaborate system that minted CPU/RAM quota for the storage system users whenever the CFS cell was grown. Then there were all the issues on the low end machines (hello, Nocona) that had not enough resources to run the storage stack AND user jobs. Stuff wouldn't schedule (and the cluster suddenly appeared to be missing some disk), but at least that was a lot more explicit than the performance issues you witnessed on GFS.
- robertlagrant 3y ago> part of the migration involved an elaborate system that minted CPU/RAM quota for the storage system users whenever the CFS cell was grown I've no idea what this is, but it sounds fascinating. Was this a way to auto-allocate more resource based on rules?
- marklar423 3y agoWhat I think the GP means is previously, GFS ran on their own machines and didn't need to participate in the Google-wide quota system for CPU/RAM. However Colossus _does_ run on the general Google compute infrastructure, and so needed a way to get CPU/RAM quota where none existed (because it was all used by existing, non-storage users).
- the-rc 3y ago
- alt227 3y agoIf anything, the interview that this was based on is from 2009 and was a much more interesting read for myself: https://queue.acm.org/detail.cfm?id=1594206 https://queue.acm.org/detail.cfm?id=1594206
- btilly 3y agoThe key thing to understand about GFS and relatives is how essential it is for distributed computation. It is natural for us to look at a big file and think, "I need to do something complicated, so I'll do a distributed calculation, then get a new file." This makes the file you start with, and the file you wind up with, both bottlenecks to the calculation. Instead you really want to have data distributed across machines. Now do a distributed map-reduce calculations, and get another distributed dataset. Then create a pipeline of such calculations. Leading to a distributed calculation with no bottlenecks. Anywhere. (Of course this is a low-level view of how it works. We may want to build that pipeline automatically off of, say, a SQL statement. To get a distributed query against a datastore.) Absolutely essential for this is the ability to have a distributed, replicated, filesystem. This is what GFS was. When you try to scale a system, you focus on your current bottlenecks, and accept that there will be later ones. GFS did not distribute metadata. It did not play well with later systems created for managing distributed machines. But it was an important step towards a distributed world. And the fact that Google was so far ahead of everyone else on this was a giant piece of their secret sauce. True story. I went to Google at about the same time that eBay shut down their first datacenter. So I was reading an article from eBay about how more work it took to shut down a datacenter without disrupting operations. And how proud they are at succeeding. And then I saw the best practices that Google was using so that users wouldn't notice if a random datacenter went offline without warning. Which was a capacity that Google regularly tested. Because if you're afraid to test your emergency systems, they probably don't work. What eBay was proud of doing with a lot of work, was entirely taken for granted and automated at Google.
- jeffbee 3y ago> you really want to have data distributed across machines It's funny how poorly Hadoop, HDFS, and HBase learned this lesson. HBase tries to keep a replica of its data on the local machine, under the incorrect belief that a local RegionServer (approx: chunk server) will be somehow better than a building full of them. This is why HBase performance is hideous. But it is all supposedly inspired by or modeled after GFS and Bigtable?
- woooooo 3y ago
- AtomicOrbital 3y agoIt's understandable hyper scalers roll their own infrastructure tools yet naturally small shops reach for 3rd party tools ... it's telling when speaking with each how justified each feels ... yet at the boundary zone between those extremes folks it gets interesting ... mid sized shops need to be aware of the benefits of rolling their own and at a minimum not frown on job candidates coming from a big shop who instinctively lean towards build versus just use off the shelf libs
- y04nn 3y agoDoes anybody know if there is any usage/implementation of a distributed hash table (with consistent hashing) for reliable file storage in a datacenter instead of internet p2p ? Would that make sense and what would be the drawbacks ?