4 ms·
The key thing to understand about GFS and relatives is how essential it is for distributed computation. It is natural for us to look at a big file and think, "
by btilly 3y ago
The key thing to understand about GFS and relatives is how essential it is for distributed computation.
It is natural for us to look at a big file and think, "I need to do something complicated, so I'll do a distributed calculation, then get a new file." This makes the file you start with, and the file you wind up with, both bottlenecks to the calculation.
Instead you really want to have data distributed across machines. Now do a distributed map-reduce calculations, and get another distributed dataset. Then create a pipeline of such calculations. Leading to a distributed calculation with no bottlenecks. Anywhere.
(Of course this is a low-level view of how it works. We may want to build that pipeline automatically off of, say, a SQL statement. To get a distributed query against a datastore.)
Absolutely essential for this is the ability to have a distributed, replicated, filesystem. This is what GFS was. When you try to scale a system, you focus on your current bottlenecks, and accept that there will be later ones. GFS did not distribute metadata. It did not play well with later systems created for managing distributed machines. But it was an important step towards a distributed world. And the fact that Google was so far ahead of everyone else on this was a giant piece of their secret sauce.
True story. I went to Google at about the same time that eBay shut down their first datacenter. So I was reading an article from eBay about how more work it took to shut down a datacenter without disrupting operations. And how proud they are at succeeding. And then I saw the best practices that Google was using so that users wouldn't notice if a random datacenter went offline without warning. Which was a capacity that Google regularly tested. Because if you're afraid to test your emergency systems, they probably don't work.
What eBay was proud of doing with a lot of work, was entirely taken for granted and automated at Google.
- jeffbee 3y ago> you really want to have data distributed across machines It's funny how poorly Hadoop, HDFS, and HBase learned this lesson. HBase tries to keep a replica of its data on the local machine, under the incorrect belief that a local RegionServer (approx: chunk server) will be somehow better than a building full of them. This is why HBase performance is hideous. But it is all supposedly inspired by or modeled after GFS and Bigtable?
- woooooo 3y agoIf I recall correctly, hbase performance problems had a lot to do with jdk 1.6 garbage collection, rather than the data location strategy.
- jimbokun 3y agoHadoop and family fell squarely in the category of something where I always felt I was missing something important. Because it was being discussed everywhere, but I could never match it up with a problem I needed to solve, even though we were dealing with other “big data” systems.
- woooooo 3y agoOne interesting idea that got lost in the last 10 years was "push compute to the data", saving a network transfer. It seems like these days most systems prefer the flexibility of separate data/compute and are willing to pay the network tax for it.
- jeffbee 3y agoIt wasn't an interesting idea, it was a footgun. There's nothing naturally advantageous about the machine your process happens to be running on. In the era of very large machines there may be substantial contention for the block device that by coincidence is in the same chassis as your CPU.
- mbreese 3y agoIt was useful at the time though. Both network and disk were substantially slower at the time. And servers had far fewer cores. So which server stored the data could make your processing faster. And you’d have less risk for IO contention within the same chassis. Fast forward a few years and compute servers can have hundreds of cores and the network is significantly faster. I think it was a “good enough” design for the time and those assumptions. And batch processing. Now? I think there are different assumptions to be made. (And I think there is a good argument that by the time Hadoop/HDFS had stabilized the assumptions had already shifted)
- woooooo 3y agoIt's a footgun if your cluster is providing data/compute to other systems, absolutely, the case you mentioned is a risk worth extra network traffic to avoid. But theoretically, if we're trying to beat a benchmark or something, the "move compute to data" idea has something to it. Nobody transfers the LLM weights off of card memory, let alone to other machines for compute.
- woadwarrior01 3y agoIt's still around in things like Spanner. Although, compute and storage are mostly disaggregated into different classes of servers now.
- pclmulqdq 3y agoI'm biased because I did some work on Colossus for a few years, but it's my opinion that Google's main advantage in its ability to build complex distributed systems is Colossus (and GFS). The Colossus layer is the point where a lot of systems actually take care of the CAP theorem, and because Colossus does it so well they almost don't have to worry about it at all. Ceph, Hadoop, scale-out ZFS, and others that came afterward got the idea of distributing a filesystem but sort of missed the point that GFS/Colossus's great advantage was that it's so damn good at being a distributed system, and has been tried and tested through a lot of different kinds of failure modes.
- tdba 3y agoLink to any resources you consider valuable for learning about Colossus?
- pclmulqdq 3y agoSadly, the best public material is the technical marketing for GCP, see the following for example (a few other materials can be found in this comments section): https://cloud.google.com/blog/products/storage-data-transfer/a-peek-behind-colossus-googles-file-system https://cloud.google.com/blog/products/storage-data-transfer... If you want to learn about modern exascale filesystems, the best paper today is probably from Facebook: https://www.usenix.org/conference/fast21/presentation/pan https://www.usenix.org/conference/fast21/presentation/pan