8 ms·
Understanding Google’s File System (2020)
- nosefrog 4y agoColossus, the successor to GFS, is one of the coolest technologies at Google. Almost everything we have internally is built on top of it.
- wuschel 4y agoCould you point me towards some materials re Colossus?
- atombender 4y agohttps://cloud.google.com/blog/products/storage-data-transfer/a-peek-behind-colossus-googles-file-system https://cloud.google.com/blog/products/storage-data-transfer...
- deleted 4y ago[deleted]
- hppb 4y agoThis talk by Denis Serenyi is good. http://www.pdsw.org/pdsw-discs17/slides/PDSW-DISCS-Google-Keynote.pdf http://www.pdsw.org/pdsw-discs17/slides/PDSW-DISCS-Google-Ke...
- JamesSwift 4y agoWhat specifically makes it so cool in your opinion?
- dekhn 4y agoafter a couple decades of breaking every cluster filesystem I've used at scale, Colossus was the first one that managed to handle my loads effortlessly. It required far less attention from SREs when abusive users sent it a lot of load. It also made using reed solomon codes easy.
- the-rc 4y agoThe truly problematic Colossus clusters usually fell into one of two buckets: a) those with shrinking machine counts (but always increasing traffic) because the hardware was ancient and repairs/upgrades were not viable and b) those where most of the quota was held by a single project, with massive jobs. One cluster at IDI was mostly Image Search's and it wasn't fun at all.
- pclmulqdq 4y agoI worked on Colossus while I was at G. It is an incredible piece of technology, and the current versions are a lot better than GFS. It makes me a little sad that no papers get written about it. Scale-out ZFS and Ceph from the outside world may be the closest competitors, but they don't hold a candle to Colossus in terms of scalability and reliability.
- ClumsyPilot 4y agodoes it use custom hardware or soes it work om commodity PCs? If I recall correctly, spanner used GPS.or atomic clocks to have precise time on every machine, and that enables it to perform much better.
- pclmulqdq 4y agoIt uses commodity hardware. Also, Spanner doesn't really need the atomic clocks, they just help performance. CockroachDB, which is similar, runs fine on commodity hardware.
- deepGem 4y agoI seem to have misunderstood Spanner by and large. I thought their achilles heel was the atomic clock since it is this clock which makes consensus at scale possible and there by transactions. Cockroach and Yugabyte both work on similar principles but they use these hybrid logical clocks. The hybrid logical clock is a function of NTP and a monotonic local clock so I guess this works ok for Raft consensus. https://www.yugabyte.com/wp-content/uploads/2021/05/Distributed-Transactions-Without-Atomic-Clocks.pdf https://www.yugabyte.com/wp-content/uploads/2021/05/Distribu... From Jepsen - "yugabyte seems mostly resilient to clock skews' - this is a very safe but at the same time a vague statement. What does mostly mean ? Good for 99% of practical use cases ?
- pclmulqdq 4y agoI think it means "we can't prove that it works, but we have tested it and we haven't found a case where it doesn't." There are conceivable cases where the hybrid logical clock won't work, but most of these are cases where other systems will fail first (eg TLS/SSL). Edit: To address the first part of your comment, the atomic clocks narrow the bounds on how bad your NTP clock can be. Spanner essentially waits out the bounds on clock variance on each write transaction to get consistency. There still would be a measurable bound on your clock precision without the atomic clock, but it would be a lot larger.
- victor106 4y agoEveryone here’s saying “ Colossus is really cool”. Can someone share why it’s so cool compared to other file systems?
- pclmulqdq 4y agoIt is absurdly scalable in contrast to other filesystems, and extremely reliable. Also, you get per-file control over replication level (file-level durability/availability), plus comprehensive permissions management.
- aschleck 4y agoThe permissions management (user/group but with cloud roles) is what I miss most. NFS style approaches assume that I will give every user an unprivileged user on their local machine and deny root access so they can't just create a user with a different ID. S3 style ACLs per-object are just very confusing and no one is going to set those. S3 access policies per role are obviously not a solution because that requires centralizing lists of what the permissions are. I just want to be able to assign a chmod and a chown on a file for a cloud user and group... And don't forget about how fantastic the group tooling was. I also miss being able to write files with /ttl=72h/ anywhere in the path and knowing they will be cleaned up automatically without me doing any more work. In contrast S3 just rots. Need to find an intern to write an SQS pipeline to schedule file deletions :)
- mohon 4y agoIs the permission also use Zanzibar? or is it using a different thing?
- pclmulqdq 4y agoIf you work at G, you should be able to see how D does permissions by looking at its ACL checking code (start with one of the files in the D server code that defines an op, and you can get to the permissions code pretty quickly). There's a lot of code in Colossus/D that is special because it has to be able to come up in a nearly empty cluster.
- arcade79 4y agoI worked as an SRE on colossus during it's infancy at Google. One of the coolest pieces of tech I've ever had the pleasure of working on.
- kirugan 4y agoOriginal whitepaper is a great overview of how the system works, but devil in the details. I still have a lot of questions about implementation, for instance: from where did master get information about chunkservers (chubby, configuration files) ? did chunkserver know about which master it belongs to? etc.
- lokar 4y agoGFS predated chubby. The master is configured with the endpoints for all the chunkaervers. A separate control/config system supported this including things like automatic replacement of dead nodes.
- kirugan 4y agohow do you know?
- lokar 4y agoI was there
- kirugan 4y agoAwesome! Just checked the whitepaper there was a cluster with 300 machines, so all machines had to be described in configuration file that was used by master, right?
- lokar 4y agoMostly. Binaries of that era took everything on the command line. The config system would generate that from files and keep the running binaries up to date via ssh.
- codethief 4y agoTo all Ex-Googlers here who used to work on Colossus and say what an incredible piece of tech it is: Why do you think there's no open-source equivalent?
- amelius 4y agoDoes it require atomic clocks? Perhaps that's a reason? Also, there don't seem to be public research papers about it.
- codethief 4y ago> Also, there don't seem to be public research papers about it. The question of atomic clocks aside (whose answer I don't know), Ex-Googlers have created open-source alternatives to Google-internal software in a lot of other cases, too, so I don't think the lack of public research papers could be the reason.
- SuperQue 4y agoAtomic clocks are not required. That's a misunderstanding of what they're used for. The atomic clock thing helps improve performance of Spanner, by tightening the SLO of node-to-node and continent-to-continent clock drift.
- riedel 4y agoHDFS was closely modelled after GFS . So the question maybe would be if progress was made here: we have OrangeFS, SwiftFS, CephFS, CassandraFS . Then for HPC there is lustreFS. Most focus is though on object storage though. We use CEPH and Lustre and I am quite happy. I wonder if there really is a concrete demand FOSS cannot serve. Also most of us are not hyperscalers...
- dub 4y agoI think what people really want is: - SaaS solution, not just open source (improvements to s3, for example) - A global hierarchy of files rather than an assortment of different buckets like s3 - Per-subdirectory permissions and chargeback so anyone can create project-specific subdirectories and have their organization billed for storage costs - Filesystem drivers allowing every production and development computer and coding environment to access remote files in the same way and with the same ease as remote files. The difference between accessing remote files and local files in Google code is literally just the file prefix. In all other ways, remote files are _better_ than local files (they have all the same features, plus more) so there's very little reason to directly use local disk most of the time.
- bborud 4y agoThe exciting thing about GFS wasn't even the technology. It was the discovery you made as a new engineer starting at Google (in the 2000s) that from your workstation (and from any job you ran on Borg) you had access to this enormous amount of data stored on GFS across different data centers. This is the thing that people keep missing. The technology was only the enabler. What made the difference was to have thousands of developers with wide access to the biggest data lake in the world. Almost nobody gets that when trying to make data lakes, or whatever they are called now, in the corporate world. They focus on vendors and technologies and whatnot. Then ignore that what makes data valuable is access to it by people who don't even know yet what they might do with it.
- Razengan 4y ago> data lakes, or whatever they are called now Clouds? Databergs? Smaug-hoards? No wait that's for anime collectors
- fh973 4y agoPlus all data is stored in the same format (protobufs in recordio or sstables), so was very easy to access.
- lazide 4y agoWith accessible schemas (due to single repo) and fast (performance wise). Of course now folks just complain that their job is shoveling protos haha.
- ethbr0 4y agoAbsolutely! IMHO, new engineers start with "What technology should we use to do this?", while experienced engineers start with "Assume we've already done this successfully with some (i.e. any) technology, what will the impact be?" And yes, if you have a hierarchical eng org then there's different job responsibilities at different levels. But at some point, new engineers end up making their own calls, and they usually don't pay enough attention to the forest. Some technically complex things just aren't worth doing, and some simple things are incredibly valuable to do. E.g. Porting the fundamentals of an MMU + append optimizations into a distributed storage system: not theoretically groundbreaking. But the impact for its users was!
- smoochy 4y agoCan someone articulate the differences from zfs? Apart from the fact that, Google's file system seems to be working better with large files and allow simultaneous writing (or, rather, appending) without blinking an eye. But that'd be useful for, maybe very large tech companies.
- KaiserPro 4y agoZFS is grand for one machine. GFS/colossus is designed to offer similar level of features (well in terms of performance and namespace, not snapshots) but over many machines. ZFS doesn't have to worry too much about latency, or another machine writing to the block that it wants to update. As soon as you break free from one machine, you have to think about how you deal with: where data lives ( is the data stored seperate from the metadata?) how a client signals who has a write lock, permissions (ie user authentication, nfs basically just uses UserIds, which can be faked, unless you are running kerberos/NFSv4) data affinity (pull data from another data center is expensive, do we sync, cache or clone? how do we resolve write clashes?) All of those questions have an impact on speed, scalability, reliability and durability. You need to choose your designs carefully to get a filesystem that does what you want it to
- dekhn 4y agoOne of the most important differences is that Colossus and GFS are user-space filesystems for the client. You're just making RPCs to a service. There is no kernel filesystem. Upgrading the client and the server is just software upgrade, no kernel changes. And the operations were intentionally restricted- for example, instead of the debacle that is locking in NFS, locking is part of a different (and even more fundamental) tech at Google (chubby, which is NOT a filesystem, or so they say).
- cbm-vic-20 4y agoFor you oldsters out there- how does GFS/Collossus compare to Andrew FS? https://en.wikipedia.org/wiki/Andrew_File_System https://en.wikipedia.org/wiki/Andrew_File_System
- lizknope 4y agoI was just about to ask if anyone had used AFS. When I went to college from 1993-97 our university used AFS for all the Unix workstations. We had a mix of Solaris, SunOS, HP-UX, DEC Ultrix, SGI IRIX, and then some Linux boxes. When you logged in at any workstation from any vendor in any computer lab your home dir would be mounted and everything was encrypted. We had ACLs where we could give permission by username and create our own ACL groups and give permission to a project directory to just our project/lab partners. It was so much easier than the chmod uid/gid stuff. In theory we could access other universities under /afs/mit or whatever site they were but I don't think we ever had that active.
- ricktdotorg 4y agoI spent a couple of years at Morgan Stanley and when we cut over from SunOS to Solaris they included a rollout of AFS worldwide. It was pretty successful. It was pretty mind blowing to be able to login to a Solaris workstation while visiting the NYC office and have your [local-to-London] $HOME and CDE environment just kinda sync automatically (albeit slowly back then). The administration of AFS is ... interesting, but considering this was the mid-to-late 90s, it really was "ahead of its time". FWIW, Morgan Stanley back then had an incredible UNIX Engineering department.
- lizknope 4y agoI finally got this book a couple of years ago. "MIT Project Athena: A Model for Distributed Campus Computing" https://www.amazon.com/gp/product/0135853249/ https://www.amazon.com/gp/product/0135853249/ I really thought AFS, DCE, and SSI (single system image) were going to take over. It was kind of depressing when I got a job and everything was NIS/NFS and we manually logged into a machine and started a job. Then we got clustering software like LFS to distribute jobs but I thought the dynamic process migration using checkpoint / restart kind of stuff would take over but it seems to be more trouble than it is worth. At work we would always run stuff off a powerful 4 to 16 CPU Sun and display to our cheaper SPARCstation 5 or Linux desktop via X11. I will say it is great now to have a virtual desktop session Exceed/NoMachine/X2go and be able to go home and connect to it and have all my windows in the same place just like when I left. I've got X2go set up at home so even when I travel I can get a desktop at home if I need it. https://en.wikipedia.org/wiki/Project_Athena https://en.wikipedia.org/wiki/Project_Athena https://en.wikipedia.org/wiki/Distributed_Computing_Environment https://en.wikipedia.org/wiki/Distributed_Computing_Environm... https://en.wikipedia.org/wiki/Single_system_image https://en.wikipedia.org/wiki/Single_system_image
- dekhn 4y agoGFS had some serious issues- one of the biggest being the master node RAM- since the metadata was stored in RAM, there was a limit on filesystem. Sometimes you had to bribe a hwops person with whiskey to upgrade the ram over christmas
- jll29 4y agoThe GFS paper 1. showed that software can overcome faults in the hardware and 2. formed the storage basis for Google MapReduce. From there, Apache Hadoop, the open source clone by the Nutch team took off, which in turn led to Zacharia's Spark (now Apache Spark), which provided the RDD abstraction to remove the batch-only limitations of Hadoop, thus enabling distributed iterative and graph methods at scale.
- chrislusf 4y agoCan someone please critique on the difference between SeaweedFS and Colossus? https://github.com/seaweedfs/seaweedfs https://github.com/seaweedfs/seaweedfs Need some advice on how to make SeaweedFS more scalable.