3 ms·

Unless I totally misread the "colossus" papers, Google does not in fact have an "accessible from anywhere" storage layer. GFS/Colossus is a per-cluster filesyst

Unless I totally misread the "colossus" papers, Google does not in fact have an "accessible from anywhere" storage layer. GFS/Colossus is a per-cluster filesystem capable of read and append only. It is very much NOT generic and only supports custom applications. Presumably that means that on every Google machine there is a directory that is shared for the entire cluster BUT: 1) It is per-cluster, not global (given Google's cluster sizes one imagine's that's not that much of a limitation, but then again, it seems like that is a clear limit) 2) you can only create or append to files (or delete them I guess) (and this is therefore non-posix, and does not support things like mysql or postgres) 3) this is very different from what GlusterFS, NFS, persistent volumes, etc provide. Therefore disks on google cloud are presumably very much not just files on this GFS/colossus thing. 4b) it was single master at least until 2004. Maybe until 2010. Apparently that can work. https://cloud.google.com/files/storage_architecture_and_challenges.pdf https://cloud.google.com/files/storage_architecture_and_chal... https://static.googleusercontent.com/media/research.google.com/en/us/archive/gfs-sosp2003.pdf https://static.googleusercontent.com/media/research.google.c...

philsnow 8y ago
> GFS/Colossus is a per-cluster filesystem capable of read and append only. It is very much NOT generic and only supports custom applications. For the most part, google only has custom applications. Just about everything is written in-house, and takes advantage of things like being able to open a file from the local disk file just as easily as opening one in Colossus or their equivalent of Zookeeper.
steelframe 8y ago
Would regional Persistent Disks be acceptable? https://cloud.google.com/compute/docs/disks/#repds https://cloud.google.com/compute/docs/disks/#repds Global might mean you're accessing blocks stored in Europe from a container running in Australia. If your workload is doing that, you might want to consider network costs and latency, and then reconsider whether you really want to architect things that way.
nemothekid 8y ago
You are entirely correct, Colossus isn't POSIX. However, the solution Google has, is since they control all the software, essentially everything they write acts as if Colossus is their storage layer. This "solves" the persistent storage layer in a way Kubernetes cannot. Inside Google you might deploy your "Spanner" database container which knows how to interface with Colossus and doesn't require any special setup or colocation (below the cluster level). You cannot deploy MySQL on K8s and expect the same.

philsnow 8y ago

> GFS/Colossus is a per-cluster filesystem capable of read and append only. It is very much NOT generic and only supports custom applications. For the most part, google only has custom applications. Just about everything is written in-house, and takes advantage of things like being able to open a file from the local disk file just as easily as opening one in Colossus or their equivalent of Zookeeper.

steelframe 8y ago

Would regional Persistent Disks be acceptable? https://cloud.google.com/compute/docs/disks/#repds https://cloud.google.com/compute/docs/disks/#repds Global might mean you're accessing blocks stored in Europe from a container running in Australia. If your workload is doing that, you might want to consider network costs and latency, and then reconsider whether you really want to architect things that way.

nemothekid 8y ago

You are entirely correct, Colossus isn't POSIX. However, the solution Google has, is since they control all the software, essentially everything they write acts as if Colossus is their storage layer. This "solves" the persistent storage layer in a way Kubernetes cannot. Inside Google you might deploy your "Spanner" database container which knows how to interface with Colossus and doesn't require any special setup or colocation (below the cluster level). You cannot deploy MySQL on K8s and expect the same.