4 ms·

Everyone here’s saying “ Colossus is really cool”. Can someone share why it’s so cool compared to other file systems?

pclmulqdq 4y ago
It is absurdly scalable in contrast to other filesystems, and extremely reliable. Also, you get per-file control over replication level (file-level durability/availability), plus comprehensive permissions management.
- aschleck 4y ago
  The permissions management (user/group but with cloud roles) is what I miss most. NFS style approaches assume that I will give every user an unprivileged user on their local machine and deny root access so they can't just create a user with a different ID. S3 style ACLs per-object are just very confusing and no one is going to set those. S3 access policies per role are obviously not a solution because that requires centralizing lists of what the permissions are. I just want to be able to assign a chmod and a chown on a file for a cloud user and group... And don't forget about how fantastic the group tooling was. I also miss being able to write files with /ttl=72h/ anywhere in the path and knowing they will be cleaned up automatically without me doing any more work. In contrast S3 just rots. Need to find an intern to write an SQS pipeline to schedule file deletions :)
  - mohon 4y ago
    Is the permission also use Zanzibar? or is it using a different thing?
    - pclmulqdq 4y ago
      If you work at G, you should be able to see how D does permissions by looking at its ACL checking code (start with one of the files in the D server code that defines an op, and you can get to the permissions code pretty quickly). There's a lot of code in Colossus/D that is special because it has to be able to come up in a nearly empty cluster.
  - rajamaka 4y ago
    I mean creating a lifecycle rule on a ln s3 bucket is pretty easy
    - KaiserPro 4y ago
      yes, but its per bucket rather than namespace/key. plus IAMs is a pain in the arse to setup
    - xyzzy_plugh 4y ago
      The scope is entirely different. You can't grant privileges to create lifecycle rules on a subtree of a bucket. You set the rules on the entire bucket. It's practically useless by comparison.
  - rfoo 4y ago
    > NFS style approaches assume that I will give every user an unprivileged user on their local machine With Kerberos it kinda works, each client is bound to a UID instead of blindly trusting the client, but very few people nowadays deploy this ancient tech (and it's fragile).
jeffbee 4y ago
For me, largely ancillary reasons. One is its extreme degree of observability. At any point in time you can instantly answer complex, detailed questions like how many disk heads had to move to complete one given gmail request. Most of Google’s observability stack is attributable to the need to observe Colossus and D.
SuperQue 4y ago
Hundreds of petabytes and billions of files in a single filesystem are now "easy mode". That was 10+ years ago. (Former GFS/Colossus/Borg SRE)

It is absurdly scalable in contrast to other filesystems, and extremely reliable. Also, you get per-file control over replication level (file-level durability/availability), plus comprehensive permissions management.

aschleck 4y ago
The permissions management (user/group but with cloud roles) is what I miss most. NFS style approaches assume that I will give every user an unprivileged user on their local machine and deny root access so they can't just create a user with a different ID. S3 style ACLs per-object are just very confusing and no one is going to set those. S3 access policies per role are obviously not a solution because that requires centralizing lists of what the permissions are. I just want to be able to assign a chmod and a chown on a file for a cloud user and group... And don't forget about how fantastic the group tooling was. I also miss being able to write files with /ttl=72h/ anywhere in the path and knowing they will be cleaned up automatically without me doing any more work. In contrast S3 just rots. Need to find an intern to write an SQS pipeline to schedule file deletions :)
- mohon 4y ago
  Is the permission also use Zanzibar? or is it using a different thing?
  - pclmulqdq 4y ago
    If you work at G, you should be able to see how D does permissions by looking at its ACL checking code (start with one of the files in the D server code that defines an op, and you can get to the permissions code pretty quickly). There's a lot of code in Colossus/D that is special because it has to be able to come up in a nearly empty cluster.
- rajamaka 4y ago
  I mean creating a lifecycle rule on a ln s3 bucket is pretty easy
  - KaiserPro 4y ago
    yes, but its per bucket rather than namespace/key. plus IAMs is a pain in the arse to setup
  - xyzzy_plugh 4y ago
    The scope is entirely different. You can't grant privileges to create lifecycle rules on a subtree of a bucket. You set the rules on the entire bucket. It's practically useless by comparison.
- rfoo 4y ago
  > NFS style approaches assume that I will give every user an unprivileged user on their local machine With Kerberos it kinda works, each client is bound to a UID instead of blindly trusting the client, but very few people nowadays deploy this ancient tech (and it's fragile).

jeffbee 4y ago

For me, largely ancillary reasons. One is its extreme degree of observability. At any point in time you can instantly answer complex, detailed questions like how many disk heads had to move to complete one given gmail request. Most of Google’s observability stack is attributable to the need to observe Colossus and D.

SuperQue 4y ago

Hundreds of petabytes and billions of files in a single filesystem are now "easy mode". That was 10+ years ago. (Former GFS/Colossus/Borg SRE)