3 ms·

I have been hankering for this kind of thing but for remote filesystem sharing. My specific use case is having separate NAS and compute machines in my homelab,

I have been hankering for this kind of thing but for remote filesystem sharing. My specific use case is having separate NAS and compute machines in my homelab, and I have workloads running in nomad/k8s/docker-swarm/whatever on the compute machines, and I want the workloads to keep their state on the NAS. NFS is this mire of needing users and groups to be the same everywhere (possibly also needing to configure some translation layer(s)). MacOS starts numbering users at 501, most linux distros I'm aware of start at uid 1000, and I don't know, can you even mount NFS on windows? Most of the guides I've seen completely punt on any kind of homelab authentication, just advising to squash all users to nobody and have everything on the share be 0777 or whatever completely open facl that equates to. With NFS, you don't authenticate as a storage consumer directly to the remote storage, you authenticate as a user to the OS and it deals with the storage on your behalf, but this seems like a layering/concerns violation. From what I can tell SMB is a bit better because your client authenticates directly to the SMB server, but I have never figured out all the Windows-specific identity, groups, ACLs, etc. All of these are kind of heavyweight for certain use cases too -- it would be handy to have userspace remote filesystems available on microcontrollers, but I don't necessarily want to run a tiny linux on them. But.. if clients could sign remote storage connection requests with private keys and then that lets them mount whatever storage is on the other end with FUSE or something similar for microcontrollers.. then access to remote storage is completely agnostic of operating system.

mattpallissard 3y ago
NFS has -o sec=krb5p which uses Kerberos principles, which can be mapped to accounts server side. Additionally, the the traffic is encrypted. OpenAFS also uses kerberos for auth, but has a separate 'aklog' for AFS tokens on top of that. It also solved permissions in an interesting way by completely foregoing OS permissions. This allowed them to have more fine grained than POSIX specified and made the permissions identical across platforms.
- philsnow 3y ago
  Interesting; in a past life my school actually used AFS and had competent admins for it, and the experience was good (although I stayed very much “on the beaten path”). It seems like a bit more of a lift to go with Kerberos than even using something like LDAP in my home network for user provisioning. My whole screed above comes from reluctance to set up freeIPA.
  - mattpallissard 3y ago
    That sounds about right, its roots are education and research. Hence the strong federation support. If you do decide to go down the krb route (I highly recommend), skip free IPA and just run vanilla openldap and MIT Kerberos. It's one of those things that has a learning curve, but mostly just runs itself after that.
ericbarrett 3y ago
Do the workloads need to share their state simultaneously? If not then you probably want SAN (exported block devices) instead of NAS (exported file systems).
- philsnow 3y ago
  Most of mine don’t, but I do have a couple that require it. For those I use SMB and squash all users to a “smb_users” users that I created in the NAS for this purpose While I was writing the comment, I had in the back of my mind, “do I actually want iSCSI?” but it’s a whole can of worms I haven’t yet opened beyond reading the docs for csi-provisioner (which has direct support for provisioning per-workload zvols on my NAS)

mattpallissard 3y ago

NFS has -o sec=krb5p which uses Kerberos principles, which can be mapped to accounts server side. Additionally, the the traffic is encrypted. OpenAFS also uses kerberos for auth, but has a separate 'aklog' for AFS tokens on top of that. It also solved permissions in an interesting way by completely foregoing OS permissions. This allowed them to have more fine grained than POSIX specified and made the permissions identical across platforms.

philsnow 3y ago
Interesting; in a past life my school actually used AFS and had competent admins for it, and the experience was good (although I stayed very much “on the beaten path”). It seems like a bit more of a lift to go with Kerberos than even using something like LDAP in my home network for user provisioning. My whole screed above comes from reluctance to set up freeIPA.
- mattpallissard 3y ago
  That sounds about right, its roots are education and research. Hence the strong federation support. If you do decide to go down the krb route (I highly recommend), skip free IPA and just run vanilla openldap and MIT Kerberos. It's one of those things that has a learning curve, but mostly just runs itself after that.

ericbarrett 3y ago

Do the workloads need to share their state simultaneously? If not then you probably want SAN (exported block devices) instead of NAS (exported file systems).

philsnow 3y ago
Most of mine don’t, but I do have a couple that require it. For those I use SMB and squash all users to a “smb_users” users that I created in the NAS for this purpose While I was writing the comment, I had in the back of my mind, “do I actually want iSCSI?” but it’s a whole can of worms I haven’t yet opened beyond reading the docs for csi-provisioner (which has direct support for provisioning per-workload zvols on my NAS)