4 ms·

What would you recommend?

by hknapp 6y ago

Polylactic_acid 6y ago
Remote file systems are the worlds biggest pain in the ass. NFS seems like the easiest option but its ancient and insecure over the network. SFTP can be mounted and is also easy while being secure but is super CPU heavy on a single thread. All of the options I looked in to would fail when a program expects to be able to set permissions on things in the remote directory.
- duckMuppet 6y ago
  NFS using Kerberos over the network is very secure while being fast. Complaining about nfs being insecure is to be ignorant regarding how NFS was originally intended to be implemented.
- cat199 6y ago
  > NFS seems like the easiest option but its ancient and insecure over the network. r.e. old: nfsv4 is a different beast than nfs3; r.e. secure: these things don't hold true with krb5 auth.
  - pjc50 6y ago
    It's still not link layer encrypted, IIRC?
  - Polylactic_acid 6y ago
    Wikipedia says v4 came out in 2000 and a search for encryption on the page shows no results. sftp gives you really good key based authentication and encryption over the network. I wouldn't trust NFS for anything other than a highly secure internal network.
    - yrro 6y ago
      Mount with sec=krb5p and you get encryption (the p is short for Privacy).
    - starfallg 6y ago
      >a search for encryption on the page shows no results https://wiki.debian.org/NFS/Kerberos https://wiki.debian.org/NFS/Kerberos krb5p is pretty secure. You just need a Kerberos implementation. The alternative is to run NFS over Stunnel, which is what Amazon does for EFS.
qiqitori 6y ago
I would recommend redesigning the system to no longer require a filesystem shared over the network. (It appears to mostly work okay on a virtual network on the same host though.)
- kjs3 6y ago
  Thanks for this. Not only did I get a much needed laugh out loud, I'll be using it at a staff meeting tomorrow to get a laugh from the whole team. Sorry it's at your expense.
- st_goliath 6y ago
  > I would recommend redesigning the system to no longer require a filesystem shared over the network One of the first things that come to my mind when I hear "networked filesystem" is the typical setup at most companies I worked at in the past (those with more than a dozen or so people; plus of course schools and university I attended), where you have log on authentication via LDAP/AD/... and the system would mount network shares with your user files, files from the teams/projects you were on, etc. I would have a desktop PC in the office, but could also just walk over to the lab, login with the same credentials at a PC there and have all my files. Meeting rooms would also have permanently installed PCs and you wouldn't have to fidget with the projector connections/settings, you just log on and are good to go. On top of that, none of those systems would be my personal property and I wouldn't take them home with me, or bother installing/updating software. Other people would be paid full time to make sure everything works and I have access to all the files and software I need (and only those I need). Sounds crazy, right? I'm eager to hear your recommended redesign.
  - qiqitori 6y ago
    That's an okay use case. I was more thinking of what appears to be described in the article, i.e. running a large gitlab instance with the data directory in an NFS mount.
    - dnsmichi 6y ago
      Hi, Developer Evangelist at GitLab here. Scalability is a great point. With 13.0 we have added Gitaly clusters removing the dependency on NFS, whilst also improving high availability. Gitaly is the backend daemon for accessing Git repositories where GitLab communicates with. https://about.gitlab.com/releases/2020/05/22/gitlab-13-0-released/#gitaly-cluster-for-high-availability-git-storage https://about.gitlab.com/releases/2020/05/22/gitlab-13-0-rel... NFS support in Gitaly has been deprecated and will be removed in 14.0 next year. https://about.gitlab.com/releases/2020/05/22/gitlab-13-0-released/#nfs-for-git-repository-storage-deprecated https://about.gitlab.com/releases/2020/05/22/gitlab-13-0-rel... There are environments and use cases for NFS which are being discussed in this epic: https://gitlab.com/groups/gitlab-org/-/epics/1489 https://gitlab.com/groups/gitlab-org/-/epics/1489
lmm 6y ago
If you absolutely need to share one server's filesystem, Samba tends to have more reasonable failure modes. If you absolutely need a truly distributed filesystem then AFS is your only real option. But really a filesystem is almost certainly not the right interface; you're almost certainly better off using something higher-level - maybe HDFS if you need to store file-like data, maybe a key-value store or a distributed queuing system if you're doing something more structured.
- geofft 6y ago
  AFS (at least OpenAFS) is not "truly distributed": it has a single point of failure for read/write volumes. Yes, you can make read-only replicas easily and many large AFS users make good use of that, but it doesn't fundamentally solve the problem or fundamentally do something NFS can't. Also you can avoid a SPOF in NFS with Isilon's commercial offering (which worked great in my experience, at least back when they were using an implementation based on FreeBSD's kernel NFS server), or potential Red Hat's HA setup. Also, CephFS is an option too and avoids a SPOF by design. I've only run Ceph block storage but it's absolutely a real distributed system and works well.

What would you recommend?

by hknapp 6y ago

What would you recommend?

Remote file systems are the worlds biggest pain in the ass. NFS seems like the easiest option but its ancient and insecure over the network. SFTP can be mounted and is also easy while being secure but is super CPU heavy on a single thread. All of the options I looked in to would fail when a program expects to be able to set permissions on things in the remote directory.

duckMuppet 6y ago
NFS using Kerberos over the network is very secure while being fast. Complaining about nfs being insecure is to be ignorant regarding how NFS was originally intended to be implemented.
cat199 6y ago
> NFS seems like the easiest option but its ancient and insecure over the network. r.e. old: nfsv4 is a different beast than nfs3; r.e. secure: these things don't hold true with krb5 auth.
- pjc50 6y ago
  It's still not link layer encrypted, IIRC?
- Polylactic_acid 6y ago
  Wikipedia says v4 came out in 2000 and a search for encryption on the page shows no results. sftp gives you really good key based authentication and encryption over the network. I wouldn't trust NFS for anything other than a highly secure internal network.
  - yrro 6y ago
    Mount with sec=krb5p and you get encryption (the p is short for Privacy).
  - starfallg 6y ago
    >a search for encryption on the page shows no results https://wiki.debian.org/NFS/Kerberos https://wiki.debian.org/NFS/Kerberos krb5p is pretty secure. You just need a Kerberos implementation. The alternative is to run NFS over Stunnel, which is what Amazon does for EFS.

qiqitori 6y ago

I would recommend redesigning the system to no longer require a filesystem shared over the network. (It appears to mostly work okay on a virtual network on the same host though.)

kjs3 6y ago
Thanks for this. Not only did I get a much needed laugh out loud, I'll be using it at a staff meeting tomorrow to get a laugh from the whole team. Sorry it's at your expense.
st_goliath 6y ago
> I would recommend redesigning the system to no longer require a filesystem shared over the network One of the first things that come to my mind when I hear "networked filesystem" is the typical setup at most companies I worked at in the past (those with more than a dozen or so people; plus of course schools and university I attended), where you have log on authentication via LDAP/AD/... and the system would mount network shares with your user files, files from the teams/projects you were on, etc. I would have a desktop PC in the office, but could also just walk over to the lab, login with the same credentials at a PC there and have all my files. Meeting rooms would also have permanently installed PCs and you wouldn't have to fidget with the projector connections/settings, you just log on and are good to go. On top of that, none of those systems would be my personal property and I wouldn't take them home with me, or bother installing/updating software. Other people would be paid full time to make sure everything works and I have access to all the files and software I need (and only those I need). Sounds crazy, right? I'm eager to hear your recommended redesign.
- qiqitori 6y ago
  That's an okay use case. I was more thinking of what appears to be described in the article, i.e. running a large gitlab instance with the data directory in an NFS mount.
  - dnsmichi 6y ago
    Hi, Developer Evangelist at GitLab here. Scalability is a great point. With 13.0 we have added Gitaly clusters removing the dependency on NFS, whilst also improving high availability. Gitaly is the backend daemon for accessing Git repositories where GitLab communicates with. https://about.gitlab.com/releases/2020/05/22/gitlab-13-0-released/#gitaly-cluster-for-high-availability-git-storage https://about.gitlab.com/releases/2020/05/22/gitlab-13-0-rel... NFS support in Gitaly has been deprecated and will be removed in 14.0 next year. https://about.gitlab.com/releases/2020/05/22/gitlab-13-0-released/#nfs-for-git-repository-storage-deprecated https://about.gitlab.com/releases/2020/05/22/gitlab-13-0-rel... There are environments and use cases for NFS which are being discussed in this epic: https://gitlab.com/groups/gitlab-org/-/epics/1489 https://gitlab.com/groups/gitlab-org/-/epics/1489

lmm 6y ago

If you absolutely need to share one server's filesystem, Samba tends to have more reasonable failure modes. If you absolutely need a truly distributed filesystem then AFS is your only real option. But really a filesystem is almost certainly not the right interface; you're almost certainly better off using something higher-level - maybe HDFS if you need to store file-like data, maybe a key-value store or a distributed queuing system if you're doing something more structured.

geofft 6y ago
AFS (at least OpenAFS) is not "truly distributed": it has a single point of failure for read/write volumes. Yes, you can make read-only replicas easily and many large AFS users make good use of that, but it doesn't fundamentally solve the problem or fundamentally do something NFS can't. Also you can avoid a SPOF in NFS with Isilon's commercial offering (which worked great in my experience, at least back when they were using an implementation based on FreeBSD's kernel NFS server), or potential Red Hat's HA setup. Also, CephFS is an option too and avoids a SPOF by design. I've only run Ceph block storage but it's absolutely a real distributed system and works well.