4 ms·
I think that deploying onto a shared volume is exactly the workload that EFS encourages. That things will stop functioning can be said about any (AWS) technolo
by Corrado 10y ago
I think that deploying onto a shared volume is exactly the workload that EFS encourages. That things will stop functioning can be said about any (AWS) technology. "Lord help you when (S3|RDS|SQS|etc.) goes down and every (thing) you have is hung up on a broken (thing) connection."
The rational argument here is that if uptime is important to you, the solution is to utilize multiple regions, of which EFS is now in 4. You are using AWS because of the SLA and the assumption that when something goes wrong there are legions of technical folks trying to fix it as soon as possible.
- jedberg 10y agoThe difference is that all of those services are accessed via an API, which means your client can do clever things when it fails, like timeout, give a fallback, find an equivalent resource in another zone or region, etc. With EFS, it's exposed as NFS, which hooks in much deeper. If it goes down, there isn't anything you can do to work around the problem, unless you start hacking your own file system kernel modules.
- zeisss 10y agoCan't you measure and timeout on the file-descriptor writes too?
- pyre 10y agoDo you want to start wrapping your system calls to read files in code that forks to periodically check for a hung system call?
- jabl 10y agoIt's worse that that, actually. With hard mounts (the default), the only way to interrupt is with SIGKILL, which if my memory serves me correctly, is a process-wide signal. So you'd have to do all NFS I/O in a separate child process. With soft mounts (does EFS support this?) you can exchange SIGKILL for religiously checking return values for all I/O syscalls, retrying partial writes and whatnot.
- takeda 10y agoCan't you just use soft mounts? Especially since in the scenario you described you can use read only mounts.
- acdha 10y agoIt depends on the nature of the failure. If you're doing something which triggers hard failures – i.e. where a read() call on a ocket would unblock with an error – soft mounts will eventually recover. The problem is that, in my experience anyways, the vast majority of failures aren't that clean – things like a server which processes packets but never responds, a network connection which drops packets but doesn't change the link status, etc. — and in those cases soft mounts behave no better than hard mounts. There also used to be kernel bugs in *BSD, Darwin, and Linux where the client could deadlock in heavy activity, which were hard to reproduce and get fixed. In all of those cases, anything which tries to access something on the NFS mount will block in the kernel (i.e. “kill -9“ won't work) and the mount cannot be unmounted normally. I wrote https://github.com/acdha/mountstatus https://github.com/acdha/mountstatus awhile back – if memory serves, 2004 or so – because we found that on Linux a lazy unmount would still work in this case and so you could have a process monitor the mount status (fork() a child to check the mount, alert if it doesn't get a response within a set interval) and a watchdog could respond to an alert by issuing a “umount -l” and remounting, which doesn't fix the blocked process but is less disruptive than rebooting and that new processes won't block because they tried to access that mount.
- walden42 10y agoAssuming all assets get stored on S3, the other common writing scenario is logging. How would multiple machines writing to the same log file work?