6 ms·

> But: What possible semantics could you want for mkdir() such that a timeout is sane? nfs is mentioned a few lines later. Creating directories on any sort of

by TheDong 10y ago

> But: What possible semantics could you want for mkdir() such that a timeout is sane? nfs is mentioned a few lines later. Creating directories on any sort of network mount can and will time out, and it's better if your application can handle that sort of error rather than just wait forever. Yes, the error can't conclusively say "the directory wasn't actually created", but at least it can say "it might not have been, dunno" FUSE filesystems also can have bugs resulting in mkdir hanging forever. Neither of those are all that insane.

loeg 10y ago
> nfs is mentioned a few lines later. Creating directories on any sort of network mount can and will time out NFS basically assumes a reliable network. Operations should complete more or less immediately. How is a syscall-level timeout more useful than a whole-application watchdog for handling flaky mounted filesystems? If you want a flaky network filesystem, I think it's probably better to go with some application-level protocol rather than putting it below Unix's filesystem interface. > FUSE filesystems also can have bugs resulting in mkdir hanging forever. Filesystem bugs are filesystem bugs -- the application layer shouldn't have to work around them.
- xenadu02 10y ago
  I'm not picking on you specifically but when people say things like "Unix gives you brain damage" this is what they mean. Your suggestion is that the network should be reliable (a strict impossibility) or that every application should use a bespoke protocol to do simple things like working with a network file system. The Unix IO model isn't well designed and is far from the only Unix-ism to work poorly in the face of threads (signals and fork/exec are common offenders; not even memcpy is safe in a signal handler!) This is hardly surprising: it was slapped together ad-hoc over the years.
  - loeg 10y ago
    > I'm not picking on you specifically but when people say things like "Unix gives you brain damage" this is what they mean. You understand that's still offensive and implies I have brain damage, right? > every application should use a bespoke protocol to do simple things like working with a network file system. It doesn't have to be bespoke, but I agree that the Unix VFS interface isn't appropriate for it. What operating system are you aware of that provides a mkdir interface with timeout? Windows' CreateDirectory() does not. OS X is just BSD. We're departing the realm of mainstream systems. QNX mkdir does not appear to take a timeout. Plan9 uses create() with a DIR flag, which does not take a timeout. Help me out here?
    - fiddlerwoaroof 10y ago
      I suspect old systems like Multics did. But, this is the sort of thing that Richard Gabriel is talking about in worse is better[1] [1]: https://www.dreamsongs.com/WorseIsBetter.html https://www.dreamsongs.com/WorseIsBetter.html
    - deleted 10y ago
      [deleted]
    - xenadu02 10y ago
      > You understand that's still offensive and implies I have brain damage, right? My apologize, I didn't intend that. I mean that some people are used to Unix and use their familiarity as a post-hoc rationalization for some of the poorly designed aspects of it. To be clear: my position is that several Unix APIs are bad designs. Rather than making excuses for them we should demand better cross-platform APIs. > It doesn't have to be bespoke, but I agree that the Unix VFS interface isn't appropriate for it. I don't agree. Network filesystems should be mountable and traversable just like local filesystems, but the OS and libraries should do more to be async and multi-thread aware, as should applications.
      - pjmlp 10y ago
        > Rather than making excuses for them we should demand better cross-platform APIs. We already have them in the form of language runtimes. POSIX as cross-platform API is mostly relevant to C. Even C++ is moving beyond POSIX, by providing proper OS abstractions on the standard library.
  - fulafel 10y ago
    NFS violates a number of basic Unix filesystem guarantees. It's well documented that it violates POSIX guaranteed filesystem semantics. The abstraction NFS chose is "filesystem operations don't time out", and implements this by just making your process hang as long as the network is dead - possibly forever. I'm not sure it's a bad thing that the Unix FS API doesn't do networked filesystems. Everbody is using userspace APIs for remote file access, and the big and varied design space of remote file access semantics doesn't get bottlenecked on 30 year old system call APIs.
  - slrz 10y ago
    > The Unix IO model isn't well designed and is far from the only Unix-ism to work poorly in the face of threads Right, threads in the form of pthreads should've never been introduced into Unix, as they made a huge mess of many interfaces. The rfork "you have processes that can share resources" model is a much better fit (although it doesn't solve all problems). It's what Linux as well as FreeBSD implement at the kernel level anyway except they add tons of ugly stuff on top of it to allow for performant implementations of POSIX threads semantics.
- TheDong 10y ago
  > How is a syscall-level timeout more useful than a whole-application watchdog for handling flaky mounted filesystems? It's more useful because once you enter into a syscall, it's out of the application's control. The kernel's off in its happy place doing whatever it does, and the application has no control over that other than being able to specify timeouts to begin with or having async epoll-like interfaces.
  - ec109685 10y ago
    s/application/thread/
  - AstralStorm 10y ago
    You can also interrupt syscalls using signals or pthread_cancel.
- spc476 10y ago
  As if. At $WORK, we service requests (millions per day) that require access to NFS mounted files [1]. A network hiccup could effect one request but not another. I'm not sure how a "whole-application watchdog" would work for us. As it is, the main thread will schedule a "wakeup" from another thread whose sole existence is to interrupt file system calls stuck in NFS. Yes, it would be nice if such code didn't have to exist. But it does. On the plus side, it's there for a feature that isn't used that often. On the bad side, it's there for a feature that isn't used that often. [1] Like most such decisions, it was a political decision [2] rather than a technical decision. [2] Strippers and steak, baby! Strippers and steak! And those responsible are not bound by their decisions. We lowly engineers are.
- geofft 10y ago
  > NFS basically assumes a reliable network. Operations should complete more or less immediately. I don't know what you mean by this; are you claiming that NFS' design, by nature of being part of the kernel VFS, only makes sense in a reliable network because it pretends to work like a local filesystem? Then yes, we're discussing ways to fix that. But if you're claiming NFS only works in reliable networks, that's absolutely untrue. NFSv3, at least, is designed to work in unreliable networks by being stateless (dare I say RESTful) and having well-defined semantics for operations that may not have completed. Close-to-open consistency is a way of sacrificing consistency for availability. And so forth. I've used it in production on remarkably unreliable networks and it absolutely works. But, of course, you need the application to be aware that it's using a filesystem that will expose the network's unreliability. By "unreliable network" I mean a real-world network. I'm not suggesting NFS-over-wifi where you're expecting some sort of moshfs, I'm talking about inter-data-center WAN links, LAN links that occasionally get congested, etc. > Filesystem bugs are filesystem bugs -- the application layer shouldn't have to work around them. Shouldn't have to, no. But with the UNIX API, it doesn't even have the option of doing so. And being liberal in what you accept from others is a good way to build robust systems; this continues to apply when "you" is userspace and "others" is kernelspace. For instance, a specific type of application, maybe a high-concurrency web server, might want to add timeouts on filesystem operations that can interrupt the current request and raise a 500 or something, but not abort the entire server. We have exactly this problem with https://lost-contact.mit.edu https://lost-contact.mit.edu, a web server that quite intentionally provides access to all public AFS content via the web. It's running normal Apache on top of the normal OpenAFS client. Sometimes a remote AFS server maintained by someone else will go unresponsive, and eventually all threads on the entire Apache server will hang, one-by-one, making requests to the stuck server. There's no way to solve this in the UNIX design; we'd have to write a userspace AFS client to make this work.
  - AstralStorm 10y ago
    Not true, you can set up an alarm signal to interrupt most syscalls. All fs ones at least.
    - geofft 10y ago

by TheDong 10y ago

loeg 10y ago

> nfs is mentioned a few lines later. Creating directories on any sort of network mount can and will time out NFS basically assumes a reliable network. Operations should complete more or less immediately. How is a syscall-level timeout more useful than a whole-application watchdog for handling flaky mounted filesystems? If you want a flaky network filesystem, I think it's probably better to go with some application-level protocol rather than putting it below Unix's filesystem interface. > FUSE filesystems also can have bugs resulting in mkdir hanging forever. Filesystem bugs are filesystem bugs -- the application layer shouldn't have to work around them.

xenadu02 10y ago
I'm not picking on you specifically but when people say things like "Unix gives you brain damage" this is what they mean. Your suggestion is that the network should be reliable (a strict impossibility) or that every application should use a bespoke protocol to do simple things like working with a network file system. The Unix IO model isn't well designed and is far from the only Unix-ism to work poorly in the face of threads (signals and fork/exec are common offenders; not even memcpy is safe in a signal handler!) This is hardly surprising: it was slapped together ad-hoc over the years.
- loeg 10y ago
  > I'm not picking on you specifically but when people say things like "Unix gives you brain damage" this is what they mean. You understand that's still offensive and implies I have brain damage, right? > every application should use a bespoke protocol to do simple things like working with a network file system. It doesn't have to be bespoke, but I agree that the Unix VFS interface isn't appropriate for it. What operating system are you aware of that provides a mkdir interface with timeout? Windows' CreateDirectory() does not. OS X is just BSD. We're departing the realm of mainstream systems. QNX mkdir does not appear to take a timeout. Plan9 uses create() with a DIR flag, which does not take a timeout. Help me out here?
  - fiddlerwoaroof 10y ago
    I suspect old systems like Multics did. But, this is the sort of thing that Richard Gabriel is talking about in worse is better[1] [1]: https://www.dreamsongs.com/WorseIsBetter.html https://www.dreamsongs.com/WorseIsBetter.html
  - deleted 10y ago
    [deleted]
  - xenadu02 10y ago
    > You understand that's still offensive and implies I have brain damage, right? My apologize, I didn't intend that. I mean that some people are used to Unix and use their familiarity as a post-hoc rationalization for some of the poorly designed aspects of it. To be clear: my position is that several Unix APIs are bad designs. Rather than making excuses for them we should demand better cross-platform APIs. > It doesn't have to be bespoke, but I agree that the Unix VFS interface isn't appropriate for it. I don't agree. Network filesystems should be mountable and traversable just like local filesystems, but the OS and libraries should do more to be async and multi-thread aware, as should applications.
    - pjmlp 10y ago
      > Rather than making excuses for them we should demand better cross-platform APIs. We already have them in the form of language runtimes. POSIX as cross-platform API is mostly relevant to C. Even C++ is moving beyond POSIX, by providing proper OS abstractions on the standard library.
- fulafel 10y ago
  NFS violates a number of basic Unix filesystem guarantees. It's well documented that it violates POSIX guaranteed filesystem semantics. The abstraction NFS chose is "filesystem operations don't time out", and implements this by just making your process hang as long as the network is dead - possibly forever. I'm not sure it's a bad thing that the Unix FS API doesn't do networked filesystems. Everbody is using userspace APIs for remote file access, and the big and varied design space of remote file access semantics doesn't get bottlenecked on 30 year old system call APIs.
- slrz 10y ago
  > The Unix IO model isn't well designed and is far from the only Unix-ism to work poorly in the face of threads Right, threads in the form of pthreads should've never been introduced into Unix, as they made a huge mess of many interfaces. The rfork "you have processes that can share resources" model is a much better fit (although it doesn't solve all problems). It's what Linux as well as FreeBSD implement at the kernel level anyway except they add tons of ugly stuff on top of it to allow for performant implementations of POSIX threads semantics.
TheDong 10y ago
> How is a syscall-level timeout more useful than a whole-application watchdog for handling flaky mounted filesystems? It's more useful because once you enter into a syscall, it's out of the application's control. The kernel's off in its happy place doing whatever it does, and the application has no control over that other than being able to specify timeouts to begin with or having async epoll-like interfaces.
- ec109685 10y ago
  s/application/thread/
- AstralStorm 10y ago
  You can also interrupt syscalls using signals or pthread_cancel.
spc476 10y ago
As if. At $WORK, we service requests (millions per day) that require access to NFS mounted files [1]. A network hiccup could effect one request but not another. I'm not sure how a "whole-application watchdog" would work for us. As it is, the main thread will schedule a "wakeup" from another thread whose sole existence is to interrupt file system calls stuck in NFS. Yes, it would be nice if such code didn't have to exist. But it does. On the plus side, it's there for a feature that isn't used that often. On the bad side, it's there for a feature that isn't used that often. [1] Like most such decisions, it was a political decision [2] rather than a technical decision. [2] Strippers and steak, baby! Strippers and steak! And those responsible are not bound by their decisions. We lowly engineers are.
geofft 10y ago
> NFS basically assumes a reliable network. Operations should complete more or less immediately. I don't know what you mean by this; are you claiming that NFS' design, by nature of being part of the kernel VFS, only makes sense in a reliable network because it pretends to work like a local filesystem? Then yes, we're discussing ways to fix that. But if you're claiming NFS only works in reliable networks, that's absolutely untrue. NFSv3, at least, is designed to work in unreliable networks by being stateless (dare I say RESTful) and having well-defined semantics for operations that may not have completed. Close-to-open consistency is a way of sacrificing consistency for availability. And so forth. I've used it in production on remarkably unreliable networks and it absolutely works. But, of course, you need the application to be aware that it's using a filesystem that will expose the network's unreliability. By "unreliable network" I mean a real-world network. I'm not suggesting NFS-over-wifi where you're expecting some sort of moshfs, I'm talking about inter-data-center WAN links, LAN links that occasionally get congested, etc. > Filesystem bugs are filesystem bugs -- the application layer shouldn't have to work around them. Shouldn't have to, no. But with the UNIX API, it doesn't even have the option of doing so. And being liberal in what you accept from others is a good way to build robust systems; this continues to apply when "you" is userspace and "others" is kernelspace. For instance, a specific type of application, maybe a high-concurrency web server, might want to add timeouts on filesystem operations that can interrupt the current request and raise a 500 or something, but not abort the entire server. We have exactly this problem with https://lost-contact.mit.edu https://lost-contact.mit.edu, a web server that quite intentionally provides access to all public AFS content via the web. It's running normal Apache on top of the normal OpenAFS client. Sometimes a remote AFS server maintained by someone else will go unresponsive, and eventually all threads on the entire Apache server will hang, one-by-one, making requests to the stuck server. There's no way to solve this in the UNIX design; we'd have to write a userspace AFS client to make this work.
- AstralStorm 10y ago
  Not true, you can set up an alarm signal to interrupt most syscalls. All fs ones at least.
  - geofft 10y ago