5 ms·

I can't claim to speak for the designers, but my intuition (having had my first Unix exposure in the early/mid-80s) is that their assumptions were along the lin

by mmt 8y ago

I can't claim to speak for the designers, but my intuition (having had my first Unix exposure in the early/mid-80s) is that their assumptions were along the lines that it would be used consistent with the part of the Unix philosophy stating: > Make each program do one thing well. To do a new job, build afresh rather than complicate old programs by adding new "features". This seems inconsistent with the (now totally common) programming styles that use multiple, unrelated libraries or have a single program do something that requires keeping files with locks open, while going off on a tangent to scan a directory. Back then, I suspect the standard practice would be to fork() if not outright delegate the task to existing tools like find/xargs. I certainly agree that hidden side effects like closing a duplicate fd causing all the locks to disappear is undesirable, but it might not even have occurred to them that anyone would care. Maybe they thought about it and the alternative would be too memory intensive (remember when memory was actually scarce?). It would be interesting if anyone had pointers to something like actual mailing list archives. Characterizing it as "just braindead" as uncharitable, when it's decades later, without at least being able to point to contemporaneous pressures and constraints.

zAy0LfpBZLC8mAC 8y ago
Well, take a simpler case: A program that takes a lock and then reads a file specified by the user on the command line. There are tons of perfectly reasonable scenarios that are hilariously complicated to implement correctly, even ones that fit within the more traditional unix approach. Yes, maybe this is a result of some sort of tradeoff, but I would say it almost certainly was not a sensible tradeoff even back then. I don't see how using a sensible strategy could be much worse in terms of any resources used (I mean, really, it's a matter of where you put the pointers to the lock structures and where you call the automatic unlock handler), and the costs of using the semantics as they are now correctly are so high it's just not a useful abstraction really. If anything, this smells like one of those tradeoffs where a terrible abstraction that is incredibly difficult to use correctly was chosen simply because it was less code to write, while thus causing every user of the abstraction to have to write tons more code if they want to have correct code. Though if you ask me this looks much more like someone not thinking it through. Locks are for coordinating concurrency, the unit of concurrency is the process, so locks are held by processes. Oh, and we have to clean up locks that aren't released by the process! Well, let's do that when the file is closed ... OK, done!
- mmt 8y ago
  > Well, take a simpler case: A program that takes a lock and then reads a file specified by the user on the command line. I'm having trouble seeing the use case under that Unix Philosophy constraint. I admit I might not be imaginative enough, but a program/utility that would be normally expected to operate on its own locked file (by being asked to do so by the user) still seems incongruous. > it was less code to write, while thus causing every user of the abstraction to have to write tons more code if they want to have correct code. Actually, that's a completely valid design decision and philosophically compatible, even if you happen to find it distasteful. It's one of the (caricatured) tenets of "Worse is Better" which has been credited with the (original) success of Unix over more "correct" competitors http://dreamsongs.com/RiseOfWorseIsBetter.html http://dreamsongs.com/RiseOfWorseIsBetter.html : > Simplicity -- the design must be simple, both in implementation and interface. It is more important for the implementation to be simple than the interface. Simplicity is the most important consideration in a design. > Correctness -- the design must be correct in all observable aspects. It is slightly better to be simple than correct. > Consistency -- the design must not be overly inconsistent. Consistency can be sacrificed for simplicity in some cases, but it is better to drop those parts of the design that deal with less common circumstances than to introduce either implementational complexity or inconsistency. > Completeness -- the design must cover as many important situations as is practical. All reasonably expected cases should be covered. Completeness can be sacrificed in favor of any other quality. In fact, completeness must be sacrificed whenever implementation simplicity is jeopardized. Consistency can be sacrificed to achieve completeness if simplicity is retained; especially worthless is consistency of interface. It seems like advisory locking has made a "worse" sacrifice in each category, in favor of the purported benefit. There's even a reference (though I don't recall if it's in that essay, specifically) to making the user do extra coding work for correctness. Of course, there are plenty of critics of "Worse is Better" or the "New Jersey approach", including the author of that essay. However, his whole point is that it improves survivability, which, arguably, was still important to Unix in the 80s and maybe even into the 90s. > not thinking it through. Locks are for coordinating concurrency, the unit of concurrency is the process, so locks are held by processes. Oh, and we have to clean up locks that aren't released by the process! Well, let's do that when the file is closed It's absolutely not thinking every possible future scenario and corner case through, but that, I would argue, is the opposite of "braindead". It worked, especially for the use case you described, which seems perfectly reasonable for early Unix (and adherents to the Unix Philosophy of any era).
xenadu02 8y ago
Nope. ATT’s implementation had this really stupid behavior and they snuck it into POSIX because that meant they wouldn’t have to change their existing code. No one else realized how stupid it was until it was too late. The design is stupid. There is no excuse for it.
- nwellnhof 8y ago
  Right. This article by Jeremy Allison sheds some light on how this broken design ended up in POSIX. https://www.samba.org/samba/news/articles/low_point/tale_two_stds_os2.html https://www.samba.org/samba/news/articles/low_point/tale_two... > The reason is historical and reflects a flaw in the POSIX standards process, in my opinion, one that hopefully won't be repeated in the future. I finally tracked down why this insane behavior was standardized by the POSIX committee by talking to long-time BSD hacker and POSIX standards committee member Kirk McKusick (he of the BSD daemon artwork). As he recalls, AT&T brought the current behavior to the standards committee as a proposal for byte-range locking, as this was how their current code implementation worked. The committee asked other ISVs if this was how locking should be done. The ISVs who cared about byte range locking were the large database vendors such as Oracle, Sybase and Informix (at the time). All of these companies did their own byte range locking within their own applications, none of them depended on or needed the underlying operating system to provide locking services for them. So their unanimous answer was "we don't care". In the absence of any strong negative feedback on a proposal, the committee added it "as-is", and took as the desired behavior the specifics of the first implementation, the brain-dead one from AT&T.
  - mmt 8y ago
    That's the first design-related documentation I've seen presented, so thank you. However, it's merely an indictment of the POSIX standards process and sheds no light on why the AT&T implementation was the way it was in the first place. I keep reading what, to me (essentially an outsider with no horse in the race), sound like either hyperboles at worst ("brain-dead", "really stupid", "no excuse") and arguments advocating the MIT Method (of "Worse is Better" fame) at best. It tends to beg the question, "If it's so horrible, why did anyone bother to implement it that way or use it once it was there?" I'd expect, if it's actually as broken as everyone makes it out to be, that it would be worse than nothing and would get no use. Apparently, the major db vendors didn't bother, at least at the time, but that could well be because there was no reliable [1] cross-platform option. So, again, how about some actual, contemporaneous evidence of the original design process, for a fair, contextual critique? [1] i.e. reliable or good enough, not reliable in the sense of implementation correctness covering all corner cases, as seems to be demanded by certain commenters and the MIT Method in general.
aasasd 8y ago
Eeeh, this seems like a far fetched interpretation of the principle. The dbm library, for one, was around since '79,and I'd guess Ken Thompson would adhere to the principles instead if they were so pervasive.
- mmt 8y ago
  I wasn't asserting that they were pervasive or even universal (in the Unix universe). Perhaps I'm missing your point about dbm, unless it's just that libraries existed, even long ago, that happened to open files. That doesn't refute my point, which is that a process operating on those files via that library and outside of that library is incongruous. As further evidence for there being at least an idea of such a dual "class" of files, I present the fact that the * glob does not match files starting with a dot.

by mmt 8y ago

zAy0LfpBZLC8mAC 8y ago

Well, take a simpler case: A program that takes a lock and then reads a file specified by the user on the command line. There are tons of perfectly reasonable scenarios that are hilariously complicated to implement correctly, even ones that fit within the more traditional unix approach. Yes, maybe this is a result of some sort of tradeoff, but I would say it almost certainly was not a sensible tradeoff even back then. I don't see how using a sensible strategy could be much worse in terms of any resources used (I mean, really, it's a matter of where you put the pointers to the lock structures and where you call the automatic unlock handler), and the costs of using the semantics as they are now correctly are so high it's just not a useful abstraction really. If anything, this smells like one of those tradeoffs where a terrible abstraction that is incredibly difficult to use correctly was chosen simply because it was less code to write, while thus causing every user of the abstraction to have to write tons more code if they want to have correct code. Though if you ask me this looks much more like someone not thinking it through. Locks are for coordinating concurrency, the unit of concurrency is the process, so locks are held by processes. Oh, and we have to clean up locks that aren't released by the process! Well, let's do that when the file is closed ... OK, done!

mmt 8y ago
> Well, take a simpler case: A program that takes a lock and then reads a file specified by the user on the command line. I'm having trouble seeing the use case under that Unix Philosophy constraint. I admit I might not be imaginative enough, but a program/utility that would be normally expected to operate on its own locked file (by being asked to do so by the user) still seems incongruous. > it was less code to write, while thus causing every user of the abstraction to have to write tons more code if they want to have correct code. Actually, that's a completely valid design decision and philosophically compatible, even if you happen to find it distasteful. It's one of the (caricatured) tenets of "Worse is Better" which has been credited with the (original) success of Unix over more "correct" competitors http://dreamsongs.com/RiseOfWorseIsBetter.html http://dreamsongs.com/RiseOfWorseIsBetter.html : > Simplicity -- the design must be simple, both in implementation and interface. It is more important for the implementation to be simple than the interface. Simplicity is the most important consideration in a design. > Correctness -- the design must be correct in all observable aspects. It is slightly better to be simple than correct. > Consistency -- the design must not be overly inconsistent. Consistency can be sacrificed for simplicity in some cases, but it is better to drop those parts of the design that deal with less common circumstances than to introduce either implementational complexity or inconsistency. > Completeness -- the design must cover as many important situations as is practical. All reasonably expected cases should be covered. Completeness can be sacrificed in favor of any other quality. In fact, completeness must be sacrificed whenever implementation simplicity is jeopardized. Consistency can be sacrificed to achieve completeness if simplicity is retained; especially worthless is consistency of interface. It seems like advisory locking has made a "worse" sacrifice in each category, in favor of the purported benefit. There's even a reference (though I don't recall if it's in that essay, specifically) to making the user do extra coding work for correctness. Of course, there are plenty of critics of "Worse is Better" or the "New Jersey approach", including the author of that essay. However, his whole point is that it improves survivability, which, arguably, was still important to Unix in the 80s and maybe even into the 90s. > not thinking it through. Locks are for coordinating concurrency, the unit of concurrency is the process, so locks are held by processes. Oh, and we have to clean up locks that aren't released by the process! Well, let's do that when the file is closed It's absolutely not thinking every possible future scenario and corner case through, but that, I would argue, is the opposite of "braindead". It worked, especially for the use case you described, which seems perfectly reasonable for early Unix (and adherents to the Unix Philosophy of any era).

xenadu02 8y ago

Nope. ATT’s implementation had this really stupid behavior and they snuck it into POSIX because that meant they wouldn’t have to change their existing code. No one else realized how stupid it was until it was too late. The design is stupid. There is no excuse for it.

nwellnhof 8y ago
Right. This article by Jeremy Allison sheds some light on how this broken design ended up in POSIX. https://www.samba.org/samba/news/articles/low_point/tale_two_stds_os2.html https://www.samba.org/samba/news/articles/low_point/tale_two... > The reason is historical and reflects a flaw in the POSIX standards process, in my opinion, one that hopefully won't be repeated in the future. I finally tracked down why this insane behavior was standardized by the POSIX committee by talking to long-time BSD hacker and POSIX standards committee member Kirk McKusick (he of the BSD daemon artwork). As he recalls, AT&T brought the current behavior to the standards committee as a proposal for byte-range locking, as this was how their current code implementation worked. The committee asked other ISVs if this was how locking should be done. The ISVs who cared about byte range locking were the large database vendors such as Oracle, Sybase and Informix (at the time). All of these companies did their own byte range locking within their own applications, none of them depended on or needed the underlying operating system to provide locking services for them. So their unanimous answer was "we don't care". In the absence of any strong negative feedback on a proposal, the committee added it "as-is", and took as the desired behavior the specifics of the first implementation, the brain-dead one from AT&T.
- mmt 8y ago
  That's the first design-related documentation I've seen presented, so thank you. However, it's merely an indictment of the POSIX standards process and sheds no light on why the AT&T implementation was the way it was in the first place. I keep reading what, to me (essentially an outsider with no horse in the race), sound like either hyperboles at worst ("brain-dead", "really stupid", "no excuse") and arguments advocating the MIT Method (of "Worse is Better" fame) at best. It tends to beg the question, "If it's so horrible, why did anyone bother to implement it that way or use it once it was there?" I'd expect, if it's actually as broken as everyone makes it out to be, that it would be worse than nothing and would get no use. Apparently, the major db vendors didn't bother, at least at the time, but that could well be because there was no reliable [1] cross-platform option. So, again, how about some actual, contemporaneous evidence of the original design process, for a fair, contextual critique? [1] i.e. reliable or good enough, not reliable in the sense of implementation correctness covering all corner cases, as seems to be demanded by certain commenters and the MIT Method in general.

aasasd 8y ago

Eeeh, this seems like a far fetched interpretation of the principle. The dbm library, for one, was around since '79,and I'd guess Ken Thompson would adhere to the principles instead if they were so pervasive.

mmt 8y ago
I wasn't asserting that they were pervasive or even universal (in the Unix universe). Perhaps I'm missing your point about dbm, unless it's just that libraries existed, even long ago, that happened to open files. That doesn't refute my point, which is that a process operating on those files via that library and outside of that library is incongruous. As further evidence for there being at least an idea of such a dual "class" of files, I present the fact that the * glob does not match files starting with a dot.