4 ms·
Why everything which doesn't work as someone expected is called "damned"? These features have been there for decades and we know what to do, what not to do, and
by bayindirh 5y ago
Why everything which doesn't work as someone expected is called "damned"? These features have been there for decades and we know what to do, what not to do, and how to exploit these features.
I personally like how POSIX works, and how well documented its operation and failure modes. I love systems which don't prevent foot guns, and go bang spectacularly when I foot gun myself.
It's also ironic how we don't get to like systems which do limits us in severe ways, and then get angry to a system which doesn't limit us the same way. Fun times.
- ynik 5y agoWe know what to do, but no single developer is in a position to do it -- POSIX file locking requires global coordination between all developers of code running inside a single process. This is fundamentally incompatible with the use of libraries. Observe that there is no way SQLite could fix this bug, the SQLite developers instead have to shove off the responsibility to everyone one using the SQLite library. Add a few more layers of libraries where this requirement isn't documented as clearly, and this is basically guaranteed to go wrong somewhere.
- zvrba 5y ago> Why everything which doesn't work as someone expected is called "damned"? Because POSIX doesn't provide features that are necessary for reliability. Workarounds, if they exist, are highly platform-specific. That I'd say is damnation of the standard. For example, how do you, within POSIX, ensure that a write to the storage media is "durable"? People argue about what combination of various "sync" calls and open flags you need to achieve this. (Disregard the case of hardware cheating -- nothing can be done about that.) How do you make an "owning" lock on a file: a kind of that you and only you (the owner) can remove? Why is a lock applied per for file descriptor and not the file object? What is the workaround here? In general, POSIX + threads = damnation because there's too much per-process global state, which was kind of OK "before MT". It's clear that the standard was written before multithreading was even foreseen to be the norm, and it contains many SNAFUs wrt threads, not just files. (Signals and fork are the first to come to mind.) Oh, yeah, don't mention Linux. It has so many extensions and additions to POSIX to alleviate POSIX messups that it's not even funny.
- GauntletWizard 5y agoIn posix, you call "sync". Everything beyond that is workarounds for hardware and software level cheating. There are good reasons that modern filesystems aren't quite as simple as that, and some of those reasons are shameful, but it's not on POSIX. Edit: I just reread the spec, and it is on POSIX: They recognized that "sync" was not actually writing, and documented the real usage. Unix 5 has this to say about sync: > Sync causes all information in core memory that should be on disk to be written out..This includes modifi~ super blocks, modified i-nodes, and delayed block I/0. > It should be used by programs which examine a file system, for example check, d.£ etc. It is mandatory before a boot. But by System 6, Lion's book is already referring to the peculiarities of delayed writes.
- jmalicki 5y agoThere was also the entire fiasco where postgres and Linux kernel developers discovered it was damn-near impossible to get the intended behavior with fsync as there were way too many pitfalls, underspecification, and filesystem-level incompatibilities to know what happened in the non-happy case. https://www.postgresql.org/message-id/CAMsr+YHh+5Oq4xziwwoEfhoTZgr07vdGG+hu=1adXx59aTeaoQ@mail.gmail.com https://www.postgresql.org/message-id/CAMsr+YHh+5Oq4xziwwoEf...
- ufmace 5y agoSeems to me, it's a lot tougher to make all of this stuff always work right in the face of buggy or hostile programs. If you have a super-serious hardcore owner lock on a file that only the opening process can release, what happens when a buggy or hostile program locks a file and never unlocks it, even after exiting?
- imtringued 5y agoThere is nothing more beautiful than not being able to delete malware because it is currently running.
- mannykannot 5y ago> I love systems which don't prevent foot guns, and go bang spectacularly when I foot gun myself. Do I need to say anything more?
- bayindirh 5y ago> Do I need to say anything more? No. Let me re-iterate what I mean. Systems without guardrails are explicit, transparent and flexible. It's easy to understand how they behave. It might be a little harder to get things right, but when you get it right, there's nothing to second guess. Systems without guardrails are low in overhead. It's possible to get raw performance. Yes, I love these systems. I might spend a little more time and need a little more concentration when I develop stuff on these systems, but when it runs, it runs for good. Trying to make systems foolproof by limiting them is not good. If that's good, we should all love iOS, right?
- mannykannot 5y agoYour argument against footgun avoidance is only applicable in narrow cases. In aviation, for example, there are quite a few footguns you can kill yourself with - and, which is very relevant to the issue in general, other people as well - and the practice is to mitigate them wherever feasible. We have stall warnings and stick pushers, ground proximity warnings and TCAS, ABS and interlocks to prevent doing things like engaging reverse thrust when the wheels are off the ground. Now, with software-controlled fly-by-wire systems, the ability to mitigate pilot footguns has been greatly expanded, and is being taken advantage of. Furthermore, these mitigations appeared first in high-performance military aviation, before fly-by-wire appeared in commercial aircraft. For a more specifically computer-systems example, perhaps we should consider security? - oh, wait...
- bayindirh 5y agoThe analogy is not very accurate I may say, because in a flight, you prevent human footguns during flight, every day. However, a developer need to avoid that footgun once, and when the developer verifies the scenario with testing, this part of the code can be considered "sealed", never to be touched again unless a bug is found, or a serious need arises to enter that part of the code. Most security problems arise from memory management problems from my experience. This has nothing to do with POSIX. The problem is universal there, and we may delightfully argue that a stricter programming language like Rust can alleviate most problems, however tools like Valgrind can also find a lot of problems during a simple testing run. If we return to aviation, flight safety and memory management, there are some standards AFAIK which ponder about determinism and memory management on these systems (and smaller "single use" systems). These systems neither know POSIX, nor has the capacity to do so (they spend all their power to catch and tag the object they're tasked with). Again, the security, reliability and other footgun related stuff ends at the program we're designing and running, and not on POSIX itself.
- zaphar 5y agoI love systems which don't prevent foot guns, and go bang spectacularly when I foot gun myself. It's good that you know what you like. Just make sure that none of your software is run by me because when your software goes bang spectacularly and foot guns me instead of you I emphatically don't like it.
- bayindirh 5y agoNeither do I. That's because all my code is tested for both leaks and all scenarios. None of my code ever have gone bang in production. To be honest, no service I have written ever restarted outside system reboots or configuration changes. Having systems with no guardrails doesn't equate to having bad code automatically.
- tzmudzin 5y ago> my code is tested for both leaks and all scenarios That‘s a bold claim. Reeks of hubris though. Healthy scepticism would convince me a lot more that we can trust your products. You may also consider if maintaining that test suite for „all cases” is a good investment of your time.
- bayindirh 5y agoActually, there's no hubris. I'd rather put the money where my mouth is. Let's take an example which I worked on for almost a decade (and still working on). How can I make sure that I have no leaks? 1. I design software by hand. I design construction and destruction chains beforehand. 2. I implement the modules one by one, create some test suites, plug to valgrind, make sure that it has no leaks. 3. Chain the modules together, re-run the tests. 4. For every component and chain which passes the test, "Seal" the unit. Any change requires whole set of tests again. For this set of components, since components doesn't change, test suites are also "sealed". After every build, I have a CI/CD pipeline which runs a series of tests including, unit, module and end to end scenarios. Test the result with ground truth up to 32 significant digits. If something doesn't hold, flag the build and fail. I also keep timing values for certain states, and they shouldn't deviate much. Remember, we need speed. We should have invalid inputs. However these inputs shouldn't reach to the processing state and just be marked invalid and thrown out. Apply the above pipeline. Implement, test and seal. Since the stack is stable, and this is an "old school" C++ code, I don't need to migrate libraries and other stuff around much. So, the test suite I've written doesn't need maintenance unless the seal is broken. However, with the feature I'm implementing, I need to break a couple of these seals, but no biggie. Extend a function, write a couple more unit tests, run the test suite and hammer the code, seal it again. Usual tests will go on, of course. End to end valgrind tests are done periodically, with not every build. It takes around ~12 hours to complete that with full tracing and reporting. I'd rather be methodical and give the code I've written a torturous shake down, instead of saying this looks good and move on. I trust myself with the code I write, but not so blindly to go over the top and say that "I'm the one". Instead I do my best, but I believe that I'm the worst coder around here, so I test to break my code. Not to validate.
- prirun 5y agoI also like it when systems go bang spectacularly on a failure, vs silently allowing it. But, related to the Posix thing about reusing fd's, I just ran into this a couple of days ago and it took me 6 hours to figure out. I made a change in HashBackup to interrupt saving a big file if the backup time limit had been reached. Tested it a while, seemed like it worked fine. I was using a 15-second timeout. For whatever reason I tried it with a 5s timeout. This went bang, but in a completely different area of the program responsible for copying backup files to a destination, where it raised an exception "hey, this file was X bytes but I only copied Y". Go look at the source file, it's fine. Look at the destination file, and indeed, it is only Y bytes. Go look at the copy code, it looks fine but obviously isn't, so I start putting all kinds of debug stuff there to check file sizes, do double reads at EOF, ... nothing helps. Then I add an lseek to report the current file position and indeed it is at X, even though only Y bytes have been copied. So I realize that something else has moved this file pointer on me. To help track that down, I override Python's os.open and os.close and display the pathname and fd so I have a history of who is using what fd's. After going through that with a microscope, I see that the file being backed up is open on fd 6, and when the timeout occurs, it gets closed. Then that fd gets used by the copy function for the source file. BUT, there is an asynchronous read process used during the backup. It has exception handling if an fd is closed by higher levels and the read thread gets an EBADF error, then resets itself. But there is a race condition: if the fd gets reopened fast enough, the read thread will never see the EBADF error and thinks it is still reading from the file to backup, which it continues to do. Now there are 2 processes reading from the same file and mayhem ensues. Of course it's ultimately my fault, but we do have at least 32 bits for a file descriptor number. It would be a lot nicer if Posix kept incrementing it instead of using the low numbers and then wrapped around, like Unix PIDs do. And yeah, I realize this would screw up select, dup, etc because of the way they are designed.
- bhawks 5y ago> To help track that down, I override Python's os.open and os.close and display the pathname and fd so I have a history of who is using what fd's. Nice! I'd recommend trying strace to track system calls in the future though. Monkey patching open and close will only catch code using those functions and not any c libraries or very smart people using ctypes. Strace captures every syscall and dumps the parameters and returns.
- dolni 5y ago> Why everything which doesn't work as someone expected is called "damned"? > ... > It's also ironic how we don't get to like systems which do limits us in severe ways, and then get angry to a system which doesn't limit us the same way. Fun times. You are conflating surprising systems and flexible systems. Something that is flexible doesn't have to be surprising. Nor does something surprising have to be flexible. The Principle of Least Surprise says that surprising behavior in software is bad. Systems should strive not to do things that catch users off guard absent a good reason.
- bayindirh 5y agoI personally don't get surprised by POSIX to consider it's violating this "least surprise" principle. Instead, for example Java has given me much more surprises, and at catastrophic levels. I was using Choco Solver [0] back in the day, and I created two instances of it, attached to different classes. Which is perfectly normal, right? Somehow they've cross linked between these two instances, affected the results they have computed, and created persistent memory leaks which needed system reboots to claim back. Java should be immune to that, but no. Preventing that needed to run only one instance of Choco, which limited my performance greatly. Luckily, the system had a queue/consumer structure, so running only one didn't need extensive changes. [0]: https://choco-solver.org/ https://choco-solver.org/
- zvrba 5y ago> Java should be immune to that, but no. I can't comment on Choco, but no garbage collector is immune against programmer's mistakes. For example, it's easy to create leaks with observable pattern (short-lived observers not unsubscribing from a long-lived observable). IOW: know when to use weak references.