3 ms·

> how much better you could do by skipping the OS entirely I'd say this is the main reason distributed systems even exist, because no amount of crash proofing

> how much better you could do by skipping the OS entirely I'd say this is the main reason distributed systems even exist, because no amount of crash proofing file writes for the filesystem is going to make disks, raids correct and reliable, individual machines, datacenters reliable, operational mistakes not happen, interpretation and other mistakes not happen and so on. On the other hand consistency on random writes to files is not something absolutely necessary. We can get away with just sequential writes and there would be not much to reorder and to mess up to begin with.

josephg 9y ago
> On the other hand consistency on random writes to files is not something absolutely necessary. This stuff does matter; because most applications aren't distributed systems. Using file system operations in the way they were intended to be used shouldn't sometimes accidentally corrupt files by default. Look again at the table in the article showing consistency bugs in application code. They even found bugs in sqlite. If not even sqlite does this stuff correctly, mortals have no hope. And distributed systems themselves depend on the guarantees provided by filesystems for correctness. The guarantees filesystems provide are poorly documented, poorly understood and inconsistently implemented across filesystems and operating systems. Take some time to read about what redis, postgres and sqlite do to correctly write files - its an awful morass. At a minimum we should have a standard atomic write primitive that does all the stuff listed at the top of that article. I don't really care how slow it is - atomic writes are what everyone expects write() to actually do. Atomic write shouldn't be poorly reimplemented by every text editor and configuration tool. Video games shouldn't need to ask players not to switch off their consoles while they're saving. (What an absolute embarrassment.) If filesystems want to compete on performance, compete on the performance of atomic append. Right now any atomic append implementation in userland is both complicated and dog slow. With help from kernel-level filesystem code, we can do way better on both counts. Then for high performance applications (databases, etc) the OS should expose either fence() operations or some sort of durable write order. Thats the API databases will actually use. For extra credit I would also like a write API that notified me when my data has been durably synced to disk. A kqueue event would be perfect. And once we have those APIs I have no idea when anyone would use the current write/fsync implementation. Its a huge foot-gun. Its what you would get if mongodb made a filesystem API.
- zzzcpan 9y ago
  These days mere mortals have access to enough information about brokenness of operating systems, filesystems, disks, raids to make better decisions, than sqlite ever did. I even suggest to go purely log structured and completely get rid of random writes to save people some time. Also these days I disagree about perfect APIs, they are not realistically achievable. It's a mistake to think there will be no mistakes. For example, another common mistake people make is in event loops, when they assume kernels are going to report descriptors as readable or writable correctly all the time for correct descriptors. These are all mistakes that come from the assumption that other people don't make mistakes, instead of treating lower levels as black boxes and not relying on promises. So, while it sounds like a nice idea to have yet another API to fix broken promises, it's not. A good library can fix broken promises. A bad library still won't realize it's relying on broken promises.
  - josephg 9y ago
    I recommend log structured too, although probably for different reasons ( https://twitter.com/josephgentle/status/978616782607998977 https://twitter.com/josephgentle/status/978616782607998977 ) > It's a mistake to think there will be no mistakes. Its more programming philosophy at this point, but I disagree with this. There's lots of assumptions we 100% rely on for modern software. Things like ADD / MUL working correctly in our CPUs, TCP delivering packets in the order they were sent, correctness in our programming languages, the correctness of cryptographic primitives and the safety of operations like constant-time comparisons. Database transactions. Malloc. GC. Etc etc. Actually I would say that "it works all the time" is the norm in computing. Its so pervasive and great that when things work all the time we don't even need to think about them or understand them at all. Not like "it works almost all the time". Things that only work most of the time we have to think about constantly. "Does my page still render correctly in all browsers?", etc. Its easy to find examples of API promises being violated because these problems take up all our time. But this is no way to do engineering. In aggregate its almost always far slower than just spending the time to fix the bugs themselves. And fixing the actual issue will usually take less code, and run faster than making a pile of abstractions around the bug. An example of this: A few years ago I worked at a startup where we had a search indexer. The indexer wasn't architected well, and it kept having issues and leaving stale data around. So we kept needing to invest engineering time into fixing small search issues. Patching around it was a constant low level time sink for years. A few years later I was asked to implement search at a different company. I used the lessons I learned at the startup to do it right this time. And the code I wrote has been almost untouched to this day. Now, the interesting part is that at both companies management thought the amount of time we were allocating toward search seemed normal. But the second approach was objectively better. At the startup it would have taken less time overall to fix the indexer's bugs (or rewrite it) than we spent patching around it and dealing with its brokenness. Which is to say, sometimes better is better.

josephg 9y ago

> On the other hand consistency on random writes to files is not something absolutely necessary. This stuff does matter; because most applications aren't distributed systems. Using file system operations in the way they were intended to be used shouldn't sometimes accidentally corrupt files by default. Look again at the table in the article showing consistency bugs in application code. They even found bugs in sqlite. If not even sqlite does this stuff correctly, mortals have no hope. And distributed systems themselves depend on the guarantees provided by filesystems for correctness. The guarantees filesystems provide are poorly documented, poorly understood and inconsistently implemented across filesystems and operating systems. Take some time to read about what redis, postgres and sqlite do to correctly write files - its an awful morass. At a minimum we should have a standard atomic write primitive that does all the stuff listed at the top of that article. I don't really care how slow it is - atomic writes are what everyone expects write() to actually do. Atomic write shouldn't be poorly reimplemented by every text editor and configuration tool. Video games shouldn't need to ask players not to switch off their consoles while they're saving. (What an absolute embarrassment.) If filesystems want to compete on performance, compete on the performance of atomic append. Right now any atomic append implementation in userland is both complicated and dog slow. With help from kernel-level filesystem code, we can do way better on both counts. Then for high performance applications (databases, etc) the OS should expose either fence() operations or some sort of durable write order. Thats the API databases will actually use. For extra credit I would also like a write API that notified me when my data has been durably synced to disk. A kqueue event would be perfect. And once we have those APIs I have no idea when anyone would use the current write/fsync implementation. Its a huge foot-gun. Its what you would get if mongodb made a filesystem API.

zzzcpan 9y ago
These days mere mortals have access to enough information about brokenness of operating systems, filesystems, disks, raids to make better decisions, than sqlite ever did. I even suggest to go purely log structured and completely get rid of random writes to save people some time. Also these days I disagree about perfect APIs, they are not realistically achievable. It's a mistake to think there will be no mistakes. For example, another common mistake people make is in event loops, when they assume kernels are going to report descriptors as readable or writable correctly all the time for correct descriptors. These are all mistakes that come from the assumption that other people don't make mistakes, instead of treating lower levels as black boxes and not relying on promises. So, while it sounds like a nice idea to have yet another API to fix broken promises, it's not. A good library can fix broken promises. A bad library still won't realize it's relying on broken promises.
- josephg 9y ago
  I recommend log structured too, although probably for different reasons ( https://twitter.com/josephgentle/status/978616782607998977 https://twitter.com/josephgentle/status/978616782607998977 ) > It's a mistake to think there will be no mistakes. Its more programming philosophy at this point, but I disagree with this. There's lots of assumptions we 100% rely on for modern software. Things like ADD / MUL working correctly in our CPUs, TCP delivering packets in the order they were sent, correctness in our programming languages, the correctness of cryptographic primitives and the safety of operations like constant-time comparisons. Database transactions. Malloc. GC. Etc etc. Actually I would say that "it works all the time" is the norm in computing. Its so pervasive and great that when things work all the time we don't even need to think about them or understand them at all. Not like "it works almost all the time". Things that only work most of the time we have to think about constantly. "Does my page still render correctly in all browsers?", etc. Its easy to find examples of API promises being violated because these problems take up all our time. But this is no way to do engineering. In aggregate its almost always far slower than just spending the time to fix the bugs themselves. And fixing the actual issue will usually take less code, and run faster than making a pile of abstractions around the bug. An example of this: A few years ago I worked at a startup where we had a search indexer. The indexer wasn't architected well, and it kept having issues and leaving stale data around. So we kept needing to invest engineering time into fixing small search issues. Patching around it was a constant low level time sink for years. A few years later I was asked to implement search at a different company. I used the lessons I learned at the startup to do it right this time. And the code I wrote has been almost untouched to this day. Now, the interesting part is that at both companies management thought the amount of time we were allocating toward search seemed normal. But the second approach was objectively better. At the startup it would have taken less time overall to fix the indexer's bugs (or rewrite it) than we spent patching around it and dealing with its brokenness. Which is to say, sometimes better is better.