4 ms·
The point of the article is how if you want the filesystem API to write data safely, you need to jump through a lot of hoops. (Which, spoilers: most developers
by josephg 6y ago
The point of the article is how if you want the filesystem API to write data safely, you need to jump through a lot of hoops. (Which, spoilers: most developers don't do. And even developers who try really hard often get it wrong because the APIs are so awful and poorly documented. Hence the ever present spectre of data corruption.)
According to the article, if I use ext4 with journaling and have a file with "a foo" and write "bar" at position 2, possible resulting states are: "a foo", "a boo", "a bao" or "a bar". This is the best scenario - without journaling, I can also end up with uninitialized junk.
The authors of the filesystem APIs consider this to be correct behaviour. But I can't think of any use case for this nonsense behaviour. Real data, in every file on your computer is structured. And structured files simply can't handle this kind of data corruption. Maybe this API has slightly higher throughput, but by the time you've peppered your code with fsync() calls to make it correct, you've killed performance. And working around that takes whole philosophies of threading and thousands of lines of code.
I want that write call to either complete completely or fail completely. Actually, I want more than that - for a bunch of successive write calls touching multiple files to collectively succeed together or fail together. Which is not what the article is saying.
> Its like saying, well software just shouldnt have bugs
The kernel shouldn't have bugs in its filesystem. We don't tolerate bugs in the scheduler, or in OpenSSL or in the CPU. If Postgres randomly corrupted its database every month or two, we would fix the bug. Or call it broken and use something else. If SQLite can implement transactions on top of the current filesystem APIs, then obviously its possible to put that behaviour in the kernel. And in the kernel, that code would be easier to optimize, and would be useful everywhere.
If it was a simple bug, we would fix it. The problem is that the bug is that we have the wrong abstraction. Why do we have the current filesystem API? The story I heard is it was written by some intern at Berkeley decades ago who thought it would be neat to have files and network streams use the same API. We know more now, and its past time to fix the mistake.
> Thats why all the FS as database efforts in the early 2000s like Longhorn failed.
I don't buy the cynicism. Just because microsoft overengineered longhorn (and successfully shipped TxF btw), doesn't mean the problem is unsolvable.
- BlueTemplar 6y agoAbstraction has a cost, and there are no guarantees that if implemented, the resulting performance would be considered acceptable. (It's even possible that many have already tried, and failed.) Of course this doesn't mean that we shouldn't try again and again. But we also have to realize that even if we're successful, this doesn't mean that the solution won't be left completely forgotten, as long as 'good enough' systems exist. (See the other threads for examples.)
- josephg 6y agoThats fine. Disks are so fast now that I would trade away 50% of the filesystem's performance for correctness in 95% of the software I use. (Eg my code editor, git, compiler, etc). Except I bet in a good implementation the actual performance cost of transactions would be negligible anyway. Especially on top of something like ZFS. And legacy software can just keep using the current unreliable APIs with all the performance that entails anyway.
- the8472 6y ago> According to the article, if I use ext4 with journaling and have a file with "a foo" and write "bar" at position 2, possible resulting states are: "a foo", "a boo", "a bao" or "a bar". It makes a lot more sense if you consider it shared, concurrently accessed memory. If you try to access it without atomics you can get funny results. And if you kill one process in the middle you might be stuck with reordered accesses indefinitely. Something like ext4 provides more than that. If you want ACID then you'll have to opt into that. The same as you have to do with concurrent programming (e.g. via std::atomic). But there are many cases where none of this is needed, where all you need is a bag to hold a few bytes for a while. E.g. cache files or large media blobs. The only issue is ubiquity of the necessary tooling, or the lack thereof. Many standard libraries now ship with concurrency-safe data structures while you often have to reach for 3rd-party libraries for safe file abstractions if you don't want to code your own. > If SQLite can implement transactions on top of the current filesystem APIs, then obviously its possible to put that behaviour in the kernel. If userspace can already implement it (when needed) based on provided kernel primitives, why should the kernel maintain two APIs side by side? Userspace needs many different levels of granularity. Sometimes integrity doesn't matter at all. Sometimes it only matters that writes make it to disk in correct order but it's ok for the last ones to get lost. And sometimes you need multiple processes memory-mapping a file and doing shared byte-level access and only random byte-ranges need to be committed to storage at any given time.
- josephg 6y ago> It makes a lot more sense if you consider it shared, concurrently accessed memory. But its not concurrently accessed memory - unlike memory you often want filesystem operations to run asynchronously. And even if you want to think about it like memory, we're missing all the tools we have for managing concurrently accessed memory like atomics and granular load and store barriers. And even then only about 1 in 10000 programmers know how to use those primitives to implement correct software anyway, so thats not exactly what I'd reach for when I think of good API design. But even if you want to use that analogy, the filesystem API doesn't have APIs for granular memory barriers. Until recently all we had was fsync. Peppering your code with fsync calls destroys your performance even though it doesn't change your program's behaviour. And if you batch up your changes and then fsync returns an error, you have no idea which writes succeeded and which failed. inotify helps, but oh goodness does it get complicated. > If userspace can already implement it (when needed) based on provided kernel primitives, why should the kernel maintain two APIs side by side? This is a good question. I guess I don't really think much about caches, or processes sharing a pseudofile to communicate. For those programs, sure - the current behaviour is fine. I'm thinking about the 95% of applications out there which want to use the filesystem to save user data to disk. Those programs shouldn't randomly end up in a corrupt state if power goes out at the wrong time. This is the core use case for the filesystem and I think its pretty reasonable for the kernel to make that case easy and correct by default, without resorting to storing all user data in every application in sqlite. From Linus talking about this issue ( https://lkml.org/lkml/2009/3/25/632 https://lkml.org/lkml/2009/3/25/632 ): > The problem is not that we have a lot of fsync() calls. Quite the reverse. fsync() is really really rare. So is being careful in general. The number of applications that do even the _minimal_ safety-net of "create new file, rename it atomically over an old one" is basically zero. Almost everybody ends up rewriting files with something like open(name, O_CREAT | O_TRUNC, 0666); write(); close(); Basically, the reality on the ground right now is that: - Its almost impossible to reliably update a saved file right now without sometimes corrupting data. From the article, researchers found sqlite was the only correct database that existed based on their survey. - Most programmers don't even bother to use fsync. Anything which starts with "you can solve the trivial problem of saving a file in this super complex way" is a nonstarter and will almost never be used. Why should the kernel maintain an API for this? Because the default, obvious way people use the filesystem should be correct. A good API makes simple things simple and complex things possible. The posix API fails here. It makes the simple task of updating a save file almost impossible. I'd be happy for an argument that the posix filesystem is too low level to be used to directly store user data. But nobody seems to be making that argument, and thats not what reality looks like. So long as the filesystem API is advertised (and used) as the standard way to persist user data, it should do that in a simple, safe and correct way.
- unionpivo 6y agoA I think reason is simple. It works ok most of the time. And if you really care about some data, you will stick it some database anyway. And if you are going to stick data into database, you want to be able to select database that suits your needs best, and not be stuck with one provided by os. B I think you are underestimating how hard is it to implement good and performant transactional database. I am not sure it belongs into kernel.
- emmelaich 6y agoYes! C. A transaction based api would fail anyway occasionally. D. Note that some OSs (MVS/OS360) that are famous for transactions have the reputation that locking and so-forth are done in the kernel. Which is not actually true.
- josephg 6y agoLMDB implements a persistent transactional API around a B+ tree in just a few thousand lines of code. We put things in the kernel for several reasons - performance, but also consistency and interoperability. If every program embeds its own database and its own semantics, its much harder to have programs share data.