4 ms·
This situation is so terrible, and its embarrassing that despite decades of work, our industry doesn't seem to be able to solve this problem in any modern opera
by josephg 6y ago
This situation is so terrible, and its embarrassing that despite decades of work, our industry doesn't seem to be able to solve this problem in any modern operating system. (Edit: Apparently except for windows vista!)
It seems obvious to me that we need a transactional filesystem API with methods like:
fbegin()
fwrite() / rename / etc
fcommit()
Most of the write calls wouldn't return errors directly. Then the fcommit() call would atomically flush the locally buffered changes to the filesystem. If an error happened, the API should guarantee that the final state either contains all changes or none of them. The commit call should either block the thread until changes are safely stored on disk, or use a completion callback. This API probably couldn't span multiple filesystems, but thats fine in practice. And bonus points if the API supports snapshot reads within the transaction.
This would dramatically simplify the code in every program that wants to safely interact with the filesystem. And it would remove the perennial, justified anxiety database engineers carry that they're doing something subtly wrong without even knowing. Because fsync() is so coarse, I suspect performance would improve with this API, too. (So long as the transactions are mostly small.)
But why don't we have this API already? My theory is that kernel developers don't write enough userspace software to understand how bad the status quo is. And most people writing software in userland don't understand the kernel well enough to feel empowered to change it. Adding an API like this might also need changes in the filesystems.
But wow, it would be a huge boon to humanity to have a filesystem API which isn't so braindead. How many more decades do we need to have our files corrupt by default when errors happen? I wince every time I'm playing a video game and it has that "saving - don't turn off your console!" animation. Or during OS updates. Data corruption happens because we implemented the wrong abstractions 50 years ago and somehow forgot that code is mutable.
- jfim 6y agoInterestingly enough, that feature already exists and is being removed from Windows. From the transactional NTFS Wikipedia page[0]: > Microsoft has deprecated TxF and stated that it may be removed in a future version of Windows [0] https://en.wikipedia.org/wiki/Transactional_NTFS https://en.wikipedia.org/wiki/Transactional_NTFS
- josephg 6y agoCool! Does anyone know why its being removed? The article says it supports atomic operations spanning multiple computers. Thats pretty tricky to implement correctly. My guess would be that its being removed because the feature is overbaked and it impacts too many parts of windows. But I'd love to know for sure if anyone has insider knowledge into the feature.
- jdjtktktmf 6y agoBasically nobody is using it: > While TxF is a powerful set of APIs, there has been extremely limited developer interest in this API platform since Windows Vista primarily due to its complexity and various nuances which developers need to consider as part of application development. > As a result, Microsoft is considering deprecating TxF APIs in a future version of Windows to focus development and maintenance efforts on other features and APIs which have more value to a larger majority of customers.
- adontz 6y agoActually, it is hard to remove it, since Windows Update is dependent on it. "TxF is presently deprecated, but the functionality is still available. We don't recommend disabling this feature on the C: volume." Microsoft supports APIs which were declared deprecated for decades. However, I totally agree in "Basically nobody is using it" part. It's because it was not properly implemented in .Net Framework. And now, when .Net Core is multiplatform, i.e. worst of all worlds, chances it will be implemented are very close to zero. ReFS does not support TxF. Windows has a lot of awesome APIs which are accessible only to C/C++ and are not advertised well.
- simonh 6y agoIsnt the point of the article that these file systems are trying to implement something like this, with journals and write logs, and such as implementation details, but these have failure modes and bugs that mean these mechanisms can still give unexpected results. Its like saying, well software just shouldnt have bugs or be vulnerable to side effects. Of course it is possible to mitigate these issues more than most file systems do, but at the cost of reduced space or performance efficiency. Thats what using a further abstraction like SQLite buys you, but its not for free. Thats why all the FS as database efforts in the early 2000s like Longhorn failed.
- josephg 6y agoThe point of the article is how if you want the filesystem API to write data safely, you need to jump through a lot of hoops. (Which, spoilers: most developers don't do. And even developers who try really hard often get it wrong because the APIs are so awful and poorly documented. Hence the ever present spectre of data corruption.) According to the article, if I use ext4 with journaling and have a file with "a foo" and write "bar" at position 2, possible resulting states are: "a foo", "a boo", "a bao" or "a bar". This is the best scenario - without journaling, I can also end up with uninitialized junk. The authors of the filesystem APIs consider this to be correct behaviour. But I can't think of any use case for this nonsense behaviour. Real data, in every file on your computer is structured. And structured files simply can't handle this kind of data corruption. Maybe this API has slightly higher throughput, but by the time you've peppered your code with fsync() calls to make it correct, you've killed performance. And working around that takes whole philosophies of threading and thousands of lines of code. I want that write call to either complete completely or fail completely. Actually, I want more than that - for a bunch of successive write calls touching multiple files to collectively succeed together or fail together. Which is not what the article is saying. > Its like saying, well software just shouldnt have bugs The kernel shouldn't have bugs in its filesystem. We don't tolerate bugs in the scheduler, or in OpenSSL or in the CPU. If Postgres randomly corrupted its database every month or two, we would fix the bug. Or call it broken and use something else. If SQLite can implement transactions on top of the current filesystem APIs, then obviously its possible to put that behaviour in the kernel. And in the kernel, that code would be easier to optimize, and would be useful everywhere. If it was a simple bug, we would fix it. The problem is that the bug is that we have the wrong abstraction. Why do we have the current filesystem API? The story I heard is it was written by some intern at Berkeley decades ago who thought it would be neat to have files and network streams use the same API. We know more now, and its past time to fix the mistake. > Thats why all the FS as database efforts in the early 2000s like Longhorn failed. I don't buy the cynicism. Just because microsoft overengineered longhorn (and successfully shipped TxF btw), doesn't mean the problem is unsolvable.
- the8472 6y ago> Because fsync() is so coarse You can do some work in advance with sync_file_range. fsync is still needed to write out the metadata, but it'll have to do less work and you may be able to get better pacing and backpressure that way. > Then the fcommit() call would atomically flush the locally buffered changes to the filesystem. If an error happened, the API should guarantee that the final state either contains all changes or none of them. Many existing filesystems simply cannot support such an API because they're not journaling or CoW. Another issue might be that multiple transactions could overlap and be of unbounded size, so now you would need MVCC, exclusive locking or the ability for the kernel to provide feedback to userspace that it will immediately rollback the transaction due to resource limits or conflicts. This suddenly looks more and more like a database than a posix filesystem. There are many use-cases that don't need such hard guarantees and want throughput, minimal overhead and latency instead. Lazy writes that the kernel can flush to disk whenever it feels like provide a lot of the performance that we take for granted when working with large amounts of data. > And it would remove the perennial, justified anxiety database engineers carry that they're doing something subtly wrong without even knowing. I think the postgres/fsync error saga raised awareness of that and there were some proposals how to provide a better error channel to userspace.
- BlueTemplar 6y ago> CoW Are you talking about copy-on-write or redirect-on-write, which seems to be the real copy on write ? https://en.wikipedia.org/wiki/Copy-on-write https://en.wikipedia.org/wiki/Copy-on-write
- the8472 6y agoFrom the same article: > Despite their names, copy-on-write usually refers to the first technique.
- BlueTemplar 6y agoYes, hence my question.
- martinhath 6y agoThis [0] mail from Linus might shed some light over this. [0]: https://lkml.org/lkml/2009/3/25/632 https://lkml.org/lkml/2009/3/25/632
- josephg 6y agoThanks. It looks like I have a deep philosophical disagreement with Linus: > Theory and practice sometimes clash. And when that happens, theory loses. Every single time. I think he's wrong. Compiler / PL theory works. Distributed systems theory (TLA+, etc) allow us to write correct software. I've used fuzzers to find and fix bugs in databases and concurrency algorithms, and the result is robust, working code. Theory can add complexity, but complexity added in one place can remove complexity from another. (Eg TCP is complex but reliable, in order delivery in the network layer makes the application layer much simpler and free of bugs.) Linus's argument is that users will keep writing open(); write(); close();. I think he's half right - thats the right level of complexity the API should have, with no tricky, buggy fsync barriers. But we've known how to write simple, correct APIs for modifying data like this for decades. The answer is transactions. Keeping the kernel as it is just keeps us in a dark age of buggy software.
- BlueTemplar 6y agoWait, isn't TCP(/IP) the quick and dirty practical implementation that beat the much more theoretical OSI model ? (And the same thing happened to GNU/Linux, so Linus would know something about it !)
- msla 6y ago> Wait, isn't TCP(/IP) the quick and dirty practical implementation that beat the much more theoretical OSI model? Yes, it is. It even has its own (four-layer) model but the IETF People aren't nearly as interested in layering as the OSI people were: Layering Considered Harmful https://tools.ietf.org/html/rfc3439#section-3 https://tools.ietf.org/html/rfc3439#section-3 Point being, this is why people who know history get a bit perturbed when people insist on shoving TCP/IP into the OSI Model, or, worse, insist that TCP/IP "implements" OSI somehow. OSI had its chance, it lost, and now we use POP and SMTP for email, not X.400.
- dTal 6y ago>I wince every time I'm playing a video game and it has that "saving - don't turn off your console!" animation. That seems harsh. Even a transactional FS would have this message. Saving takes time, and the best case scenario for shutting off the machine before it's complete is still data loss. What alternative interface do you propose?
- josephg 6y agoCurrently the worst case scenario is that power loss while overwriting the save file leaves the file corrupted and you have to start the game from scratch. How many games do the rename trick correctly when they save? I bet it’s less than half. The behaviour should be that if power goes out while saving, the game either updated the save file or it didn’t. Saving should be atomic. That’s the best we can do, and what the user should expect if power goes out before it’s finished saving the game.
- branko_d 6y agoI'm sad Transactional NTFS is being deprecated. But it has been that way for a few years now and there is no indication it will be actually removed any time soon. I'm quietly hopeful it will de-depreceted at some point. :)