3 ms·

> According to the article, if I use ext4 with journaling and have a file with "a foo" and write "bar" at position 2, possible resulting states are: "a foo", "a

> According to the article, if I use ext4 with journaling and have a file with "a foo" and write "bar" at position 2, possible resulting states are: "a foo", "a boo", "a bao" or "a bar". It makes a lot more sense if you consider it shared, concurrently accessed memory. If you try to access it without atomics you can get funny results. And if you kill one process in the middle you might be stuck with reordered accesses indefinitely. Something like ext4 provides more than that. If you want ACID then you'll have to opt into that. The same as you have to do with concurrent programming (e.g. via std::atomic). But there are many cases where none of this is needed, where all you need is a bag to hold a few bytes for a while. E.g. cache files or large media blobs. The only issue is ubiquity of the necessary tooling, or the lack thereof. Many standard libraries now ship with concurrency-safe data structures while you often have to reach for 3rd-party libraries for safe file abstractions if you don't want to code your own. > If SQLite can implement transactions on top of the current filesystem APIs, then obviously its possible to put that behaviour in the kernel. If userspace can already implement it (when needed) based on provided kernel primitives, why should the kernel maintain two APIs side by side? Userspace needs many different levels of granularity. Sometimes integrity doesn't matter at all. Sometimes it only matters that writes make it to disk in correct order but it's ok for the last ones to get lost. And sometimes you need multiple processes memory-mapping a file and doing shared byte-level access and only random byte-ranges need to be committed to storage at any given time.

josephg 6y ago
> It makes a lot more sense if you consider it shared, concurrently accessed memory. But its not concurrently accessed memory - unlike memory you often want filesystem operations to run asynchronously. And even if you want to think about it like memory, we're missing all the tools we have for managing concurrently accessed memory like atomics and granular load and store barriers. And even then only about 1 in 10000 programmers know how to use those primitives to implement correct software anyway, so thats not exactly what I'd reach for when I think of good API design. But even if you want to use that analogy, the filesystem API doesn't have APIs for granular memory barriers. Until recently all we had was fsync. Peppering your code with fsync calls destroys your performance even though it doesn't change your program's behaviour. And if you batch up your changes and then fsync returns an error, you have no idea which writes succeeded and which failed. inotify helps, but oh goodness does it get complicated. > If userspace can already implement it (when needed) based on provided kernel primitives, why should the kernel maintain two APIs side by side? This is a good question. I guess I don't really think much about caches, or processes sharing a pseudofile to communicate. For those programs, sure - the current behaviour is fine. I'm thinking about the 95% of applications out there which want to use the filesystem to save user data to disk. Those programs shouldn't randomly end up in a corrupt state if power goes out at the wrong time. This is the core use case for the filesystem and I think its pretty reasonable for the kernel to make that case easy and correct by default, without resorting to storing all user data in every application in sqlite. From Linus talking about this issue ( https://lkml.org/lkml/2009/3/25/632 https://lkml.org/lkml/2009/3/25/632 ): > The problem is not that we have a lot of fsync() calls. Quite the reverse. fsync() is really really rare. So is being careful in general. The number of applications that do even the _minimal_ safety-net of "create new file, rename it atomically over an old one" is basically zero. Almost everybody ends up rewriting files with something like open(name, O_CREAT | O_TRUNC, 0666); write(); close(); Basically, the reality on the ground right now is that: - Its almost impossible to reliably update a saved file right now without sometimes corrupting data. From the article, researchers found sqlite was the only correct database that existed based on their survey. - Most programmers don't even bother to use fsync. Anything which starts with "you can solve the trivial problem of saving a file in this super complex way" is a nonstarter and will almost never be used. Why should the kernel maintain an API for this? Because the default, obvious way people use the filesystem should be correct. A good API makes simple things simple and complex things possible. The posix API fails here. It makes the simple task of updating a save file almost impossible. I'd be happy for an argument that the posix filesystem is too low level to be used to directly store user data. But nobody seems to be making that argument, and thats not what reality looks like. So long as the filesystem API is advertised (and used) as the standard way to persist user data, it should do that in a simple, safe and correct way.
- the8472 6y ago
  > But its not concurrently accessed memory - unlike memory you often want filesystem operations to run asynchronously. Well, but it is. Files can be opened multiple times and file descriptors can be shared between any number of processes and threads. mmap, atomics and msync also give you granular access. Most people might not use it that way, but the thing is that it can be used that way. Just like RAM can be accessed randomly from many threads but higher-level languages limit what you're allowed to do concurrently. It's the same fast-but-unsafe-guts, use safe abstractions thing. > I'm thinking about the 95% of applications out there which want to use the filesystem to save user data to disk Those 95% of the applications aren't necessarily those doing 95% of the write IO though. Looking at the activity on my system then most of the write traffic is a) logging (including messaging) b) firefox occasioally showing 1GB/s places.sqlite spikes (hah) c) temporary files d) docker e) p2p storage. For logging O_APPEND's atomicity promise may be good enough. I don't think text editors need transactions either, they only need file-level atomic replace which can be handled by a single library function. Such functions should be available in more libraries, I can agree with that. > Why should the kernel maintain an API for this? Because the default, obvious way people use the filesystem should be correct. Why the kernel and not libc? Or some other userspace lib for that matter. > I'd be happy for an argument that the posix filesystem is too low level to be used to directly store user data. That might be a matter of perspective. open/write/close is perfectly fine if all you do is move 100s of RAW photos from your SD card to your file server. There's no rewrite-in-place happening, it's a one-time thing that at worst would leave you with one incompletely copied file if the PC crashes shortly after copying. And the SD card wasn't a bastion of reliability anyway. Docker layers and containers are quite discardable and I wouldn't even mind if they had 0 durability. git, sqlite and postgres already seem to be doing the right thing. Whether document processing software does, I don't know but I assume they put some effort into it with all their autosaving features. But, as mentioned above, all they really need is nicer standard libs not a new kernel API. That doesn't leave that many things I deem critical. Perhaps it is a little annoying if some of the configuration text files of various GUI applications get lost in a crash but on the other hand I wouldn't want them to fsync my preferences every time I tick a checkbox so I'd only want atomicity but not synchronous durability. > - Most programmers don't even bother to use fsync. Anything which starts with "you can solve the trivial problem of saving a file in this super complex way" is a nonstarter and will almost never be used. If people can't even be bothered to fsync then why would they be bothered to wrap IO into a transaction? After all APIs still would have to continue to work in non-transactional mode for performance and backwards-compatibility reasons so there would be no pressure on people to use transactions. Maybe this also is a documentation problem since many developers don't seem to be aware of the non-durable-by-default behavior and different steps needed to get atomicity or durability. Man pages and standard libraries lack a high level overview of filesystem semantics. Some of it is noted in fsync() for example but not in rename()

josephg 6y ago

> It makes a lot more sense if you consider it shared, concurrently accessed memory. But its not concurrently accessed memory - unlike memory you often want filesystem operations to run asynchronously. And even if you want to think about it like memory, we're missing all the tools we have for managing concurrently accessed memory like atomics and granular load and store barriers. And even then only about 1 in 10000 programmers know how to use those primitives to implement correct software anyway, so thats not exactly what I'd reach for when I think of good API design. But even if you want to use that analogy, the filesystem API doesn't have APIs for granular memory barriers. Until recently all we had was fsync. Peppering your code with fsync calls destroys your performance even though it doesn't change your program's behaviour. And if you batch up your changes and then fsync returns an error, you have no idea which writes succeeded and which failed. inotify helps, but oh goodness does it get complicated. > If userspace can already implement it (when needed) based on provided kernel primitives, why should the kernel maintain two APIs side by side? This is a good question. I guess I don't really think much about caches, or processes sharing a pseudofile to communicate. For those programs, sure - the current behaviour is fine. I'm thinking about the 95% of applications out there which want to use the filesystem to save user data to disk. Those programs shouldn't randomly end up in a corrupt state if power goes out at the wrong time. This is the core use case for the filesystem and I think its pretty reasonable for the kernel to make that case easy and correct by default, without resorting to storing all user data in every application in sqlite. From Linus talking about this issue ( https://lkml.org/lkml/2009/3/25/632 https://lkml.org/lkml/2009/3/25/632 ): > The problem is not that we have a lot of fsync() calls. Quite the reverse. fsync() is really really rare. So is being careful in general. The number of applications that do even the _minimal_ safety-net of "create new file, rename it atomically over an old one" is basically zero. Almost everybody ends up rewriting files with something like open(name, O_CREAT | O_TRUNC, 0666); write(); close(); Basically, the reality on the ground right now is that: - Its almost impossible to reliably update a saved file right now without sometimes corrupting data. From the article, researchers found sqlite was the only correct database that existed based on their survey. - Most programmers don't even bother to use fsync. Anything which starts with "you can solve the trivial problem of saving a file in this super complex way" is a nonstarter and will almost never be used. Why should the kernel maintain an API for this? Because the default, obvious way people use the filesystem should be correct. A good API makes simple things simple and complex things possible. The posix API fails here. It makes the simple task of updating a save file almost impossible. I'd be happy for an argument that the posix filesystem is too low level to be used to directly store user data. But nobody seems to be making that argument, and thats not what reality looks like. So long as the filesystem API is advertised (and used) as the standard way to persist user data, it should do that in a simple, safe and correct way.

the8472 6y ago
> But its not concurrently accessed memory - unlike memory you often want filesystem operations to run asynchronously. Well, but it is. Files can be opened multiple times and file descriptors can be shared between any number of processes and threads. mmap, atomics and msync also give you granular access. Most people might not use it that way, but the thing is that it can be used that way. Just like RAM can be accessed randomly from many threads but higher-level languages limit what you're allowed to do concurrently. It's the same fast-but-unsafe-guts, use safe abstractions thing. > I'm thinking about the 95% of applications out there which want to use the filesystem to save user data to disk Those 95% of the applications aren't necessarily those doing 95% of the write IO though. Looking at the activity on my system then most of the write traffic is a) logging (including messaging) b) firefox occasioally showing 1GB/s places.sqlite spikes (hah) c) temporary files d) docker e) p2p storage. For logging O_APPEND's atomicity promise may be good enough. I don't think text editors need transactions either, they only need file-level atomic replace which can be handled by a single library function. Such functions should be available in more libraries, I can agree with that. > Why should the kernel maintain an API for this? Because the default, obvious way people use the filesystem should be correct. Why the kernel and not libc? Or some other userspace lib for that matter. > I'd be happy for an argument that the posix filesystem is too low level to be used to directly store user data. That might be a matter of perspective. open/write/close is perfectly fine if all you do is move 100s of RAW photos from your SD card to your file server. There's no rewrite-in-place happening, it's a one-time thing that at worst would leave you with one incompletely copied file if the PC crashes shortly after copying. And the SD card wasn't a bastion of reliability anyway. Docker layers and containers are quite discardable and I wouldn't even mind if they had 0 durability. git, sqlite and postgres already seem to be doing the right thing. Whether document processing software does, I don't know but I assume they put some effort into it with all their autosaving features. But, as mentioned above, all they really need is nicer standard libs not a new kernel API. That doesn't leave that many things I deem critical. Perhaps it is a little annoying if some of the configuration text files of various GUI applications get lost in a crash but on the other hand I wouldn't want them to fsync my preferences every time I tick a checkbox so I'd only want atomicity but not synchronous durability. > - Most programmers don't even bother to use fsync. Anything which starts with "you can solve the trivial problem of saving a file in this super complex way" is a nonstarter and will almost never be used. If people can't even be bothered to fsync then why would they be bothered to wrap IO into a transaction? After all APIs still would have to continue to work in non-transactional mode for performance and backwards-compatibility reasons so there would be no pressure on people to use transactions. Maybe this also is a documentation problem since many developers don't seem to be aware of the non-durable-by-default behavior and different steps needed to get atomicity or durability. Man pages and standard libraries lack a high level overview of filesystem semantics. Some of it is noted in fsync() for example but not in rename()