5 ms·

In-Memory Filesystems in Rust

125 points by ingve 1y ago

IshKebab 1y ago
I guess the overhead from the syscalls simply wasn't that significant. Syscalls aren't that slow (~0.5us). This never says how many you were doing.
- ozgrakkurt 1y ago
  Also kernel code is probably more optimized than a mock library
  - jitl 1y ago
    wat. a write to address in memory + a context switch is not going to beat a write to an address in memory
adastra22 1y ago
I no mention of fsync/sync_all. That’s why your disk file system is acting as fast as your in memory file system (for small tests). Both are effectively in-memory.
- maxbond 1y ago
  A Rust-specific danger is that, if you don't explicitly sync a file before dropping it, any errors from syncing are ignored. So if you care about atomicity, call eg `File::sync_all()`.
  - dezgeg 1y ago
    Is that really rust-specific? I would be really surprised if any other languages do fsync() in their destructor either
    - maxbond 1y ago
      ETA: See correction below. To be clear `File::drop()` does sync, it just ignores errors (because `drop()` doesn't have a way of returning an error). It's not really Rust specific I guess, I just don't know off the top of my head what other languages behave this way.
      - aw1621107 1y ago
        I believe C++'s fstreams also ignore errors on destruction for similar reasons. I've wondered for a while what it'd take to eliminate such pitfalls in the "traditional" RAII approach. Something equivalent to deleting the "normal" RAII destructor and forcing consumption via a close() could be interesting, but I don't know how easy/hard that would be to pull off.
        vbezhenar 1y ago
        There's no reason that throwing exceptions in finalisers must be prohibited. It's just bad design. Modern Java provides the concept of suppressed exceptions. Basically an exception can maintain the list of suppressed exceptions. When stack unwinds, just allow finaliser to throw an exception. If it threw an exception, either propagate it to the handler, unwinding stack as necessary, or add it to the current exception as a suppressed exception. Exception handler can inspect the suppressed exceptions, if necessary. Exception print routine will print all suppressed exceptions for log visibility. Java does not do it properly for finally blocks, instead overwriting current exception, probably because the concept of suppressed exception was introduced in the later versions and they wanted to keep the compatibility. But it can be done properly.
        aw1621107 1y ago
        IIRC C++ and Rust don't technically prohibit throwing exceptions out of destructors; it's triggering unwinding during unwinding that's the main problem. Does make me wonder about the specifics behind that. I had assumed that there are some kind of soundness issues that force that particular approach (e.g., https://github.com/rust-lang/rust/pull/110975 https://github.com/rust-lang/rust/pull/110975, "Any panics while the panic hook is executing will force an immediate abort. This is necessary to avoid potential deadlocks like rustc hangs after ICEing due to memory limit #110771 where a panic happens while holding the backtrace lock."; alternatively, some other kind of soundness issue?), but I don't have the knowledge to say whether this is a fundamental limitation or "just" an implementation quirk that basically got standardized. Rust' first public release was after Java 7, so in principle the precedent was there, for what it's worth.
  - 01HNNWZ0MV43FF 1y ago
    For context - cppreference.com doesn't say anything about `fstream` syncing on drop, but it does have an explicit `sync` function. `QFile` from Qt doesn't even have a sync function, which I find odd.
    - aw1621107 1y ago
      I had always assumed that fstream flushes on destruction, but after digging through the standard all I can conclude is that I'm confused. According to the standard, fstream doesn't have an explicit destructor, but the standard says "It uses a basic_filebuf<charT, traits> object to control the associated sequences." ~basic_filebuf(), in turn, is defined to call close() (which I think flushes to disk?) and swallow exceptions. However, I can't seem to find anything that explicitly ties the lifetime of the fstream to the corresponding basic_filebuf. fstream doesn't have an explicitly declared destructor and the standard doesn't require that the basic_filebuf is a member of fstream, so the obvious ways the file would be closed don't seem to be explicitly required. In addition, all of fstream's parents' destructors are specified to perform no operations on the underlying rdbuf(). Which leaves... I don't know? cppreference says the underlying file is closed, though, which should flush it. And that's what I would expect for an RAII class! But I just can't seem to find the requirement...
  - silon42 1y ago
    I'd almost never want do to fsync in normal code (unless implementing something transactional)... but I'd want an explicit close almost always (or drop should panic/abort).
    - maxbond 1y ago
      I use transactional file operations when for instance when I write tools that change their own human-readable configuration files. If something is important enough to write to disk, then it's probably important enough that you can't tolerate torn writes.
    - adastra22 1y ago
      Why? The explicit close does almost nothing.
  - znpy 1y ago
    so the good old `sync; sync; sync;` ?
  - goodpoint 1y ago
    This is not correct. Programming languages do not and should not call sync automatically.
  - the8472 1y ago
    On most filesystems close(2) is nearly a noop, so even if you surfaced errors from close it returning successfully would not guarantee an absence of errors. close without fsync (or direct IO) essentially is telling the OS that you don't need immediate durability and prefer performance instead.
  - cyphar 1y ago
    This is actually true for all programs (on Linux at least) because closing a file does not mean it will be synced and so close(2) may not return an error even if the later sync will error out. The more general issue (not checking close(2) errors) is mostly true for most programming languages. I can count on one hand how many C programs I've seen that attempt to check the return value from close(2) consistently, let alone programs in languages like Go where handling it is far more effort than ignoring it. Also, close(2) doesn't consistently return errors. Because filesystem errors are often a property of the whole filesystem and data written during sync has been disassociated from the filesystem, the error usually can't be linked to a particular file descriptor. Most filesystems instead just return EIO if the filesystem had an error at all. This is arguably less useful than not returning an error at all because the error might be triggered by a completely unrelated process and (as above) you might not receive errors that you do care about. Filesystems also have different approaches to which close(2) calls will get filesystem errors. Some only return the error to the first close(2) call, which means another thread or process could clear the error bit. Other filesystems keep the error bit set until a remount, which means that any program checking close(2) will get lots of spurious errors. From memory there was a data corruption bug in PostgreSQL a few years ago because they were relying on close(2) error semantics that didn't work for all filesystems.
- indirect 1y ago
  I guess I wasn't sufficiently clear in the post, but the part I think is interesting is not that tmpfs and SSD bench at the same speed. I am aware of in-memory filesystem caches, and explicitly mention them twice in the last few paragraphs. The interesting part, to me, was that using the vfs crate or the rsfs crate didn't produce any differences from using tmpfs or an SSD. In theory, those crates completely cut out the actual filesystem and the OS entirely. Somehow, avoiding all those syscalls didn't make it any faster? Not what I expected. Anyway, if you have examples of in-process filesystem mocks that run faster than the in-memory filesystem cache, I'd love to hear about them.
eumon 1y ago
you may try /dev/shm for the testing purpose, which is effectively an in memory filesystem that linux provides, it is very performant
j1elo 1y ago
> It turns out the intended primary use case of the crate is to store files inside Rust binaries but still have an API sort of like the filesystem API to interact with them. Unfortunately, that information is hidden away in a comment on a random GitHub issue, rather than included in the project readme. A+ on technical prowess, F- on being able to articulate a couple words about it on a text file.
kolektiv 1y ago
It always surprised me somewhat that there isn't a set of traits covering some kind of `fs` like surface. It's not a trivial surface, but it's not huge either, and I've also found myself in a position of wanting to have multiple implementations of a filesystem-like structure (not even for the same reasons). Tricky to make that kind of change to std lib now I appreciate, but it seems like an odd gap.
- ozgrakkurt 1y ago
  Mocking file system or network seems counter productive. Complicated logic can be in pure functions and not be intertwined with IO if it needs to be tested. Mocking IO seems like it won’t really capture the problems you might encounter in reality anyway.
  - kolektiv 1y ago
    It's not always about mocking (in my cases it hasn't been). Sometimes it is about multiple "real" implementations - a filesystem is itself an abstraction, and a very common one, it seems like it would at least sometimes be useful to be able to leverage that more flexibly.
    - eru 1y ago
      I have a little system that takes a .git and mounts it as a fuse filesystem. Every commit becomes a directory with a snapshot of the project at that point in time. You could read the whole .git in at once, and then you'd have an in-memory file-system, if you wanted to. In any case, I agree with you: it's not about mocking.
    - kryptiskt 1y ago
      Some examples where it would be useful: Exposing a zip file or exe-embedded data as a filesystem, or making an FS backed by 9P, WebDAV or SFTP.
  - stingraycharles 1y ago
    Same like mocking databases; yes they make your tests run faster and more “pure”, but you’re suddenly not really testing reality anymore.
the8472 1y ago
> but all my benchmarks seem to disagree. Well, benchmarks could be wrong or misleading. Did you make sure that the IO actually happens and that it dominates the process execution time?
bhawks 1y ago
Posix file system semantics are very complex. An in memory implementation is likely to have quality gaps that make it sub optimal. If you want fast tests /tmp is likely to be backed by tmpfs in memory. If you are paranoid you can use /dev/shm to be explicit about your desire. Going this route means you're going to leverage all the well tested Linux VFS code and your tests will execute with higher fidelity.
mpweiher 1y ago
Yeah, SSDs are really fast. Have been for a while now, so fast that system call and other kernel overheads easily dominate unless you take care: https://blog.metaobject.com/2017/02/mkfile8-is-severely-syscall-limited-on.html https://blog.metaobject.com/2017/02/mkfile8-is-severely-sysc... That was 8 years ago, and even then mkfile needed a 512K buffer size to saturate the hardware. With the 512 byte default buffer it was 8x slower than the hardware. In addition, as others have pointed out, if you are not doing something extra to ensure things are flushed to disk, you are just measuring the buffer cache in the first place.
mncalc7 1y ago
Why not just use tmpfs?
jelder 1y ago
Linux supports two different in-memory filesystems, ramfs and tmpfs. It should be easy enough to set up tests to use paths on such a mount point. Wonder why OP didn't mention these.
tayo42 1y ago
>Eat trash, be free, test directly against the filesystem. Why not. You might run into weird environment or permission issues when you run your tests in a ci job. Omg, the comment about getting lectured. Relatable about anything testing related. It's happening this comment section. Someone hand waving testing away with pure functions, I'm sure someone will say pattern matching will solve all the problems thanks to the compiler.
cbondurant 1y ago
The thing that finally made me realize just how fast file IO is today, is when I was looking into doing my own impl of 1brc in rust for practice. I had expected that without the ramdisk, that file IO would have been the bottleneck for my testing, but in fact found that even with regular file IO my cpu was still the bottleneck, and multi-threading provided a massive performance boost. Completely decimating my expectations.
tannhaeuser 1y ago
In memoriam Filesystems in Rust, is what I read ;)
jitl 1y ago
[flagged]
mirekrusin 1y ago
i/o codepaths are heavily optimized with multiple layers of cache, you're probably hitting OS in memory one in your tests?

125 points by ingve 1y ago

I guess the overhead from the syscalls simply wasn't that significant. Syscalls aren't that slow (~0.5us). This never says how many you were doing.

ozgrakkurt 1y ago
Also kernel code is probably more optimized than a mock library
- jitl 1y ago
  wat. a write to address in memory + a context switch is not going to beat a write to an address in memory

adastra22 1y ago

I no mention of fsync/sync_all. That’s why your disk file system is acting as fast as your in memory file system (for small tests). Both are effectively in-memory.

maxbond 1y ago
A Rust-specific danger is that, if you don't explicitly sync a file before dropping it, any errors from syncing are ignored. So if you care about atomicity, call eg `File::sync_all()`.
- dezgeg 1y ago
  Is that really rust-specific? I would be really surprised if any other languages do fsync() in their destructor either
  - maxbond 1y ago
    ETA: See correction below. To be clear `File::drop()` does sync, it just ignores errors (because `drop()` doesn't have a way of returning an error). It's not really Rust specific I guess, I just don't know off the top of my head what other languages behave this way.
    - aw1621107 1y ago
      I believe C++'s fstreams also ignore errors on destruction for similar reasons. I've wondered for a while what it'd take to eliminate such pitfalls in the "traditional" RAII approach. Something equivalent to deleting the "normal" RAII destructor and forcing consumption via a close() could be interesting, but I don't know how easy/hard that would be to pull off.
      - vbezhenar 1y ago
        There's no reason that throwing exceptions in finalisers must be prohibited. It's just bad design. Modern Java provides the concept of suppressed exceptions. Basically an exception can maintain the list of suppressed exceptions. When stack unwinds, just allow finaliser to throw an exception. If it threw an exception, either propagate it to the handler, unwinding stack as necessary, or add it to the current exception as a suppressed exception. Exception handler can inspect the suppressed exceptions, if necessary. Exception print routine will print all suppressed exceptions for log visibility. Java does not do it properly for finally blocks, instead overwriting current exception, probably because the concept of suppressed exception was introduced in the later versions and they wanted to keep the compatibility. But it can be done properly.
        aw1621107 1y ago
        IIRC C++ and Rust don't technically prohibit throwing exceptions out of destructors; it's triggering unwinding during unwinding that's the main problem. Does make me wonder about the specifics behind that. I had assumed that there are some kind of soundness issues that force that particular approach (e.g., https://github.com/rust-lang/rust/pull/110975 https://github.com/rust-lang/rust/pull/110975, "Any panics while the panic hook is executing will force an immediate abort. This is necessary to avoid potential deadlocks like rustc hangs after ICEing due to memory limit #110771 where a panic happens while holding the backtrace lock."; alternatively, some other kind of soundness issue?), but I don't have the knowledge to say whether this is a fundamental limitation or "just" an implementation quirk that basically got standardized. Rust' first public release was after Java 7, so in principle the precedent was there, for what it's worth.
- 01HNNWZ0MV43FF 1y ago
  For context - cppreference.com doesn't say anything about `fstream` syncing on drop, but it does have an explicit `sync` function. `QFile` from Qt doesn't even have a sync function, which I find odd.
  - aw1621107 1y ago
    I had always assumed that fstream flushes on destruction, but after digging through the standard all I can conclude is that I'm confused. According to the standard, fstream doesn't have an explicit destructor, but the standard says "It uses a basic_filebuf<charT, traits> object to control the associated sequences." ~basic_filebuf(), in turn, is defined to call close() (which I think flushes to disk?) and swallow exceptions. However, I can't seem to find anything that explicitly ties the lifetime of the fstream to the corresponding basic_filebuf. fstream doesn't have an explicitly declared destructor and the standard doesn't require that the basic_filebuf is a member of fstream, so the obvious ways the file would be closed don't seem to be explicitly required. In addition, all of fstream's parents' destructors are specified to perform no operations on the underlying rdbuf(). Which leaves... I don't know? cppreference says the underlying file is closed, though, which should flush it. And that's what I would expect for an RAII class! But I just can't seem to find the requirement...
- silon42 1y ago
  I'd almost never want do to fsync in normal code (unless implementing something transactional)... but I'd want an explicit close almost always (or drop should panic/abort).
  - maxbond 1y ago
    I use transactional file operations when for instance when I write tools that change their own human-readable configuration files. If something is important enough to write to disk, then it's probably important enough that you can't tolerate torn writes.
  - adastra22 1y ago
    Why? The explicit close does almost nothing.
- znpy 1y ago
  so the good old `sync; sync; sync;` ?
- goodpoint 1y ago
  This is not correct. Programming languages do not and should not call sync automatically.
- the8472 1y ago
  On most filesystems close(2) is nearly a noop, so even if you surfaced errors from close it returning successfully would not guarantee an absence of errors. close without fsync (or direct IO) essentially is telling the OS that you don't need immediate durability and prefer performance instead.
- cyphar 1y ago
  This is actually true for all programs (on Linux at least) because closing a file does not mean it will be synced and so close(2) may not return an error even if the later sync will error out. The more general issue (not checking close(2) errors) is mostly true for most programming languages. I can count on one hand how many C programs I've seen that attempt to check the return value from close(2) consistently, let alone programs in languages like Go where handling it is far more effort than ignoring it. Also, close(2) doesn't consistently return errors. Because filesystem errors are often a property of the whole filesystem and data written during sync has been disassociated from the filesystem, the error usually can't be linked to a particular file descriptor. Most filesystems instead just return EIO if the filesystem had an error at all. This is arguably less useful than not returning an error at all because the error might be triggered by a completely unrelated process and (as above) you might not receive errors that you do care about. Filesystems also have different approaches to which close(2) calls will get filesystem errors. Some only return the error to the first close(2) call, which means another thread or process could clear the error bit. Other filesystems keep the error bit set until a remount, which means that any program checking close(2) will get lots of spurious errors. From memory there was a data corruption bug in PostgreSQL a few years ago because they were relying on close(2) error semantics that didn't work for all filesystems.
indirect 1y ago
I guess I wasn't sufficiently clear in the post, but the part I think is interesting is not that tmpfs and SSD bench at the same speed. I am aware of in-memory filesystem caches, and explicitly mention them twice in the last few paragraphs. The interesting part, to me, was that using the vfs crate or the rsfs crate didn't produce any differences from using tmpfs or an SSD. In theory, those crates completely cut out the actual filesystem and the OS entirely. Somehow, avoiding all those syscalls didn't make it any faster? Not what I expected. Anyway, if you have examples of in-process filesystem mocks that run faster than the in-memory filesystem cache, I'd love to hear about them.

eumon 1y ago

you may try /dev/shm for the testing purpose, which is effectively an in memory filesystem that linux provides, it is very performant

j1elo 1y ago

> It turns out the intended primary use case of the crate is to store files inside Rust binaries but still have an API sort of like the filesystem API to interact with them. Unfortunately, that information is hidden away in a comment on a random GitHub issue, rather than included in the project readme. A+ on technical prowess, F- on being able to articulate a couple words about it on a text file.

kolektiv 1y ago

It always surprised me somewhat that there isn't a set of traits covering some kind of `fs` like surface. It's not a trivial surface, but it's not huge either, and I've also found myself in a position of wanting to have multiple implementations of a filesystem-like structure (not even for the same reasons). Tricky to make that kind of change to std lib now I appreciate, but it seems like an odd gap.

ozgrakkurt 1y ago
Mocking file system or network seems counter productive. Complicated logic can be in pure functions and not be intertwined with IO if it needs to be tested. Mocking IO seems like it won’t really capture the problems you might encounter in reality anyway.
- kolektiv 1y ago
  It's not always about mocking (in my cases it hasn't been). Sometimes it is about multiple "real" implementations - a filesystem is itself an abstraction, and a very common one, it seems like it would at least sometimes be useful to be able to leverage that more flexibly.
  - eru 1y ago
    I have a little system that takes a .git and mounts it as a fuse filesystem. Every commit becomes a directory with a snapshot of the project at that point in time. You could read the whole .git in at once, and then you'd have an in-memory file-system, if you wanted to. In any case, I agree with you: it's not about mocking.
  - kryptiskt 1y ago
    Some examples where it would be useful: Exposing a zip file or exe-embedded data as a filesystem, or making an FS backed by 9P, WebDAV or SFTP.
- stingraycharles 1y ago
  Same like mocking databases; yes they make your tests run faster and more “pure”, but you’re suddenly not really testing reality anymore.

the8472 1y ago

> but all my benchmarks seem to disagree. Well, benchmarks could be wrong or misleading. Did you make sure that the IO actually happens and that it dominates the process execution time?

bhawks 1y ago

Posix file system semantics are very complex. An in memory implementation is likely to have quality gaps that make it sub optimal. If you want fast tests /tmp is likely to be backed by tmpfs in memory. If you are paranoid you can use /dev/shm to be explicit about your desire. Going this route means you're going to leverage all the well tested Linux VFS code and your tests will execute with higher fidelity.

mpweiher 1y ago

Yeah, SSDs are really fast. Have been for a while now, so fast that system call and other kernel overheads easily dominate unless you take care: https://blog.metaobject.com/2017/02/mkfile8-is-severely-syscall-limited-on.html https://blog.metaobject.com/2017/02/mkfile8-is-severely-sysc... That was 8 years ago, and even then mkfile needed a 512K buffer size to saturate the hardware. With the 512 byte default buffer it was 8x slower than the hardware. In addition, as others have pointed out, if you are not doing something extra to ensure things are flushed to disk, you are just measuring the buffer cache in the first place.

mncalc7 1y ago

Why not just use tmpfs?

jelder 1y ago

Linux supports two different in-memory filesystems, ramfs and tmpfs. It should be easy enough to set up tests to use paths on such a mount point. Wonder why OP didn't mention these.

tayo42 1y ago

>Eat trash, be free, test directly against the filesystem. Why not. You might run into weird environment or permission issues when you run your tests in a ci job. Omg, the comment about getting lectured. Relatable about anything testing related. It's happening this comment section. Someone hand waving testing away with pure functions, I'm sure someone will say pattern matching will solve all the problems thanks to the compiler.

cbondurant 1y ago

The thing that finally made me realize just how fast file IO is today, is when I was looking into doing my own impl of 1brc in rust for practice. I had expected that without the ramdisk, that file IO would have been the bottleneck for my testing, but in fact found that even with regular file IO my cpu was still the bottleneck, and multi-threading provided a massive performance boost. Completely decimating my expectations.

tannhaeuser 1y ago

In memoriam Filesystems in Rust, is what I read ;)

jitl 1y ago

[flagged]

mirekrusin 1y ago

i/o codepaths are heavily optimized with multiple layers of cache, you're probably hitting OS in memory one in your tests?