3 ms·

It always surprised me somewhat that there isn't a set of traits covering some kind of `fs` like surface. It's not a trivial surface, but it's not huge either,

It always surprised me somewhat that there isn't a set of traits covering some kind of `fs` like surface. It's not a trivial surface, but it's not huge either, and I've also found myself in a position of wanting to have multiple implementations of a filesystem-like structure (not even for the same reasons). Tricky to make that kind of change to std lib now I appreciate, but it seems like an odd gap.

ozgrakkurt 1y ago
Mocking file system or network seems counter productive. Complicated logic can be in pure functions and not be intertwined with IO if it needs to be tested. Mocking IO seems like it won’t really capture the problems you might encounter in reality anyway.
- kolektiv 1y ago
  It's not always about mocking (in my cases it hasn't been). Sometimes it is about multiple "real" implementations - a filesystem is itself an abstraction, and a very common one, it seems like it would at least sometimes be useful to be able to leverage that more flexibly.
  - eru 1y ago
    I have a little system that takes a .git and mounts it as a fuse filesystem. Every commit becomes a directory with a snapshot of the project at that point in time. You could read the whole .git in at once, and then you'd have an in-memory file-system, if you wanted to. In any case, I agree with you: it's not about mocking.
  - kryptiskt 1y ago
    Some examples where it would be useful: Exposing a zip file or exe-embedded data as a filesystem, or making an FS backed by 9P, WebDAV or SFTP.
- stingraycharles 1y ago
  Same like mocking databases; yes they make your tests run faster and more “pure”, but you’re suddenly not really testing reality anymore.
  - boomlinde 1y ago
    Sure, but they're not mutually exclusive approaches. Having tests that run in a couple of seconds thanks to low I/O and cheap scaffolding setup/teardown can be a valuable addition to slower, comprehensive system and integration test suites.
    - actionfromafar 1y ago
      And you can spot differences between your understanding of reality (test mockups) and how it behaves on the real filesystem. (In this case.)
- Arch-TK 1y ago
  Fault injection is much easier if you can mock IO. And you aren't really testing your software if you're not injecting faults.
  - ozgrakkurt 1y ago
    My point is, you can give gibberish data to a processing function if it just takes in-memory data instead of being parameterized by “IO”. If you parameterize everything by IO then you have to mock the IO
    - Arch-TK 1y ago
      It's not just about gibberish data. It's read failures, write failures, open failures, for a variety of reasons including permission errors, disk errors, space errors, etc. You basically want to test your code does something sensical with either every possible combination of errors, or with some random subset of combinations of errors. This can be automated and can verify the behaviour of your program in the edge cases. It's actually not usually that helpful to only test "we read corrupt garbage from the file" which is the thing you're describing here.
Joker_vD 1y ago
See David R. Hanson's "A Portable File Directory System" [0][1], for example: a 700 lines long implementation of early UNIX's filesystem API that piggy-backs on some sort of pre-existing (block-oriented) I/O primitives, which means you can do it entirely in-memory, with about another 300 lines of code or so. I suspect that with OSes becoming much more UNIX-like the demand for such abstraction layers shrank almost to nothing. [0] https://drh.github.io/documents/pds-spe.pdf https://drh.github.io/documents/pds-spe.pdf [1] https://drh.github.io/documents/pds.pdf https://drh.github.io/documents/pds.pdf
lenkite 1y ago
Go has a basic FS abstraction in the standard library: https://dev.to/rezmoss/gos-fs-package-modern-file-system-abstraction-19-5aad https://dev.to/rezmoss/gos-fs-package-modern-file-system-abs... But the stdlib one is a bit barebones. So people created: https://github.com/spf13/afero https://github.com/spf13/afero
- tayo42 1y ago
  This is one nice thing about go. I think was trying to test something in Rust and I was surprised by how many people were OK with using real file's for unit testing. It seems like a massive oversight for being able to use rust in a corporate environment.
  - jen20 1y ago
    > It seems like a massive oversight for being able to use rust in a corporate environment. Why does being in a corporate environment matter?
    - tayo42 1y ago
      Permissions are usually more strict then a hobby projects. So you can't just be writing and leaving things all over the file system without the possibility of failure. Or maybe your using drives over a network and randomly your tests will now fail becasue of things outside your control. Things like that. That's why when writing tests you always want them to actually do io like that.
      - bccdee 1y ago
        Seems like you could just put your test files in a tmpfs, if leaving files laying around is an issue.
        tayo42 1y ago
        You need to mount and unmount fikesystems to do that. Take a look at the library mentioned,afero, and you'll see how nice it handles working with the file system in tests. You can have everything in memory, and a whole new fs in each test
hansvm 1y ago
This is something I'm hopeful will fall out accidentally from Zig's new IO interface. If everyone doing IO has the implementation injected, mocks (fault injection, etc) become trivial, along with any other sort of feature you might want like checking S3 if you don't have a sufficiently recent local copy.
Arnavion 1y ago
Having traits in the stdlib would be nice, but there's also the type parameter pollution that results from mocking which I think is also a turn-off, as TFA says about rsfs. I have a Rust library to implement the UAPI config spec (a spec that describes which files and directories a service should look for config files in), and initally wanted to test it with filesystem mocks. After making some effort to implement the mock types and traits, plus wrappers around the `<F = StdFs>` types to hide the `<F>` parameter because I didn't want to expose it in the public API, I realized it was much easier to not bother and just create all the directory trees I needed for the tests.
vpanyam 1y ago
Yeah having traits for this in the stdlib would be nice. You might find Lunchbox [1] interesting. I needed an async virtual filesystem interface for a project a few years ago (and didn't find an existing library that fit my needs) so I built one: > Lunchbox provides a common interface that can be used to interact with any filesystem (e.g. a local FS, in-memory FS, zip filesystem, etc). This interface closely matches `tokio::fs::` ... It includes a few traits (`ReadableFileSystem`, `WritableFileSystem`) along with an implementation for local filesystems. I also used those traits to build libraries that enable things like read-only filesystems backed by zip files [2] and remote filesystems over a transport (e.g. TCP, UDS, etc) [3]. [1] https://crates.io/crates/lunchbox https://crates.io/crates/lunchbox [2] https://crates.io/crates/zipfs https://crates.io/crates/zipfs [3] https://github.com/VivekPanyam/carton/tree/main/source/anywhere https://github.com/VivekPanyam/carton/tree/main/source/anywh...

ozgrakkurt 1y ago

Mocking file system or network seems counter productive. Complicated logic can be in pure functions and not be intertwined with IO if it needs to be tested. Mocking IO seems like it won’t really capture the problems you might encounter in reality anyway.

kolektiv 1y ago
It's not always about mocking (in my cases it hasn't been). Sometimes it is about multiple "real" implementations - a filesystem is itself an abstraction, and a very common one, it seems like it would at least sometimes be useful to be able to leverage that more flexibly.
- eru 1y ago
  I have a little system that takes a .git and mounts it as a fuse filesystem. Every commit becomes a directory with a snapshot of the project at that point in time. You could read the whole .git in at once, and then you'd have an in-memory file-system, if you wanted to. In any case, I agree with you: it's not about mocking.
- kryptiskt 1y ago
  Some examples where it would be useful: Exposing a zip file or exe-embedded data as a filesystem, or making an FS backed by 9P, WebDAV or SFTP.
stingraycharles 1y ago
Same like mocking databases; yes they make your tests run faster and more “pure”, but you’re suddenly not really testing reality anymore.
- boomlinde 1y ago
  Sure, but they're not mutually exclusive approaches. Having tests that run in a couple of seconds thanks to low I/O and cheap scaffolding setup/teardown can be a valuable addition to slower, comprehensive system and integration test suites.
  - actionfromafar 1y ago
    And you can spot differences between your understanding of reality (test mockups) and how it behaves on the real filesystem. (In this case.)
Arch-TK 1y ago
Fault injection is much easier if you can mock IO. And you aren't really testing your software if you're not injecting faults.
- ozgrakkurt 1y ago
  My point is, you can give gibberish data to a processing function if it just takes in-memory data instead of being parameterized by “IO”. If you parameterize everything by IO then you have to mock the IO
  - Arch-TK 1y ago
    It's not just about gibberish data. It's read failures, write failures, open failures, for a variety of reasons including permission errors, disk errors, space errors, etc. You basically want to test your code does something sensical with either every possible combination of errors, or with some random subset of combinations of errors. This can be automated and can verify the behaviour of your program in the edge cases. It's actually not usually that helpful to only test "we read corrupt garbage from the file" which is the thing you're describing here.

Joker_vD 1y ago

See David R. Hanson's "A Portable File Directory System" [0][1], for example: a 700 lines long implementation of early UNIX's filesystem API that piggy-backs on some sort of pre-existing (block-oriented) I/O primitives, which means you can do it entirely in-memory, with about another 300 lines of code or so. I suspect that with OSes becoming much more UNIX-like the demand for such abstraction layers shrank almost to nothing. [0] https://drh.github.io/documents/pds-spe.pdf https://drh.github.io/documents/pds-spe.pdf [1] https://drh.github.io/documents/pds.pdf https://drh.github.io/documents/pds.pdf

lenkite 1y ago

Go has a basic FS abstraction in the standard library: https://dev.to/rezmoss/gos-fs-package-modern-file-system-abstraction-19-5aad https://dev.to/rezmoss/gos-fs-package-modern-file-system-abs... But the stdlib one is a bit barebones. So people created: https://github.com/spf13/afero https://github.com/spf13/afero

tayo42 1y ago
This is one nice thing about go. I think was trying to test something in Rust and I was surprised by how many people were OK with using real file's for unit testing. It seems like a massive oversight for being able to use rust in a corporate environment.
- jen20 1y ago
  > It seems like a massive oversight for being able to use rust in a corporate environment. Why does being in a corporate environment matter?
  - tayo42 1y ago
    Permissions are usually more strict then a hobby projects. So you can't just be writing and leaving things all over the file system without the possibility of failure. Or maybe your using drives over a network and randomly your tests will now fail becasue of things outside your control. Things like that. That's why when writing tests you always want them to actually do io like that.
    - bccdee 1y ago
      Seems like you could just put your test files in a tmpfs, if leaving files laying around is an issue.
      - tayo42 1y ago
        You need to mount and unmount fikesystems to do that. Take a look at the library mentioned,afero, and you'll see how nice it handles working with the file system in tests. You can have everything in memory, and a whole new fs in each test

hansvm 1y ago

This is something I'm hopeful will fall out accidentally from Zig's new IO interface. If everyone doing IO has the implementation injected, mocks (fault injection, etc) become trivial, along with any other sort of feature you might want like checking S3 if you don't have a sufficiently recent local copy.

Arnavion 1y ago

Having traits in the stdlib would be nice, but there's also the type parameter pollution that results from mocking which I think is also a turn-off, as TFA says about rsfs. I have a Rust library to implement the UAPI config spec (a spec that describes which files and directories a service should look for config files in), and initally wanted to test it with filesystem mocks. After making some effort to implement the mock types and traits, plus wrappers around the `<F = StdFs>` types to hide the `<F>` parameter because I didn't want to expose it in the public API, I realized it was much easier to not bother and just create all the directory trees I needed for the tests.

vpanyam 1y ago

Yeah having traits for this in the stdlib would be nice. You might find Lunchbox [1] interesting. I needed an async virtual filesystem interface for a project a few years ago (and didn't find an existing library that fit my needs) so I built one: > Lunchbox provides a common interface that can be used to interact with any filesystem (e.g. a local FS, in-memory FS, zip filesystem, etc). This interface closely matches `tokio::fs::` ... It includes a few traits (`ReadableFileSystem`, `WritableFileSystem`) along with an implementation for local filesystems. I also used those traits to build libraries that enable things like read-only filesystems backed by zip files [2] and remote filesystems over a transport (e.g. TCP, UDS, etc) [3]. [1] https://crates.io/crates/lunchbox https://crates.io/crates/lunchbox [2] https://crates.io/crates/zipfs https://crates.io/crates/zipfs [3] https://github.com/VivekPanyam/carton/tree/main/source/anywhere https://github.com/VivekPanyam/carton/tree/main/source/anywh...