6 ms·

Show HN: PyFilesystem 2.0 – A Python interface to filesystems of all kinds

139 points by billowycoat 10y ago

amyjess 10y ago
Oh, this looks really nice. Thank you for posting it! A while back, I had an idea for modelling a filesystem as a dict, so maybe this will inspire me to get off my ass and actually write it. Basically, the idea is that this: foo = rootfs['home']['amyjess']['stuff']['foo.txt'] rootfs['home']['amyjess']['stuff']['bar.txt'] = bar would be equivalent to this: with open('/home/amyjess/stuff/foo.txt', 'r') as foo_file: foo = foo_file.read() with open('/home/amyjess/stuff/bar.txt', 'w') as bar_file: bar_file.write(bar) I really want to implement that now.
- billowycoat 10y ago
  Nice idea! You can also do this with PyFilesystem... fs.gettext('/home/amyjess/stuff/foo.txt') fs.settext('/home/amyjess/stuff/foo.txt', bar) There are equivalent bytes methods.
- dom0 10y ago
  You might want to look at pathlib: with (Path.home() / '.config' / '1234.conf').open() as fd: print('Hello', file=fd) Granted, with / precedence you'll often need parens to group / before .
- existencebox 10y ago
  It's funny you mention that, I have literally that functionality in my "blob storage" python wrapper. Currently it's just like 50 lines of code I use for internal ad-hoc storage because as it turns out, reasoning about storage as a hierarchy is really nice. (I also let you index with a list, where the list is the hierarchical path; the key part being composibility, which I imagine you can do reasonably with both models). Interesting to know that it's a more common pattern. I will have to think a bit on where that may go. I had always shot down any "value" with "it's just a thin transformation on top of path.join". (yay self-deprecation or something?)
- m45t3r 10y ago
  There is already pathlib in Python's stdlib that do something similar: >>> p = PurePath('/etc') >>> p PurePosixPath('/etc') >>> p / 'init.d' / 'apache2' PurePosixPath('/etc/init.d/apache2') And yeah, this uses operator overload. However it does look quite nice.
- gumby 10y ago
  You might want to look at how Common Lisp abstracted the filesystem 20-odd years ago as an example of how to do it right and also how to do it wrong (e.g. lack of iterators in those days)
- straphka 10y ago
  I once made something like this to mount json files as a fuse filesystem. It's not maintained and one of my first projects, so the code is somewhat questionable, but it does work. It basically mounts a dict on a filesystem, the exact opposite of what you want :) https://github.com/yhekma/datamounter https://github.com/yhekma/datamounter
  - ekiara 10y ago
    My big question is why? you wanted to navigate or display a JSON dataset using filesystem-viewer tools? i.e. "Explore" the JSON dataset?
    - straphka 10y ago
      Well, mostly because I wanted to play with fuse, but it started out as an ansible thing (doesn't work with >2 though). The ansible setup module returns the system information in json, and if you mount it with the --realtime flag, when you open a "file" like ram for instance, ansible fetches the current value for you. That way you have your infrastructure mounted so to speak. Sort of a /proc filesystem for your linux infrastructure. I never got further than a working poc though. It works, but there are some bugs and there is no regard for security.
- tcdent 10y ago
  I was inspired to write a very basic working proof-of-concept based on your idea: https://gist.github.com/tcdent/742f62f7082bd4a0b958865a893cba91 https://gist.github.com/tcdent/742f62f7082bd4a0b958865a893cb...
- billowycoat 10y ago
  Come to think of it, I have already implemented something like this in Moya. This would insert "bar/baz.txt" from filesystem "foo" in to a template: ${.fs.foo.bar.baz.txt}
O5vYtytb 10y ago
pyfilesystem is great! My only suggestion would be to improve the search optimization for pypi. It's the ~10th result for https://pypi.python.org/pypi?%3Aaction=search&term=pyfilesystem&submit=search https://pypi.python.org/pypi?%3Aaction=search&term=pyfilesys... and a page down on https://pypi.python.org/pypi?%3Aaction=search&term=fs&submit=search https://pypi.python.org/pypi?%3Aaction=search&term=fs&submit... It's really not clear what it's called since the package name is 'fs' but the project is 'pyfilesystem'.
- billowycoat 10y ago
  > pyfilesystem is great! Cheers. > My only suggestion would be to improve the search optimization for pypi. Fair point. Not entirely sure how to fix that. I'll look in to it...
  - linsomniac 10y ago
    PKG-INFO, line 2: Name: fs Looks like it's also registered in PyPI as "fs", from the URL: https://pypi.python.org/pypi/fs/ https://pypi.python.org/pypi/fs/
    - adtac 10y ago
      Maybe have a proxy https://pypi.python.org/pypi/pyfilesystem https://pypi.python.org/pypi/pyfilesystem that has `fs` as a requirement? That way even if people install with `pip install pyfilesystem`, you're good.
      - billowycoat 10y ago
        Hmm. That's an idea. Feels like a hack though. Do you know if there is a precedence for that?
        adtac 10y ago
        We're planning on something similar to move a big open source library somewhere else (so that dependencies don't break).
j_s 10y ago
As long as you can recursively process folders and files without stopping early due to an UnauthorizedAccessException-equivalent (which .NET didn't straighten out 'til v4). http://stackoverflow.com/a/2663779 http://stackoverflow.com/a/2663779
teilo 10y ago
Would this work with any object implementing the PathLike interface introduced in Python 3.6 ?
- billowycoat 10y ago
  You generally won't need to. PyFilesystem hides the differences between paths from you (https://pyfilesystem2.readthedocs.io/en/latest/concepts.html#paths https://pyfilesystem2.readthedocs.io/en/latest/concepts.html...). An exception would be in the constructor for OSFS Objects, which take a system path. I'll look in to adding support there...

139 points by billowycoat 10y ago

Oh, this looks really nice. Thank you for posting it! A while back, I had an idea for modelling a filesystem as a dict, so maybe this will inspire me to get off my ass and actually write it. Basically, the idea is that this: foo = rootfs['home']['amyjess']['stuff']['foo.txt'] rootfs['home']['amyjess']['stuff']['bar.txt'] = bar would be equivalent to this: with open('/home/amyjess/stuff/foo.txt', 'r') as foo_file: foo = foo_file.read() with open('/home/amyjess/stuff/bar.txt', 'w') as bar_file: bar_file.write(bar) I really want to implement that now.

billowycoat 10y ago
Nice idea! You can also do this with PyFilesystem... fs.gettext('/home/amyjess/stuff/foo.txt') fs.settext('/home/amyjess/stuff/foo.txt', bar) There are equivalent bytes methods.
dom0 10y ago
You might want to look at pathlib: with (Path.home() / '.config' / '1234.conf').open() as fd: print('Hello', file=fd) Granted, with / precedence you'll often need parens to group / before .
existencebox 10y ago
It's funny you mention that, I have literally that functionality in my "blob storage" python wrapper. Currently it's just like 50 lines of code I use for internal ad-hoc storage because as it turns out, reasoning about storage as a hierarchy is really nice. (I also let you index with a list, where the list is the hierarchical path; the key part being composibility, which I imagine you can do reasonably with both models). Interesting to know that it's a more common pattern. I will have to think a bit on where that may go. I had always shot down any "value" with "it's just a thin transformation on top of path.join". (yay self-deprecation or something?)
m45t3r 10y ago
There is already pathlib in Python's stdlib that do something similar: >>> p = PurePath('/etc') >>> p PurePosixPath('/etc') >>> p / 'init.d' / 'apache2' PurePosixPath('/etc/init.d/apache2') And yeah, this uses operator overload. However it does look quite nice.
gumby 10y ago
You might want to look at how Common Lisp abstracted the filesystem 20-odd years ago as an example of how to do it right and also how to do it wrong (e.g. lack of iterators in those days)
straphka 10y ago
I once made something like this to mount json files as a fuse filesystem. It's not maintained and one of my first projects, so the code is somewhat questionable, but it does work. It basically mounts a dict on a filesystem, the exact opposite of what you want :) https://github.com/yhekma/datamounter https://github.com/yhekma/datamounter
- ekiara 10y ago
  My big question is why? you wanted to navigate or display a JSON dataset using filesystem-viewer tools? i.e. "Explore" the JSON dataset?
  - straphka 10y ago
    Well, mostly because I wanted to play with fuse, but it started out as an ansible thing (doesn't work with >2 though). The ansible setup module returns the system information in json, and if you mount it with the --realtime flag, when you open a "file" like ram for instance, ansible fetches the current value for you. That way you have your infrastructure mounted so to speak. Sort of a /proc filesystem for your linux infrastructure. I never got further than a working poc though. It works, but there are some bugs and there is no regard for security.
tcdent 10y ago
I was inspired to write a very basic working proof-of-concept based on your idea: https://gist.github.com/tcdent/742f62f7082bd4a0b958865a893cba91 https://gist.github.com/tcdent/742f62f7082bd4a0b958865a893cb...
billowycoat 10y ago
Come to think of it, I have already implemented something like this in Moya. This would insert "bar/baz.txt" from filesystem "foo" in to a template: ${.fs.foo.bar.baz.txt}

O5vYtytb 10y ago

pyfilesystem is great! My only suggestion would be to improve the search optimization for pypi. It's the ~10th result for https://pypi.python.org/pypi?%3Aaction=search&term=pyfilesystem&submit=search https://pypi.python.org/pypi?%3Aaction=search&term=pyfilesys... and a page down on https://pypi.python.org/pypi?%3Aaction=search&term=fs&submit=search https://pypi.python.org/pypi?%3Aaction=search&term=fs&submit... It's really not clear what it's called since the package name is 'fs' but the project is 'pyfilesystem'.

billowycoat 10y ago
> pyfilesystem is great! Cheers. > My only suggestion would be to improve the search optimization for pypi. Fair point. Not entirely sure how to fix that. I'll look in to it...
- linsomniac 10y ago
  PKG-INFO, line 2: Name: fs Looks like it's also registered in PyPI as "fs", from the URL: https://pypi.python.org/pypi/fs/ https://pypi.python.org/pypi/fs/
  - adtac 10y ago
    Maybe have a proxy https://pypi.python.org/pypi/pyfilesystem https://pypi.python.org/pypi/pyfilesystem that has `fs` as a requirement? That way even if people install with `pip install pyfilesystem`, you're good.
    - billowycoat 10y ago
      Hmm. That's an idea. Feels like a hack though. Do you know if there is a precedence for that?
      - adtac 10y ago
        We're planning on something similar to move a big open source library somewhere else (so that dependencies don't break).

j_s 10y ago

As long as you can recursively process folders and files without stopping early due to an UnauthorizedAccessException-equivalent (which .NET didn't straighten out 'til v4). http://stackoverflow.com/a/2663779 http://stackoverflow.com/a/2663779

teilo 10y ago

Would this work with any object implementing the PathLike interface introduced in Python 3.6 ?

billowycoat 10y ago
You generally won't need to. PyFilesystem hides the differences between paths from you (https://pyfilesystem2.readthedocs.io/en/latest/concepts.html#paths https://pyfilesystem2.readthedocs.io/en/latest/concepts.html...). An exception would be in the constructor for OSFS Objects, which take a system path. I'll look in to adding support there...