6 ms·

Fsspec: Filesystem Interfaces for Python

129 points by gilad 5y ago

nivekkevin 5y ago
It's used in pandas and I love it!
dragonsh 5y ago
How does it compare to pyfilesystems [1]. In our startup using pyfilesystems as a generic interface for our webapp which makes it cloud agnostic and the app can use amazon, gcs, NFS, gzip, tar, azure or regular file system. PyFilesystem is a Python module that provides a common interface to any filesystem. It supports many filesystems [2]. [1] https://www.pyfilesystem.org/ https://www.pyfilesystem.org/ [2] https://www.pyfilesystem.org/page/index-of-filesystems/ https://www.pyfilesystem.org/page/index-of-filesystems/
- barbazoo 5y ago
  It address pyfilesystem in the introduction https://filesystem-spec.readthedocs.io/en/latest/intro.html#not-pyfilesystems https://filesystem-spec.readthedocs.io/en/latest/intro.html#...
  - fatbob 5y ago
    Not very convincingly IMO. There’s also the approach taken by [s3path](https://github.com/liormizr/s3path https://github.com/liormizr/s3path)
remram 5y ago
See also smart_open: https://github.com/RaRe-Technologies/smart_open https://github.com/RaRe-Technologies/smart_open which might be more user-friendly? Never used it myself but it was on HN before. Discussion on their bugtracker: https://github.com/RaRe-Technologies/smart_open/issues/579 https://github.com/RaRe-Technologies/smart_open/issues/579 Personally I wasn't too impressed with fsspec, though I am using it. I had to wrap files with Python's io wrappers to get acceptable performance, and found that different fsspec implementations still have significant differences that you can't ignore. They don't seem interested in supporting or documenting use cases outside of Pandas and Dask.
- acidbaseextract 5y ago
  I'd give the a thumbs up to smart_open. Happily used it to have our servers parse data dumps of up to several hundred megabytes on user-configurable remote storage without drama. Seems like it could handle larger files without issue, but we haven't had need to push it. I can't speak to fsspec, but their filesystem functionality (eg. `ls`, `cp`) isn't something that smart_open has and it does look cool.
- shrikant 5y ago
  I'm a big fan of smart_open -- it makes moving stuff around in Airflow DAGs so much simpler, esp. with its transparent compression and S3 multipart upload handling. (which you can tweak, but comes with sensible defaults!) Also, I'm no 10x developer, but I found smart_open's source code a pleasure to read through and grok -- it's nicely organised and very easy to reason about!
- mdurant 5y ago
  What use cases would you like to see? fsspec is also used by xarray, Intake, DVC (see other comments) and others, and supported by, e.g., pyarrow. Please share your performance concerns as issues.
MrPowers 5y ago
fsspec is used in dask (https://github.com/dask/dask https://github.com/dask/dask) if you want to see it in action. The author of fsspec also created fastparquet (https://github.com/dask/fastparquet https://github.com/dask/fastparquet), a native Python implementation of the Parquet file format. I'm really appreciative for Martin's extensive contributions to the PyData ecosystem.
- OnlyMortal 5y ago
  FsSpec is also the old name for a file specification on Macintosh (Classic OS). I came here because of the naming :-)
rahimiali 5y ago
it's super convenient, but watch out for bugs and performance issues. it's caching mechanism, how it uploads to S3, and how it seeks unseekable files conceal pretty big performance bottlenecks.
- mdurant 5y ago
  You may want to make issues out of this comment. fsspec tries to provide sensible defaults and lots of options for how you cache bytes, file listings and connection objects.
epylar 5y ago
If there is an in-memory filesystem library for this, it could be useful for unit testing things that want to write files.
- catlifeonmars 5y ago
  What’s wrong with /tmp?
  - mixmastamyk 5y ago
    StringIo is probably faster than temp files.
    - calpaterson 5y ago
      My own experience is that nameless temporary files (eg tempfile.TemporaryFile()) are _considerably_ faster than (c)StringIO and worth using whenever the "file" is bigger than a few bytes.
      - klmr 5y ago
        Odd, why would this be? Read and write operations on temporary files need to go through the kernel and thus lead to frequent context switches. The same shouldn’t be true for a string buffer except on reallocation.
        WGH_ 5y ago
        Maybe page cache (the thing tmpfs relies on) doesn't do full copy when reallocation, but rather, roughly speaking, appends extra pages to some sort of linked list? It would explain why it's faster.
      - mixmastamyk 5y ago
        If that’s the case, there’s a bug somewhere.
shcheklein 5y ago
Our team has been migrating DVC to fsspec. We've even started working on creating fsspec compatible wrappers for SSH, Alibaba cloud, etc. There were challenges indeed, and some storages will require more work (e.g. things like GDrive) but I would say maintainers are responsive and helpful.
rektide 5y ago
this is what the web's upcoming File System Access[1] really enables, but stealtily so. it's advertised as an implementation, as the capabilities to interact with the filesystem. but it's also an interface too. JS having a interface for filesystems is going to be extremely great. [1] https://wicg.github.io/file-system-access/ https://wicg.github.io/file-system-access/
chaz6 5y ago
One feature of a filesystem library I would like is to convert a filename into one that only includes permitted characters for a given filesystem type (for example, ":" and "?" are allowed in xfs but not exfat). For instance, mystring = fs.util.convert_name(u"What time is it?.txt", fs.fstype.exfat, "_") where "_" is the character to be substituted.
- TwoBit 5y ago
  How would you do that without potentially creating colliding names?
piqufoh 5y ago
Just a heads up - fsspec has an (optional) dependency on s3fs which has a requirement on aiobotocore, which in turn is currently locked to an ancient boto3 version. Versioning fsspec[s3fs] in projects with other boto3 dependencies is a nightmare.
- mdurant 5y ago
  Note that this is a pip issue specifically - updating boto3 might update botocore to be incompatible with aiobotocore. Users of conda are always fine. Or you can pin your version requirements and be fine however you install.
  - piqufoh 5y ago
    First: hi, and thanks for all the dask :waves: Conda - yes, because it has less strict versioning for setuptools based packages. pin-ing versions, no - because you end up with fsspec[s3fs] requiring a version incompatible with any other recent package requiring boto3.

129 points by gilad 5y ago

It's used in pandas and I love it!

How does it compare to pyfilesystems [1]. In our startup using pyfilesystems as a generic interface for our webapp which makes it cloud agnostic and the app can use amazon, gcs, NFS, gzip, tar, azure or regular file system. PyFilesystem is a Python module that provides a common interface to any filesystem. It supports many filesystems [2]. [1] https://www.pyfilesystem.org/ https://www.pyfilesystem.org/ [2] https://www.pyfilesystem.org/page/index-of-filesystems/ https://www.pyfilesystem.org/page/index-of-filesystems/

barbazoo 5y ago
It address pyfilesystem in the introduction https://filesystem-spec.readthedocs.io/en/latest/intro.html#not-pyfilesystems https://filesystem-spec.readthedocs.io/en/latest/intro.html#...
- fatbob 5y ago
  Not very convincingly IMO. There’s also the approach taken by [s3path](https://github.com/liormizr/s3path https://github.com/liormizr/s3path)

remram 5y ago

See also smart_open: https://github.com/RaRe-Technologies/smart_open https://github.com/RaRe-Technologies/smart_open which might be more user-friendly? Never used it myself but it was on HN before. Discussion on their bugtracker: https://github.com/RaRe-Technologies/smart_open/issues/579 https://github.com/RaRe-Technologies/smart_open/issues/579 Personally I wasn't too impressed with fsspec, though I am using it. I had to wrap files with Python's io wrappers to get acceptable performance, and found that different fsspec implementations still have significant differences that you can't ignore. They don't seem interested in supporting or documenting use cases outside of Pandas and Dask.

acidbaseextract 5y ago
I'd give the a thumbs up to smart_open. Happily used it to have our servers parse data dumps of up to several hundred megabytes on user-configurable remote storage without drama. Seems like it could handle larger files without issue, but we haven't had need to push it. I can't speak to fsspec, but their filesystem functionality (eg. `ls`, `cp`) isn't something that smart_open has and it does look cool.
shrikant 5y ago
I'm a big fan of smart_open -- it makes moving stuff around in Airflow DAGs so much simpler, esp. with its transparent compression and S3 multipart upload handling. (which you can tweak, but comes with sensible defaults!) Also, I'm no 10x developer, but I found smart_open's source code a pleasure to read through and grok -- it's nicely organised and very easy to reason about!
mdurant 5y ago
What use cases would you like to see? fsspec is also used by xarray, Intake, DVC (see other comments) and others, and supported by, e.g., pyarrow. Please share your performance concerns as issues.

MrPowers 5y ago

fsspec is used in dask (https://github.com/dask/dask https://github.com/dask/dask) if you want to see it in action. The author of fsspec also created fastparquet (https://github.com/dask/fastparquet https://github.com/dask/fastparquet), a native Python implementation of the Parquet file format. I'm really appreciative for Martin's extensive contributions to the PyData ecosystem.

OnlyMortal 5y ago
FsSpec is also the old name for a file specification on Macintosh (Classic OS). I came here because of the naming :-)

rahimiali 5y ago

it's super convenient, but watch out for bugs and performance issues. it's caching mechanism, how it uploads to S3, and how it seeks unseekable files conceal pretty big performance bottlenecks.

mdurant 5y ago
You may want to make issues out of this comment. fsspec tries to provide sensible defaults and lots of options for how you cache bytes, file listings and connection objects.

epylar 5y ago

If there is an in-memory filesystem library for this, it could be useful for unit testing things that want to write files.

catlifeonmars 5y ago
What’s wrong with /tmp?
- mixmastamyk 5y ago
  StringIo is probably faster than temp files.
  - calpaterson 5y ago
    My own experience is that nameless temporary files (eg tempfile.TemporaryFile()) are _considerably_ faster than (c)StringIO and worth using whenever the "file" is bigger than a few bytes.
    - klmr 5y ago
      Odd, why would this be? Read and write operations on temporary files need to go through the kernel and thus lead to frequent context switches. The same shouldn’t be true for a string buffer except on reallocation.
      - WGH_ 5y ago
        Maybe page cache (the thing tmpfs relies on) doesn't do full copy when reallocation, but rather, roughly speaking, appends extra pages to some sort of linked list? It would explain why it's faster.
    - mixmastamyk 5y ago
      If that’s the case, there’s a bug somewhere.

shcheklein 5y ago

Our team has been migrating DVC to fsspec. We've even started working on creating fsspec compatible wrappers for SSH, Alibaba cloud, etc. There were challenges indeed, and some storages will require more work (e.g. things like GDrive) but I would say maintainers are responsive and helpful.

rektide 5y ago

this is what the web's upcoming File System Access[1] really enables, but stealtily so. it's advertised as an implementation, as the capabilities to interact with the filesystem. but it's also an interface too. JS having a interface for filesystems is going to be extremely great. [1] https://wicg.github.io/file-system-access/ https://wicg.github.io/file-system-access/

chaz6 5y ago

One feature of a filesystem library I would like is to convert a filename into one that only includes permitted characters for a given filesystem type (for example, ":" and "?" are allowed in xfs but not exfat). For instance, mystring = fs.util.convert_name(u"What time is it?.txt", fs.fstype.exfat, "_") where "_" is the character to be substituted.

TwoBit 5y ago
How would you do that without potentially creating colliding names?

piqufoh 5y ago

Just a heads up - fsspec has an (optional) dependency on s3fs which has a requirement on aiobotocore, which in turn is currently locked to an ancient boto3 version. Versioning fsspec[s3fs] in projects with other boto3 dependencies is a nightmare.

mdurant 5y ago
Note that this is a pip issue specifically - updating boto3 might update botocore to be incompatible with aiobotocore. Users of conda are always fine. Or you can pin your version requirements and be fine however you install.
- piqufoh 5y ago
  First: hi, and thanks for all the dask :waves: Conda - yes, because it has less strict versioning for setuptools based packages. pin-ing versions, no - because you end up with fsspec[s3fs] requiring a version incompatible with any other recent package requiring boto3.