3 ms·
Exaggerative title aside, it's really cool to learn about how fsspec lets you essentially make a custom filesystem in a few lines of code, as well as the (impor
by btown 3y ago
Exaggerative title aside, it's really cool to learn about how fsspec lets you essentially make a custom filesystem in a few lines of code, as well as the (important) notion that the very existence of a Python package can cause a plugin to be installed at a deep level, to be used automatically by other packages.
https://filesystem-spec.readthedocs.io/en/latest/developer.html#implementing-a-backend https://filesystem-spec.readthedocs.io/en/latest/developer.h...
https://setuptools.pypa.io/en/latest/userguide/entry_point.html#advertising-behavior https://setuptools.pypa.io/en/latest/userguide/entry_point.h...
(As a side note here, you might think that "pip install" can only add functionality, and that you can add extra packages locally for your convenience, or keep them around when you switch branches, without having your dev environment diverge from production behavior. But if they register plugins with other packages via the entry point functionality, you might end up coding things that depend on behavior that's not present in other environments! This is especially common in the pytest ecosystem. CI is vital here!)
- calpaterson 3y ago> Exaggerative title aside I apologise for the title. Not intended to mislead, just to be a bit of fun :) > t's really cool to learn about how fsspec [...] I'm glad you took something from it. At the last place I worked I spent a lot of time building a library to persist dataframes (intended audience: data analysts) and in retrospect I wish we'd thought of the approach of just having a `csvbase://` url scheme via fsspec. Currently it doesn't support Parquet which they would have needed though. Still some work to do for csvbase to accept Parquet uploads.