5 ms·
I generally agree: explicit is better than implicit, magic can be dangerous. But the author is not doing any monkeypatching or changing how pandas works. He is
by philomath_mn 3y ago
I generally agree: explicit is better than implicit, magic can be dangerous.
But the author is not doing any monkeypatching or changing how pandas works. He is using fsspec [0] to create a filesystem interface that pulls data from his site. fsspec appears to be somewhat standard since pandas, polars, Dask, and other libraries use it.
As soon as I understand that fsspec exists and this library uses it, there is no more magic. I would prefer if the specification of which fsspec to use were not embedded as part of a string, but overall this approach seems pretty reasonable.
[0] https://filesystem-spec.readthedocs.io/en/latest/ https://filesystem-spec.readthedocs.io/en/latest/
- kstrauser 3y agoIt’s still magic, just implemented with a shared crystal ball. If I learned about fsspec at 2AM during a prod outage via reading tracebacks instead of reading code, I’d be a font of creative profanity. The idea is nice! I’m not saying fsspec is bad. It’s not. It’s neat! I just strongly abhor the idea that pip installing a package changes runtime behavior whether that packages is ever even imported.
- philomath_mn 3y agoYeah, I'd prefer the pandas call look like ``` from csvbase import CSVBASE_FS pd.read_csv('//calpaterson/onion-vox-pops', fsspec=CSVBASE_FS) ```
- kstrauser 3y agoI vastly, hugely, enormously prefer that pattern. It’s not as pretty as the blog post, but it’s so explicit and easy to understand at a glance. If I’d never heard of csvbase or fsspec before seeing that code, I’d still know what to look for and how to debug it. At 2PM when I’m well rested, caffeinated, and alert, the original code is clever and lovely. At 2AM, I want this kind of easy to understand explicitness.
- philomath_mn 3y agoLooks like explicit fsspec usage is possible: ``` fs = fsspec.implementations.local.LocalFileSystem() with fs.open('test.csv', mode='r') as fp: print(pd.read_csv(StringIO(fp.read()))) ```
- rglullis 3y agoIf fsspec is an interface, then he is using an API and the whole blog post is just a pile of clickbaity BS.
- philomath_mn 3y agoYes, he is using an API. Even using curl over HTTP is an API, strictly speaking. But, for simple use cases, what he is doing beats building yet another client library or defining REST or RPC endpoints.
- rglullis 3y agoHe is defining an HTTP response based on the Accept header, no? The resource is represented by the URL, no? What else is needed for this to be called a "REST API"?