13 ms·
See also smart_open: https://github.com/RaRe-Technologies/smart_open https://github.com/RaRe-Technologies/smart_open which might be more user-friendly? Never us
by remram 5y ago
See also smart_open: https://github.com/RaRe-Technologies/smart_open https://github.com/RaRe-Technologies/smart_open which might be more user-friendly? Never used it myself but it was on HN before. Discussion on their bugtracker: https://github.com/RaRe-Technologies/smart_open/issues/579 https://github.com/RaRe-Technologies/smart_open/issues/579
Personally I wasn't too impressed with fsspec, though I am using it. I had to wrap files with Python's io wrappers to get acceptable performance, and found that different fsspec implementations still have significant differences that you can't ignore. They don't seem interested in supporting or documenting use cases outside of Pandas and Dask.
- acidbaseextract 5y agoI'd give the a thumbs up to smart_open. Happily used it to have our servers parse data dumps of up to several hundred megabytes on user-configurable remote storage without drama. Seems like it could handle larger files without issue, but we haven't had need to push it. I can't speak to fsspec, but their filesystem functionality (eg. `ls`, `cp`) isn't something that smart_open has and it does look cool.
- shrikant 5y agoI'm a big fan of smart_open -- it makes moving stuff around in Airflow DAGs so much simpler, esp. with its transparent compression and S3 multipart upload handling. (which you can tweak, but comes with sensible defaults!) Also, I'm no 10x developer, but I found smart_open's source code a pleasure to read through and grok -- it's nicely organised and very easy to reason about!
- mdurant 5y agoWhat use cases would you like to see? fsspec is also used by xarray, Intake, DVC (see other comments) and others, and supported by, e.g., pyarrow. Please share your performance concerns as issues.