3 ms·
"In Pandas, the only change we need to make to our code, compared to a local filesystem, is to replace open_file(..) with h.open_file(..), where h is a file han
by jamesblonde 8y ago
"In Pandas, the only change we need to make to our code, compared to a local filesystem, is to replace open_file(..) with h.open_file(..), where h is a file handle to HDFS/HopsFS."
HopsFS is a drop-in replacement for HDFS. Here's some more details on native HDFS connectors in Python by Wes McKinney -http://wesmckinney.com/blog/python-hdfs-interfaces/ http://wesmckinney.com/blog/python-hdfs-interfaces/
- dekhn 8y agoI think what you are saying is that you're wire-compatible with HDFS, and Pandas has access libraries (special file objects) that support HDFS.
- daviesliu 8y agoIn my opion, Cephfs and GlusterFS have much better support for Pandas and others, you don't need change any code between local development and train an distruted way, rather than a path. Second, the Python library for HDFS provides only subset of filesystem API, which limits what you can do with it.