5 ms·

I am baffled why they have a matrix of what data science libraries they support. They support reading and writing CSVs and Parquet (and, I assume, arbitrary fil

I am baffled why they have a matrix of what data science libraries they support. They support reading and writing CSVs and Parquet (and, I assume, arbitrary files as well). How is this specific HopsFS support for Pandas? Pandas is dumping csv format to a file object which happens to write to HopsFS. This is a pretty weak sales pitch. Edit: I'm not actually baffled. It's clearly SEO to catch people googling around for "pandas support" or the like.

jamesblonde 8y ago
Author of the post here. Part of the point of the matrix was to show that not all distributed filesystems have native support for Python frameworks commonly used for Data Science - Pandas/PySpark/TensorFlow. The other part is support for NVMe hardware. Having been a professor for 17 years, i barely know what SEO is.
- dekhn 8y ago
  Can you explain what it means to have native support? Are you saying you hook into Python's file io to implement remote IO? Or is the filesystem mounted via FUSE or another technology? I Don't think that Pandas has any HopFS specific code in it.
  - jamesblonde 8y ago
    "In Pandas, the only change we need to make to our code, compared to a local filesystem, is to replace open_file(..) with h.open_file(..), where h is a file handle to HDFS/HopsFS." HopsFS is a drop-in replacement for HDFS. Here's some more details on native HDFS connectors in Python by Wes McKinney -http://wesmckinney.com/blog/python-hdfs-interfaces/ http://wesmckinney.com/blog/python-hdfs-interfaces/
    - dekhn 8y ago
      I think what you are saying is that you're wire-compatible with HDFS, and Pandas has access libraries (special file objects) that support HDFS.
    - daviesliu 8y ago
      In my opion, Cephfs and GlusterFS have much better support for Pandas and others, you don't need change any code between local development and train an distruted way, rather than a path. Second, the Python library for HDFS provides only subset of filesystem API, which limits what you can do with it.
- rpedela 8y ago
  I get a 404 for the first link in your article. Thanks for reference to the Baidu study, it is very interesting.
mtanski 8y ago
Sadly there is no way to downvote stories. This post is factually incorrect at best... And, the author shows up and just shills harder.
- rpedela 8y ago
  You can flag.

jamesblonde 8y ago

Author of the post here. Part of the point of the matrix was to show that not all distributed filesystems have native support for Python frameworks commonly used for Data Science - Pandas/PySpark/TensorFlow. The other part is support for NVMe hardware. Having been a professor for 17 years, i barely know what SEO is.

dekhn 8y ago
Can you explain what it means to have native support? Are you saying you hook into Python's file io to implement remote IO? Or is the filesystem mounted via FUSE or another technology? I Don't think that Pandas has any HopFS specific code in it.
- jamesblonde 8y ago
  "In Pandas, the only change we need to make to our code, compared to a local filesystem, is to replace open_file(..) with h.open_file(..), where h is a file handle to HDFS/HopsFS." HopsFS is a drop-in replacement for HDFS. Here's some more details on native HDFS connectors in Python by Wes McKinney -http://wesmckinney.com/blog/python-hdfs-interfaces/ http://wesmckinney.com/blog/python-hdfs-interfaces/
  - dekhn 8y ago
    I think what you are saying is that you're wire-compatible with HDFS, and Pandas has access libraries (special file objects) that support HDFS.
  - daviesliu 8y ago
    In my opion, Cephfs and GlusterFS have much better support for Pandas and others, you don't need change any code between local development and train an distruted way, rather than a path. Second, the Python library for HDFS provides only subset of filesystem API, which limits what you can do with it.
rpedela 8y ago
I get a 404 for the first link in your article. Thanks for reference to the Baidu study, it is very interesting.

mtanski 8y ago

Sadly there is no way to downvote stories. This post is factually incorrect at best... And, the author shows up and just shills harder.

rpedela 8y ago
You can flag.