13 ms·

Distributed Filesystems for Deep Learning

91 points by jamesblonde 8y ago

habitue 8y ago
I am baffled why they have a matrix of what data science libraries they support. They support reading and writing CSVs and Parquet (and, I assume, arbitrary files as well). How is this specific HopsFS support for Pandas? Pandas is dumping csv format to a file object which happens to write to HopsFS. This is a pretty weak sales pitch. Edit: I'm not actually baffled. It's clearly SEO to catch people googling around for "pandas support" or the like.
- jamesblonde 8y ago
  Author of the post here. Part of the point of the matrix was to show that not all distributed filesystems have native support for Python frameworks commonly used for Data Science - Pandas/PySpark/TensorFlow. The other part is support for NVMe hardware. Having been a professor for 17 years, i barely know what SEO is.
  - dekhn 8y ago
    Can you explain what it means to have native support? Are you saying you hook into Python's file io to implement remote IO? Or is the filesystem mounted via FUSE or another technology? I Don't think that Pandas has any HopFS specific code in it.
    - jamesblonde 8y ago
      "In Pandas, the only change we need to make to our code, compared to a local filesystem, is to replace open_file(..) with h.open_file(..), where h is a file handle to HDFS/HopsFS." HopsFS is a drop-in replacement for HDFS. Here's some more details on native HDFS connectors in Python by Wes McKinney -http://wesmckinney.com/blog/python-hdfs-interfaces/ http://wesmckinney.com/blog/python-hdfs-interfaces/
      - dekhn 8y ago
        I think what you are saying is that you're wire-compatible with HDFS, and Pandas has access libraries (special file objects) that support HDFS.
      - daviesliu 8y ago
        In my opion, Cephfs and GlusterFS have much better support for Pandas and others, you don't need change any code between local development and train an distruted way, rather than a path. Second, the Python library for HDFS provides only subset of filesystem API, which limits what you can do with it.
  - rpedela 8y ago
    I get a 404 for the first link in your article. Thanks for reference to the Baidu study, it is very interesting.
- mtanski 8y ago
  Sadly there is no way to downvote stories. This post is factually incorrect at best... And, the author shows up and just shills harder.
  - rpedela 8y ago
    You can flag.
thstart 8y ago
I haven’t seen a practical use of DL or ML it is all over hyped. The biggest problem- nobody seems to understand what happens inside the Neural Network and why it works at all. All what NN are doing is tweaking weights till they match the desired outcome - all this very fast. But without deep understanding of how it works I doubt it would progress that way. Fact is after first evaluation at MIT it was decided it is feasible in 5 years to achieve desired outcome- now we are 50 years later at the same point with the only difference- fast hardware for matrix computations. The software is producing very fast a black box model which nobody understands.
regularfry 8y ago
Not by that graph, you don't. Pick a number in the middle: 2^24x10^6 words. That's 130TB, give or take. Sounds a lot, right? Not when you remember 16TB drives and 15 bay RAID enclosures exist...
huntaub 8y ago
What about Elastic File System?
- jamesblonde 8y ago
  It's the same as s3 - no support for NVMe.
  - dekhn 8y ago
    That's a non sequitur- EFS is a way to mount filesystems via NFS. What EFS does internally is kind of irrelevant- Amazon could implement it with NVMe, or whatever technology that they build internally, and the client would see the performance.
  - manigandham 8y ago
    What exactly are you talking about? None of these things are comparable. S3 is object storage. Ceph is a networked POSIX file system. HDFS is in between. All of them can run on NVMe devices.
deleted 8y ago
[deleted]
wskinner 8y ago
Can anyone working on this project comment on how HopsFS compares to similar alternatives? The post and the paper compare it to HDFS, but that's not really fair - HDFS was not designed to handle huge quantities of small files, and if you use it for that, you will of course not get good performance. Competitive offerings would be things like Qumulo, Amazon EFS, Objective FS, perhaps BeeGFS & Lustre, and so on.
- jamesblonde 8y ago
  HopsFS is a derivative work (a fork) of HDFS - it is still 100% wire-compatible with HDFS. It has distributed metadata (not metadata stored on the heap of a single JVM). It also supports putting small files on NVMe storage - https://www.logicalclocks.com/millions-and-millions-of-files-deep-learning-at-scale-with-hopsfs/ https://www.logicalclocks.com/millions-and-millions-of-files... It's also open-source. So, it works in places HDFS works. HopsFS is, to the best of our knowledge, the only distributed filesystem with distributed metadata (apart from Google Collosus - the hidden jewel in their crown). Lustre, BeeGFS are HPC filesystems, i believe. No data locality. Amazon EFS - feels like a MapR clone, but they expose some tunings - higher latency for higher throughput and larger clusters. We don't make that trade-off. I don't know enough about Qumulo and Objective FS to comment on them.
  - nathanwh 8y ago
    Does Ceph not have distrubuted metadata as well?
    - jamesblonde 8y ago
      You can have multiple metadata servers, but then you have multiple volumes - like MapR. When you move a file between two different volumes, the move operation is no longer a simple metadata operation (which is atomic). Instead the move operation involves removing the file from one metadata server, then adding it to the other server - an agreement problem. Except, they don't solve the agreement problem properly. You will notice it more when you move a large directory, as then it will go really slow - and you hope you dont have a failure while the operation is executing. reference: http://docs.ceph.com/docs/master/cephfs/add-remove-mds/ http://docs.ceph.com/docs/master/cephfs/add-remove-mds/
sgt101 8y ago
I'm wondering if Kudu + Impala + RAPIDS + YARN +TonY might do instead... maybe more support?
- jamesblonde 8y ago
  What's RAPIDS? How long will Impala last, post Hortonworks merger? With Hive at position 15, and Impala at position 35, who do you think the merged entity will bet on? https://db-engines.com/en/ranking https://db-engines.com/en/ranking
  - sgt101 8y ago
    https://news.ycombinator.com/item?id=18186392 https://news.ycombinator.com/item?id=18186392 for RAPIDS Impala was the wrong thing to posit in the list above - maybe SPARK SQL is better as you can run that under YARN as well.
jey 8y ago
What's the relevance of NVMe? Is that I/O latency still relevant in the presence of networking and scheduling overheads?
- jamesblonde 8y ago
  If your V-100 GPU can process 150 images/sec, can S3 keep up? What if you now have 10 GPUs or 100 GPUs? Now, HDFS can't keep up. But HopsFS can....
  - dekhn 8y ago
    Many people on these forums train directly from S3 and know how to get that kind of throughput. Like any blobject store, the key is to use multithreaded readers and high concurrency and set up your storage properly.
  - jey 8y ago
    Interesting. What's the bottleneck for HDFS, is it that the namenode has high latency under load?
xfactor973 8y ago
Gluster/Ceph support NVMe. That matrix isn't right
manigandham 8y ago
What does NVMe have do to with any of this? That's hardware storage technology, and it's being compared to Python versions of deep learning frameworks? The whole point of having a file system is that programs are abstracted from storage. What does native support even mean unless the file system somehow hooks into the running code? This makes no sense. It seems like a post trying to push HopsFS as more performant, but comes across as inaccurate and misleading.
- aogl 8y ago
  I agree, it appears as though NVMe was simply added in there to look impressive and give another tick when in actual fact it's not relevant to this layer of tech. This should be abstracted beyond hardware storage and be based solely on the filesystem itself.
xvilka 8y ago
Problem of both HDFS and HopFS - they written in Java, which makes them slower and eating much more memory than native filesystems like Ceph.
_zoltan_ 8y ago
we're using Spectrum Scale with its HDFS connector. while we're talking about options, I thought I'd mention it as well.
- gnufx 8y ago
  For what it's worth, you can do the same freely with at least OrangeFS, Gluster, and Lustre if you must do things a Hadoop-ish way; I assume also with Ceph.

91 points by jamesblonde 8y ago

I am baffled why they have a matrix of what data science libraries they support. They support reading and writing CSVs and Parquet (and, I assume, arbitrary files as well). How is this specific HopsFS support for Pandas? Pandas is dumping csv format to a file object which happens to write to HopsFS. This is a pretty weak sales pitch. Edit: I'm not actually baffled. It's clearly SEO to catch people googling around for "pandas support" or the like.

jamesblonde 8y ago
Author of the post here. Part of the point of the matrix was to show that not all distributed filesystems have native support for Python frameworks commonly used for Data Science - Pandas/PySpark/TensorFlow. The other part is support for NVMe hardware. Having been a professor for 17 years, i barely know what SEO is.
- dekhn 8y ago
  Can you explain what it means to have native support? Are you saying you hook into Python's file io to implement remote IO? Or is the filesystem mounted via FUSE or another technology? I Don't think that Pandas has any HopFS specific code in it.
  - jamesblonde 8y ago
    "In Pandas, the only change we need to make to our code, compared to a local filesystem, is to replace open_file(..) with h.open_file(..), where h is a file handle to HDFS/HopsFS." HopsFS is a drop-in replacement for HDFS. Here's some more details on native HDFS connectors in Python by Wes McKinney -http://wesmckinney.com/blog/python-hdfs-interfaces/ http://wesmckinney.com/blog/python-hdfs-interfaces/
    - dekhn 8y ago
      I think what you are saying is that you're wire-compatible with HDFS, and Pandas has access libraries (special file objects) that support HDFS.
    - daviesliu 8y ago
      In my opion, Cephfs and GlusterFS have much better support for Pandas and others, you don't need change any code between local development and train an distruted way, rather than a path. Second, the Python library for HDFS provides only subset of filesystem API, which limits what you can do with it.
- rpedela 8y ago
  I get a 404 for the first link in your article. Thanks for reference to the Baidu study, it is very interesting.
mtanski 8y ago
Sadly there is no way to downvote stories. This post is factually incorrect at best... And, the author shows up and just shills harder.
- rpedela 8y ago
  You can flag.

thstart 8y ago

I haven’t seen a practical use of DL or ML it is all over hyped. The biggest problem- nobody seems to understand what happens inside the Neural Network and why it works at all. All what NN are doing is tweaking weights till they match the desired outcome - all this very fast. But without deep understanding of how it works I doubt it would progress that way. Fact is after first evaluation at MIT it was decided it is feasible in 5 years to achieve desired outcome- now we are 50 years later at the same point with the only difference- fast hardware for matrix computations. The software is producing very fast a black box model which nobody understands.

regularfry 8y ago

Not by that graph, you don't. Pick a number in the middle: 2^24x10^6 words. That's 130TB, give or take. Sounds a lot, right? Not when you remember 16TB drives and 15 bay RAID enclosures exist...

huntaub 8y ago

What about Elastic File System?

jamesblonde 8y ago
It's the same as s3 - no support for NVMe.
- dekhn 8y ago
  That's a non sequitur- EFS is a way to mount filesystems via NFS. What EFS does internally is kind of irrelevant- Amazon could implement it with NVMe, or whatever technology that they build internally, and the client would see the performance.
- manigandham 8y ago
  What exactly are you talking about? None of these things are comparable. S3 is object storage. Ceph is a networked POSIX file system. HDFS is in between. All of them can run on NVMe devices.

deleted 8y ago

[deleted]

wskinner 8y ago

Can anyone working on this project comment on how HopsFS compares to similar alternatives? The post and the paper compare it to HDFS, but that's not really fair - HDFS was not designed to handle huge quantities of small files, and if you use it for that, you will of course not get good performance. Competitive offerings would be things like Qumulo, Amazon EFS, Objective FS, perhaps BeeGFS & Lustre, and so on.

jamesblonde 8y ago
HopsFS is a derivative work (a fork) of HDFS - it is still 100% wire-compatible with HDFS. It has distributed metadata (not metadata stored on the heap of a single JVM). It also supports putting small files on NVMe storage - https://www.logicalclocks.com/millions-and-millions-of-files-deep-learning-at-scale-with-hopsfs/ https://www.logicalclocks.com/millions-and-millions-of-files... It's also open-source. So, it works in places HDFS works. HopsFS is, to the best of our knowledge, the only distributed filesystem with distributed metadata (apart from Google Collosus - the hidden jewel in their crown). Lustre, BeeGFS are HPC filesystems, i believe. No data locality. Amazon EFS - feels like a MapR clone, but they expose some tunings - higher latency for higher throughput and larger clusters. We don't make that trade-off. I don't know enough about Qumulo and Objective FS to comment on them.
- nathanwh 8y ago
  Does Ceph not have distrubuted metadata as well?
  - jamesblonde 8y ago
    You can have multiple metadata servers, but then you have multiple volumes - like MapR. When you move a file between two different volumes, the move operation is no longer a simple metadata operation (which is atomic). Instead the move operation involves removing the file from one metadata server, then adding it to the other server - an agreement problem. Except, they don't solve the agreement problem properly. You will notice it more when you move a large directory, as then it will go really slow - and you hope you dont have a failure while the operation is executing. reference: http://docs.ceph.com/docs/master/cephfs/add-remove-mds/ http://docs.ceph.com/docs/master/cephfs/add-remove-mds/

sgt101 8y ago

I'm wondering if Kudu + Impala + RAPIDS + YARN +TonY might do instead... maybe more support?

jamesblonde 8y ago
What's RAPIDS? How long will Impala last, post Hortonworks merger? With Hive at position 15, and Impala at position 35, who do you think the merged entity will bet on? https://db-engines.com/en/ranking https://db-engines.com/en/ranking
- sgt101 8y ago
  https://news.ycombinator.com/item?id=18186392 https://news.ycombinator.com/item?id=18186392 for RAPIDS Impala was the wrong thing to posit in the list above - maybe SPARK SQL is better as you can run that under YARN as well.

jey 8y ago

What's the relevance of NVMe? Is that I/O latency still relevant in the presence of networking and scheduling overheads?

jamesblonde 8y ago
If your V-100 GPU can process 150 images/sec, can S3 keep up? What if you now have 10 GPUs or 100 GPUs? Now, HDFS can't keep up. But HopsFS can....
- dekhn 8y ago
  Many people on these forums train directly from S3 and know how to get that kind of throughput. Like any blobject store, the key is to use multithreaded readers and high concurrency and set up your storage properly.
- jey 8y ago
  Interesting. What's the bottleneck for HDFS, is it that the namenode has high latency under load?

xfactor973 8y ago

Gluster/Ceph support NVMe. That matrix isn't right

manigandham 8y ago

What does NVMe have do to with any of this? That's hardware storage technology, and it's being compared to Python versions of deep learning frameworks? The whole point of having a file system is that programs are abstracted from storage. What does native support even mean unless the file system somehow hooks into the running code? This makes no sense. It seems like a post trying to push HopsFS as more performant, but comes across as inaccurate and misleading.

aogl 8y ago
I agree, it appears as though NVMe was simply added in there to look impressive and give another tick when in actual fact it's not relevant to this layer of tech. This should be abstracted beyond hardware storage and be based solely on the filesystem itself.

xvilka 8y ago

Problem of both HDFS and HopFS - they written in Java, which makes them slower and eating much more memory than native filesystems like Ceph.

_zoltan_ 8y ago

we're using Spectrum Scale with its HDFS connector. while we're talking about options, I thought I'd mention it as well.

gnufx 8y ago
For what it's worth, you can do the same freely with at least OrangeFS, Gluster, and Lustre if you must do things a Hadoop-ish way; I assume also with Ceph.