4 ms·
We think that there is a niche for a higher performance dist FS than S3. We have integrated NVMe hardware with our HDFS implementation (HopsFS) and made its met
by jamesblonde 8y ago
We think that there is a niche for a higher performance dist FS than S3. We have integrated NVMe hardware with our HDFS implementation (HopsFS) and made its metadata layer distributed. NVMe means you can, for example, work with datasets with millions of files for deep learning - instead of having to munge them into parquet files because your FS is slowing down your machine learning pipeline.
Reference:
https://www.logicalclocks.com/millions-and-millions-of-files-deep-learning-at-scale-with-hopsfs/ https://www.logicalclocks.com/millions-and-millions-of-files...
We have also redesigned the stack around our distributed metadata layer.
We are primarily targeting on-prem right now, but HopsFS would be the fastest DFS in the cloud if you ran it there today.
- Game_Ender 8y agoHow does HopsFS compare to Lustre?