5 ms·

I'm the author. Let me know if you have any questions.

rsync 6y ago
I searched your page, and then this HN discussion, for the string 'ssh' and got nothing ... What is the access protocol ? What tools am I using to access the POSIX presentation ?
fwip 6y ago
What tradeoffs did you make? In what situations does S3 have better characteristics than HopsFS?
fh973 6y ago
> ... but has 100X the performance of S3 for file move/rename operations Isn't rename in S3 effectively a copy-delete operation?
- booi 6y ago
  That’s my understanding too. Also rename / copy turned out not to be very useful at the end of the day. Nearly all my implementations just boil down to randomized characters as ids
  - yandie 6y ago
    Yup. Use a system like Redis/DynamoDB or even a traditional database to store the metadata and use random UUID for actual file storage. And tag the files for expiration/clean up. S3 is not a file system and people should stop treating it like one - only to get bitten by these assumptions around it being a FS.
de_Selby 6y ago
How does this differ from Objectivefs?
- jamesblonde 6y ago
  Funnily enough, i wasn't aware of ObjectiveFS - i guess it's because i can't find a research paper for it. HopsFS on S3 is similar to ADLS on Azure (built on Azure block storage). Internally, ADLS and HopsFS are different - HopsFS has a scaleout consistent metadata layer with a CDC API, while ADLS doesn't. But ADLS-v2 is also very good.
dividuum 6y ago
The linked post says "has the same cost as S3", yet on the linked pricing page there's only a "Contact Us" Enterprise plan besides a free one. Am I missing something?
- jamesblonde 6y ago
  There is no extra charge for HopsFS as part of the Hopsworks platform - so you only pay for what you read/write/store in S3. There is SaaS pricing public on hopsworks.ai.
  - runako 6y ago
    This was also not clear to me when looking at the pricing page. I assumed there was Free and Enterprise, and that to use this beyond 250 working days, I would have to go to Enterprise.
    - jamesblonde 6y ago
      No, like Dropbox if you do referrals you can get extra credit and stay on the free tier indefinitely. Or you can pay-as-you go on the non-enterprise tier.
deleted 6y ago
[deleted]
ajb 6y ago
You say it's "posix-like" - so what from posix had to be left out?
- jamesblonde 6y ago
  Random writes are not supported in the HDFS API.
hilbertseries 6y ago
Reading through your article, this solution is built on top of s3. So, moving and listing files is faster, presumably due to a new metadata system you've built for tracking files. The trade off here, is that writes must be strictly slower now than they were previously because you've added a network hop. All read and write data now flows through these workers. Which adds a point of failure, if you steam too much data through these workers, you could potentially OOM them. Reads are potentially faster, but that also depends on the hit rate of the block cache. Would be nice to see a more transparent post listing the pros and cons, rather than what reads as a technical advertisement.
- jamesblonde 6y ago
  I'm one of the co-authors. The numbers for writes are in the paper, so it is very unfair to call it an advertisement. And it is a global cache - if the block is cached somewhere, it will be used in reads.
  - xyzzy_plugh 6y ago
    The parent does make a good point about centralization of requests being a problem. S3 load balances under the hood, so different key prefixes within a bucket are usually serviced in isolation -- a DoS to one prefix will usually not affect other prefixes. It seems like you'd be limiting yourself for concurrent access -- if everything is flowing through the MySQL cluster -- not a bad thing! Just perhaps warrants a caveat note. I'd expect S3 to smoke HopFS on concurrency.
    - jamesblonde 6y ago
      Read the paper, it doesn't smoke HopsFS on concurrency. In the 2nd paper, we get 1.6m ops/sec on HopsFS HA over 3 availability zones (without S3 as a backing layer). Can you get that on S3 (maybe if you are Netflix, otherwise your starting quota is 1000s ops/sec)?
      - nostrebored 6y ago

I searched your page, and then this HN discussion, for the string 'ssh' and got nothing ... What is the access protocol ? What tools am I using to access the POSIX presentation ?

fwip 6y ago

What tradeoffs did you make? In what situations does S3 have better characteristics than HopsFS?

fh973 6y ago

> ... but has 100X the performance of S3 for file move/rename operations Isn't rename in S3 effectively a copy-delete operation?

booi 6y ago
That’s my understanding too. Also rename / copy turned out not to be very useful at the end of the day. Nearly all my implementations just boil down to randomized characters as ids
- yandie 6y ago
  Yup. Use a system like Redis/DynamoDB or even a traditional database to store the metadata and use random UUID for actual file storage. And tag the files for expiration/clean up. S3 is not a file system and people should stop treating it like one - only to get bitten by these assumptions around it being a FS.

de_Selby 6y ago

How does this differ from Objectivefs?

jamesblonde 6y ago
Funnily enough, i wasn't aware of ObjectiveFS - i guess it's because i can't find a research paper for it. HopsFS on S3 is similar to ADLS on Azure (built on Azure block storage). Internally, ADLS and HopsFS are different - HopsFS has a scaleout consistent metadata layer with a CDC API, while ADLS doesn't. But ADLS-v2 is also very good.

dividuum 6y ago

The linked post says "has the same cost as S3", yet on the linked pricing page there's only a "Contact Us" Enterprise plan besides a free one. Am I missing something?

jamesblonde 6y ago
There is no extra charge for HopsFS as part of the Hopsworks platform - so you only pay for what you read/write/store in S3. There is SaaS pricing public on hopsworks.ai.
- runako 6y ago
  This was also not clear to me when looking at the pricing page. I assumed there was Free and Enterprise, and that to use this beyond 250 working days, I would have to go to Enterprise.
  - jamesblonde 6y ago
    No, like Dropbox if you do referrals you can get extra credit and stay on the free tier indefinitely. Or you can pay-as-you go on the non-enterprise tier.

deleted 6y ago

[deleted]

ajb 6y ago

You say it's "posix-like" - so what from posix had to be left out?

jamesblonde 6y ago
Random writes are not supported in the HDFS API.

hilbertseries 6y ago

Reading through your article, this solution is built on top of s3. So, moving and listing files is faster, presumably due to a new metadata system you've built for tracking files. The trade off here, is that writes must be strictly slower now than they were previously because you've added a network hop. All read and write data now flows through these workers. Which adds a point of failure, if you steam too much data through these workers, you could potentially OOM them. Reads are potentially faster, but that also depends on the hit rate of the block cache. Would be nice to see a more transparent post listing the pros and cons, rather than what reads as a technical advertisement.

jamesblonde 6y ago
I'm one of the co-authors. The numbers for writes are in the paper, so it is very unfair to call it an advertisement. And it is a global cache - if the block is cached somewhere, it will be used in reads.
- xyzzy_plugh 6y ago
  The parent does make a good point about centralization of requests being a problem. S3 load balances under the hood, so different key prefixes within a bucket are usually serviced in isolation -- a DoS to one prefix will usually not affect other prefixes. It seems like you'd be limiting yourself for concurrent access -- if everything is flowing through the MySQL cluster -- not a bad thing! Just perhaps warrants a caveat note. I'd expect S3 to smoke HopFS on concurrency.
  - jamesblonde 6y ago
    Read the paper, it doesn't smoke HopsFS on concurrency. In the 2nd paper, we get 1.6m ops/sec on HopsFS HA over 3 availability zones (without S3 as a backing layer). Can you get that on S3 (maybe if you are Netflix, otherwise your starting quota is 1000s ops/sec)?
    - nostrebored 6y ago