5 ms·

Only great things to say about FoundationDB. We've been using it for about a year now. Got a tiny, live cluster of 35+ commodity machines (started with 3 a year

Only great things to say about FoundationDB. We've been using it for about a year now. Got a tiny, live cluster of 35+ commodity machines (started with 3 a year ago), about 5TB capacity and growing. Been removing and adding servers (on live cluster) without a glitch. We've got another 100TB cluster in testing now. Of all the things, we're actually using it as a distributed file system. We've tried Ceph, GlusterFS, HDFS, MinIO and some others, and eventually decided on a custom FDB solution. It's a breeze to setup, and seems to eclipse others in performance [0] and reliability - Kyle (aphyr) the author of Jepsen series on distributed systems correctness, said: "haven't tested foundation in part because their testing appears to be waaaay more rigorous than mine." [1] The way we use FDB, if anyone is interested, is we simply split files into small chunks (per FDB data design recommendations), and store all file's & folder's meta data in FDB such as byte count, create/access/write times, permissions, and a lot more. Folders are handled by the builtin Directory layer [2]. [0] https://apple.github.io/foundationdb/performance.html https://apple.github.io/foundationdb/performance.html [1] https://web.archive.org/web/20150312112552/http://blog.foundationdb.com/call-me-maybe-foundationdb-vs-jepsen https://web.archive.org/web/20150312112552/http://blog.found... [2] https://forums.foundationdb.org/t/whats-the-purpose-of-the-directory-layer/677 https://forums.foundationdb.org/t/whats-the-purpose-of-the-d...

tpetry 5y ago
That's a really interesting solution. Can you tell us more about it? Operating distributed blob storage systems is kind of fragile with every software i have yet tried.
- davgoldin 5y ago
  FDB's Directory layer provides all you need to create and edit nested paths. What's left to develop is a file chunking and assembly part, and statistics if needed. The only reason you need chunking is because FDB has very clearly defined limits in their documentation, and one of those limits is value size - it can't exceed 100kB, and should be kept below 10kB for best performance. For statistics like folder byte count, you could use FDB's atomic increment operation. Here's a good starting point (not mine): https://forums.foundationdb.org/t/object-store-on-foundationdb/387/3 https://forums.foundationdb.org/t/object-store-on-foundation... I'll be happy to answer specific question if you got any.
  - tpetry 5y ago
    What kind of read/write ratio are you using? And would your solution work for a write-heavy workload? Kafka has limits on the message size and i need a solution for storing large blobs (up to 10MB) at data ingestion on for a very short time until the job has been processed. So read/write ratio will be exactly 50% and there will be a high write load. Is FoundationDB capable for this specific task? Are there some knobs tuneable for acid? Perfect acid requirements would not really be needed, if everything is fsynced every second would be completely ok.
    - davgoldin 5y ago
      From own experience, I'd say FDB can handle it all, we've got 20% read 80 write during peak hours, and reverse: 80 read 20 write other time. Without doing your own tests, here's per core numbers, from which you can extrapolate (e.g. via CPU mips) towards your own hardware: https://apple.github.io/foundationdb/performance.html#throughput-per-core https://apple.github.io/foundationdb/performance.html#throug... FDB is ACID as is shipped, you don't need to turn knobs to make it such. Toughest part is to figure out classes/roles of the system. Here are a couple of good starting points: https://nikita.melkozerov.dev/posts/2019/06/building-a-foundationdb-cluster-roles-classes-and-processes/ https://nikita.melkozerov.dev/posts/2019/06/building-a-found... https://forums.foundationdb.org/t/roles-classes-matrix/1340/2 https://forums.foundationdb.org/t/roles-classes-matrix/1340/...
    - cre-ker 5y ago
      Kafka limits are customizable. 10MB is quite small. Kafka easily handles 100MB messages. If you need some temporal storage that works like a log then there's nothing better than kafka in terms of speed and scalability. But it's more about relatively long term storage. If you need to persist data for very short amount of time, why don't you use in-memory store? Do you have strict requirements around data loss? Something like Redis much just do the trick for out. With Redis cluster it even scales.
jamesblonde 5y ago
Interesting! We have been doing the same thing with HopsFS for a couple of years. Except, we only store the small files in our database (www.rondb.com) - RonDB is a recent fork of MySQL Cluster (NDBCluster). Very small files (<1KB) are stored in memory in RonDB, small files (typically <128KB) are stored in NVMe disks in RonDB, and other files in HopsFS (which now stores its blocks in object storage (S3, ABS). We had a paper on it as ACM Middleware and it's open-source on github. Are you going to publish your solution? (Discussed here on HN: https://news.ycombinator.com/item?id=25149154 https://news.ycombinator.com/item?id=25149154 )
- jamesblonde 5y ago
  Do you support atomic rename? Atomic rename on subtrees? Consistent directory listings?

tpetry 5y ago

That's a really interesting solution. Can you tell us more about it? Operating distributed blob storage systems is kind of fragile with every software i have yet tried.

davgoldin 5y ago
FDB's Directory layer provides all you need to create and edit nested paths. What's left to develop is a file chunking and assembly part, and statistics if needed. The only reason you need chunking is because FDB has very clearly defined limits in their documentation, and one of those limits is value size - it can't exceed 100kB, and should be kept below 10kB for best performance. For statistics like folder byte count, you could use FDB's atomic increment operation. Here's a good starting point (not mine): https://forums.foundationdb.org/t/object-store-on-foundationdb/387/3 https://forums.foundationdb.org/t/object-store-on-foundation... I'll be happy to answer specific question if you got any.
- tpetry 5y ago
  What kind of read/write ratio are you using? And would your solution work for a write-heavy workload? Kafka has limits on the message size and i need a solution for storing large blobs (up to 10MB) at data ingestion on for a very short time until the job has been processed. So read/write ratio will be exactly 50% and there will be a high write load. Is FoundationDB capable for this specific task? Are there some knobs tuneable for acid? Perfect acid requirements would not really be needed, if everything is fsynced every second would be completely ok.
  - davgoldin 5y ago
    From own experience, I'd say FDB can handle it all, we've got 20% read 80 write during peak hours, and reverse: 80 read 20 write other time. Without doing your own tests, here's per core numbers, from which you can extrapolate (e.g. via CPU mips) towards your own hardware: https://apple.github.io/foundationdb/performance.html#throughput-per-core https://apple.github.io/foundationdb/performance.html#throug... FDB is ACID as is shipped, you don't need to turn knobs to make it such. Toughest part is to figure out classes/roles of the system. Here are a couple of good starting points: https://nikita.melkozerov.dev/posts/2019/06/building-a-foundationdb-cluster-roles-classes-and-processes/ https://nikita.melkozerov.dev/posts/2019/06/building-a-found... https://forums.foundationdb.org/t/roles-classes-matrix/1340/2 https://forums.foundationdb.org/t/roles-classes-matrix/1340/...
  - cre-ker 5y ago
    Kafka limits are customizable. 10MB is quite small. Kafka easily handles 100MB messages. If you need some temporal storage that works like a log then there's nothing better than kafka in terms of speed and scalability. But it's more about relatively long term storage. If you need to persist data for very short amount of time, why don't you use in-memory store? Do you have strict requirements around data loss? Something like Redis much just do the trick for out. With Redis cluster it even scales.

jamesblonde 5y ago

Interesting! We have been doing the same thing with HopsFS for a couple of years. Except, we only store the small files in our database (www.rondb.com) - RonDB is a recent fork of MySQL Cluster (NDBCluster). Very small files (<1KB) are stored in memory in RonDB, small files (typically <128KB) are stored in NVMe disks in RonDB, and other files in HopsFS (which now stores its blocks in object storage (S3, ABS). We had a paper on it as ACM Middleware and it's open-source on github. Are you going to publish your solution? (Discussed here on HN: https://news.ycombinator.com/item?id=25149154 https://news.ycombinator.com/item?id=25149154 )

jamesblonde 5y ago
Do you support atomic rename? Atomic rename on subtrees? Consistent directory listings?