3 ms·

ZeroFS [0] outperforms JuiceFS on common small file workloads [1] while only requiring S3 and no 3rd party database. [0] https://github.com/Barre/ZeroFS https

by Eikon 9mo ago

ZeroFS [0] outperforms JuiceFS on common small file workloads [1] while only requiring S3 and no 3rd party database. [0] https://github.com/Barre/ZeroFS https://github.com/Barre/ZeroFS [1] https://www.zerofs.net/zerofs-vs-juicefs https://www.zerofs.net/zerofs-vs-juicefs

deleted 9mo ago
[deleted]
huntaub 9mo ago
Respect to your work on ZeroFS, but I find it kind of off-putting for you to come in and immediately put down JuiceFS, especially with benchmark results that don't make a ton of sense, and are likely making apples-to-oranges comparisons with how JuiceFS works or mount options. For example, it doesn't really make sense that "92% of data modification operations" would fail on JuiceFS, which makes me question a lot of the methodology in these tests.
- Eikon 9mo ago
  > but I find it kind of off-putting for you to come in and immediately put down JuiceFS, especially with benchmark results that don't make a ton of sense, and are likely making apples-to-oranges comparisons with how JuiceFS works or mount options. The benchmark suite is trivial and opensource [1]. Is performing benchmarks “putting down” these days? If you believe that the benchmarks are unfair to juicefs for a reason or for another, please put up a PR with a better methodology or corrected numbers. I’d happily merge it. EDIT: From your profile, it seems like you are running a VC backed competitor, would be fair to mention that… [1] https://github.com/Barre/ZeroFS/tree/main/bench https://github.com/Barre/ZeroFS/tree/main/bench
  - huntaub 9mo ago
    Yes, I'm working in the space too. I think it's fine to do benchmarks, I don't think it's necessary to immediately post them any time a competitor comes up on HN. I don't want to see the cloud storage sector turn as bitter as the cloud database sector. I've previously looked through the benchmarking code, and I still have some serious concerns about the way that you're presenting things on your page.
    - zaphirplane 9mo ago
      > presenting things I don’t have a dog in this race, have to say thou the vagueness of the hand waving in multiple comments is losing you credibility
  - wgjordan 9mo ago
    > The benchmark suite is trivial and opensource. The actual code being benchmarked is trivial and open-source, but I don't see the actual JuiceFS setup anywhere in the ZeroFS repository. This means the self-published results don't seem to be reproducible by anyone looking to externally validate the stated claims in more detail. Given the very large performance differences, I have a hard time believing it's an actual apples-to-apples production-quality setup. It seems much more likely that some simple tuning is needed to make them more comparable, in which case the takeaway may be that JuiceFS may have more fiddly configuration without well-rounded defaults, not that it's actually hundreds of times slower when properly tuned for the workload. (That said, I'd love to be wrong and confidently discover that ZeroFS is indeed that much faster!)
- selfhoster1312 9mo ago
  I have very limited experiences with object storage, but my humble benchmarks with juicefs + minio/garage [1] showed very bad performance (i.e. total collapse within a few hours) when running lots of small operations (torrents). I wouldn't be surprised if there's a lot of tuning that can be achieved, but after days of reading docs and experimenting with different settings i just assumed JuiceFS was a very bad fit for archives shared through Bittorrent. I hope to be proven wrong, but in the meantime i'm very glad zerofs was mentioned as an alternative for small files/operations. I'll try to find the time to benchmark it too. [1] https://git.deuxfleurs.fr/Deuxfleurs/garage/issues/1021 https://git.deuxfleurs.fr/Deuxfleurs/garage/issues/1021
- eYrKEC2 9mo ago
  I'm always curious about the of the option space. I appreciate folks talking about the alternative s. What's yours?
  - huntaub 9mo ago
    Our product is Archil [1], and we are building our service on top of a durable, distributed SSD storage layer. As a result, we have the ability to: (a) store and use data in S3 in its native format [not a block based format like the other solutions in this thread], (b) durably commit writes to our storage layer with lower latency than products which operate as installable OSS libraries and communicate with S3 directly, and (c) handle multiple writers from different instances like NFS. Our team spent years working on NFS+Lustre products at Amazon (EFS and FSx for Lustre), so we understand the performance problems that these storage products have traditionally had. We've built a custom protocol that allows our users to achieve high-performance for small file operations (git -- perfect for coding agents) and highly-parallel HPC workloads (model training, inference). Obviously, there are tons of storage products because everyone makes different tradeoffs around durability, file size optimizations, etc. We're excited to have an approach that we think can flex around these properties dynamically, while providing best-in-class performance when compared to "true" storage systems like VAST, Weka, and Pure. [1] https://archil.com https://archil.com
deleted 9mo ago
[deleted]
corv 9mo ago
Looks like the underdog beats it handily and easier deployment to boot. What's the catch?
- aeblyve 9mo ago
  ZeroFS is a single-writer architecture and therefore has overall bandwidth limited by the box it's running on. JuiceFS scales out horizontally as each individual client writes/reads directly to/from S3, as long as the metadata engine keeps up it has essentially unlimited bandwidth across many compute nodes. But as the benchmark shows, it is fiddly especially for workloads with many small files and is pretty wasteful in terms of S3 operations, which for the largest workloads has meaningful cost. I think both have their place at the moment. But the space of "advanced S3-backed filesystems" is... advancing these days.
- deleted 9mo ago
  [deleted]
wgjordan 9mo ago
For a proper comparison, also significant to note that JuiceFS is Apache-2.0 licensed while ZeroFS is dual AGPL-3.0/commercial licensed, significantly limiting the latter's ability to be easily adopted outside of open source projects.
- anonymousDan 9mo ago
  Why would this matter if you're just using the database?
  - Eikon 9mo ago
    It doesn’t, you are free to use ZeroFS for commercial and closed source products.
    - wgjordan 9mo ago
      This clarification is helpful, thanks! The README currently implies a slightly different take, perhaps it could be made more clear that it's suitable for use unmodified in closed source products: > The AGPL license is suitable for open source projects, while commercial licenses are available for organizations requiring different terms. I was a bit unclear on where the AGPL's network-interaction clause draws its boundaries- so the commercial license would only be needed for closed-source modifications/forks, or if statically linking ZeroFS crate into a larger proprietary Rust program, is that roughly it?
      - Eikon 9mo ago
        > so the commercial license would only be needed for closed-source modifications/forks Indeed.
      - andydang 9mo ago
        [dead]
      - wgjordan 9mo ago
        Also worth noting (as a sibling comment pointed out) that despite these assurances the untested legal risks of AGPL-licensed code may still cause difficulties for larger, risk-averse companies. Google notably has a blanket policy [1] banning all AGPL code entirely as "the risks outweigh the benefits", so large organizations are probably another area where the commercial license comes into play. [1] https://opensource.google/documentation/reference/using/agpl-policy https://opensource.google/documentation/reference/using/agpl...
ChocolateGod 9mo ago
Let's remember that JuiceFS can be setup very easily to not have a single point of failure (by replicating the metadata engine), meanwhile ZeroFS seems to have exactly that. If I was a company I know which one I'd prefer.
- __turbobrew__ 9mo ago
  Yea, that is a big caveat to ZeroFS. Single point of failure. It is like saying I can write a faster etcd by only having a single node. Sure, that is possible, but the hard part of distributed systems is the coordination, and coordination always makes performance worse. I personally have went with Ceph for distributed storage. I personally have a lot more confidence in Ceph over JuiceFS and ZeroFS, but realize building and running a ceph cluster is more complex, but with that complexity you get much cheaper S3, block storage, and cephfs.
  - suavesu 9mo ago
    Some users use JuiceFS with CephFS RADOS, as alternative with Ceph MDS.
  - ChocolateGod 9mo ago
    I replaced a GlusterFS cluster with JuiceFS some years ago and it's been a relief. Just much easier to manage.
dpacmittal 9mo ago
The magnitude of performance difference alone immediately makes me skeptical of your benchmarking methodology.
- selfhoster1312 9mo ago
  I'm not an expert in any way, but i personally benchmarked [1] juiceFS performance totalling collapsing under very small files/operations (torrenting). It's good to be skeptical, but it might just be that the bar is very low for this specific usecase (IIRC juiceFS was configured and optimized for block sizes of several MBs). https://git.deuxfleurs.fr/Deuxfleurs/garage/issues/1021 https://git.deuxfleurs.fr/Deuxfleurs/garage/issues/1021
maxmcd 9mo ago
does having to maintain the slatedb as a consistent singleton (even with write fencing) make this as operationally tricky as a third party db?
- Eikon 9mo ago
  It’s not great UX on that angle. I am currently working on coordination (through s3, not node to node communication), so that you can just spawn instances without thinking about it.
victorbjorklund 9mo ago
Can SQLite run on it?
Dylan16807 9mo ago
> ZeroFS supports running multiple instances on the same storage backend: one read-write instance and multiple read-only instances. Well that's a big limiting factor that needs to be at the front in any distributed filesystem comparison. Though I'm confused, the page says things like "ZeroFS makes S3 behave like a regular block device", but in that case how do read-only instances mount it without constantly getting their state corrupted out from under them? Is that implicitly talking about the NBD access, and the other access modes have logic to handle that? Edit: What I want to see is a ZeroFS versus s3backer comparison. Edit 2: changed the question at the end

by Eikon 9mo ago

deleted 9mo ago

[deleted]

huntaub 9mo ago

Respect to your work on ZeroFS, but I find it kind of off-putting for you to come in and immediately put down JuiceFS, especially with benchmark results that don't make a ton of sense, and are likely making apples-to-oranges comparisons with how JuiceFS works or mount options. For example, it doesn't really make sense that "92% of data modification operations" would fail on JuiceFS, which makes me question a lot of the methodology in these tests.

Eikon 9mo ago
> but I find it kind of off-putting for you to come in and immediately put down JuiceFS, especially with benchmark results that don't make a ton of sense, and are likely making apples-to-oranges comparisons with how JuiceFS works or mount options. The benchmark suite is trivial and opensource [1]. Is performing benchmarks “putting down” these days? If you believe that the benchmarks are unfair to juicefs for a reason or for another, please put up a PR with a better methodology or corrected numbers. I’d happily merge it. EDIT: From your profile, it seems like you are running a VC backed competitor, would be fair to mention that… [1] https://github.com/Barre/ZeroFS/tree/main/bench https://github.com/Barre/ZeroFS/tree/main/bench
- huntaub 9mo ago
  Yes, I'm working in the space too. I think it's fine to do benchmarks, I don't think it's necessary to immediately post them any time a competitor comes up on HN. I don't want to see the cloud storage sector turn as bitter as the cloud database sector. I've previously looked through the benchmarking code, and I still have some serious concerns about the way that you're presenting things on your page.
  - zaphirplane 9mo ago
    > presenting things I don’t have a dog in this race, have to say thou the vagueness of the hand waving in multiple comments is losing you credibility
- wgjordan 9mo ago
  > The benchmark suite is trivial and opensource. The actual code being benchmarked is trivial and open-source, but I don't see the actual JuiceFS setup anywhere in the ZeroFS repository. This means the self-published results don't seem to be reproducible by anyone looking to externally validate the stated claims in more detail. Given the very large performance differences, I have a hard time believing it's an actual apples-to-apples production-quality setup. It seems much more likely that some simple tuning is needed to make them more comparable, in which case the takeaway may be that JuiceFS may have more fiddly configuration without well-rounded defaults, not that it's actually hundreds of times slower when properly tuned for the workload. (That said, I'd love to be wrong and confidently discover that ZeroFS is indeed that much faster!)
selfhoster1312 9mo ago
I have very limited experiences with object storage, but my humble benchmarks with juicefs + minio/garage [1] showed very bad performance (i.e. total collapse within a few hours) when running lots of small operations (torrents). I wouldn't be surprised if there's a lot of tuning that can be achieved, but after days of reading docs and experimenting with different settings i just assumed JuiceFS was a very bad fit for archives shared through Bittorrent. I hope to be proven wrong, but in the meantime i'm very glad zerofs was mentioned as an alternative for small files/operations. I'll try to find the time to benchmark it too. [1] https://git.deuxfleurs.fr/Deuxfleurs/garage/issues/1021 https://git.deuxfleurs.fr/Deuxfleurs/garage/issues/1021
eYrKEC2 9mo ago
I'm always curious about the of the option space. I appreciate folks talking about the alternative s. What's yours?
- huntaub 9mo ago
  Our product is Archil [1], and we are building our service on top of a durable, distributed SSD storage layer. As a result, we have the ability to: (a) store and use data in S3 in its native format [not a block based format like the other solutions in this thread], (b) durably commit writes to our storage layer with lower latency than products which operate as installable OSS libraries and communicate with S3 directly, and (c) handle multiple writers from different instances like NFS. Our team spent years working on NFS+Lustre products at Amazon (EFS and FSx for Lustre), so we understand the performance problems that these storage products have traditionally had. We've built a custom protocol that allows our users to achieve high-performance for small file operations (git -- perfect for coding agents) and highly-parallel HPC workloads (model training, inference). Obviously, there are tons of storage products because everyone makes different tradeoffs around durability, file size optimizations, etc. We're excited to have an approach that we think can flex around these properties dynamically, while providing best-in-class performance when compared to "true" storage systems like VAST, Weka, and Pure. [1] https://archil.com https://archil.com

deleted 9mo ago

[deleted]

corv 9mo ago

Looks like the underdog beats it handily and easier deployment to boot. What's the catch?

aeblyve 9mo ago
ZeroFS is a single-writer architecture and therefore has overall bandwidth limited by the box it's running on. JuiceFS scales out horizontally as each individual client writes/reads directly to/from S3, as long as the metadata engine keeps up it has essentially unlimited bandwidth across many compute nodes. But as the benchmark shows, it is fiddly especially for workloads with many small files and is pretty wasteful in terms of S3 operations, which for the largest workloads has meaningful cost. I think both have their place at the moment. But the space of "advanced S3-backed filesystems" is... advancing these days.
deleted 9mo ago
[deleted]

wgjordan 9mo ago

For a proper comparison, also significant to note that JuiceFS is Apache-2.0 licensed while ZeroFS is dual AGPL-3.0/commercial licensed, significantly limiting the latter's ability to be easily adopted outside of open source projects.

anonymousDan 9mo ago
Why would this matter if you're just using the database?
- Eikon 9mo ago
  It doesn’t, you are free to use ZeroFS for commercial and closed source products.
  - wgjordan 9mo ago
    This clarification is helpful, thanks! The README currently implies a slightly different take, perhaps it could be made more clear that it's suitable for use unmodified in closed source products: > The AGPL license is suitable for open source projects, while commercial licenses are available for organizations requiring different terms. I was a bit unclear on where the AGPL's network-interaction clause draws its boundaries- so the commercial license would only be needed for closed-source modifications/forks, or if statically linking ZeroFS crate into a larger proprietary Rust program, is that roughly it?
    - Eikon 9mo ago
      > so the commercial license would only be needed for closed-source modifications/forks Indeed.
    - andydang 9mo ago
      [dead]
    - wgjordan 9mo ago
      Also worth noting (as a sibling comment pointed out) that despite these assurances the untested legal risks of AGPL-licensed code may still cause difficulties for larger, risk-averse companies. Google notably has a blanket policy [1] banning all AGPL code entirely as "the risks outweigh the benefits", so large organizations are probably another area where the commercial license comes into play. [1] https://opensource.google/documentation/reference/using/agpl-policy https://opensource.google/documentation/reference/using/agpl...

ChocolateGod 9mo ago

Let's remember that JuiceFS can be setup very easily to not have a single point of failure (by replicating the metadata engine), meanwhile ZeroFS seems to have exactly that. If I was a company I know which one I'd prefer.

__turbobrew__ 9mo ago
Yea, that is a big caveat to ZeroFS. Single point of failure. It is like saying I can write a faster etcd by only having a single node. Sure, that is possible, but the hard part of distributed systems is the coordination, and coordination always makes performance worse. I personally have went with Ceph for distributed storage. I personally have a lot more confidence in Ceph over JuiceFS and ZeroFS, but realize building and running a ceph cluster is more complex, but with that complexity you get much cheaper S3, block storage, and cephfs.
- suavesu 9mo ago
  Some users use JuiceFS with CephFS RADOS, as alternative with Ceph MDS.
- ChocolateGod 9mo ago
  I replaced a GlusterFS cluster with JuiceFS some years ago and it's been a relief. Just much easier to manage.

dpacmittal 9mo ago

The magnitude of performance difference alone immediately makes me skeptical of your benchmarking methodology.

selfhoster1312 9mo ago
I'm not an expert in any way, but i personally benchmarked [1] juiceFS performance totalling collapsing under very small files/operations (torrenting). It's good to be skeptical, but it might just be that the bar is very low for this specific usecase (IIRC juiceFS was configured and optimized for block sizes of several MBs). https://git.deuxfleurs.fr/Deuxfleurs/garage/issues/1021 https://git.deuxfleurs.fr/Deuxfleurs/garage/issues/1021

maxmcd 9mo ago

does having to maintain the slatedb as a consistent singleton (even with write fencing) make this as operationally tricky as a third party db?

Eikon 9mo ago
It’s not great UX on that angle. I am currently working on coordination (through s3, not node to node communication), so that you can just spawn instances without thinking about it.

victorbjorklund 9mo ago

Can SQLite run on it?

Dylan16807 9mo ago

> ZeroFS supports running multiple instances on the same storage backend: one read-write instance and multiple read-only instances. Well that's a big limiting factor that needs to be at the front in any distributed filesystem comparison. Though I'm confused, the page says things like "ZeroFS makes S3 behave like a regular block device", but in that case how do read-only instances mount it without constantly getting their state corrupted out from under them? Is that implicitly talking about the NBD access, and the other access modes have logic to handle that? Edit: What I want to see is a ZeroFS versus s3backer comparison. Edit 2: changed the question at the end