4 ms·

Show HN: ZeroFS, the Filesystem That Makes S3 Your Primary Storage

64 points by Eikon 1y ago

siliconc0w 1y ago
Very cool to see NFS, NDB, or even P9 supported. Looking at those benchmarks, I think you must be using a local disk to sync writes before uploading to S3?
- Eikon 1y ago
  There’s no on-disk write cache.
jauntywundrkind 1y ago
Built stop the excellent SlateDB! Breaks files down into 256k chunks. Encrypted. Much much better posix compatibility than most FUSE alternatives. SlateDB has snapshots & clones, so that could be another great superpower of zerofs. Incredible performance figures, rocketing to probably the best way to use object storage in an fs like way. There's a whole series of comparisons, & they probably need a logarithmic scale given the scale of the lead slatedb has! https://www.zerofs.net/zerofs-vs-juicefs https://www.zerofs.net/zerofs-vs-juicefs Speaks 9p, NFS, or NBD. Some great demos of ZFS with l2arc caches giving a near local performance while having s3 persistence. Totally what I was thinking of when in the Immich someone mentioned wanting a way to run it on cheap object storage. https://news.ycombinator.com/item?id=45169036 https://news.ycombinator.com/item?id=45169036
- ramses0 1y ago
  The concept of "auto"-tiered, transparent storage is _really_ compelling! ...and the use of AGPL is really clever as an "enterprise poison pill". It still sucks that S3 is ~$20/mo/TB basically "in perpetuity", while random SSD drives are ~$80/TB and I'd feel comfortable effectively amortizing them out at ~$20/yr for local storage instead of $20/mo for S3. :-/
  - Eikon 1y ago
    > It still sucks that S3 is ~$20/mo/TB basically "in perpetuity", while random SSD drives are ~$80/TB and I'd feel comfortable effectively amortizing them out at ~$20/yr for local storage instead of $20/mo for S3. :-/ ZeroFS will work with any S3 implementation that supports conditional writes. Even something a bit premium-priced like CloudFlare R2 goes for around $15/TB and has no egress costs. If you want to go for as cheap as possible, I'd throw minio or similar on a few cheap Hetzner storage servers https://www.hetzner.com/dedicated-rootserver/matrix-sx/ https://www.hetzner.com/dedicated-rootserver/matrix-sx/
    - aspenmayer 1y ago
      How does this compare to rclone, as I also asked in a top level comment? rclone can mount S3 as a union mount so it seems to have the same feature set? Have you benchmarked ZeroFS against rclone? https://news.ycombinator.com/item?id=45178548 https://news.ycombinator.com/item?id=45178548
      - Eikon 1y ago
        Unless I am missing something, rclone will probably compare similarly to https://www.zerofs.net/zerofs-vs-mountpoint-s3 https://www.zerofs.net/zerofs-vs-mountpoint-s3
        aspenmayer 1y ago
        One big difference I see is that rclone is MIT licensed, whereas ZeroFS is licensed under AGPL with commercial license also available? https://www.zerofs.net/licensing https://www.zerofs.net/licensing
monkaiju 1y ago
Finally a way to get more than 2gb of local storage on digital ocean's app platform!
- nodesocket 1y ago
  2gb? They have storage optimized nvme droplets that at the high end support 4.6tb (though absurdly expensive at $2,000/mo).
visualphoenix 1y ago
Seems to be nfs v3 [0] - curious to test it out - the only userspace nfsv4 implementation I’m aware of is in buildbarn (golang) [1]. The example of their nfs v3 implementation disables locking. Still pretty cool to see all the ways the rust ecosystem is empowering stuff like this. I’m kinda surprised someone hasn’t integrated the buildbarn nfs v4 stuff into docker/podman - the virtiofs stuff is pretty bad on osx and the buildbarn nfs 4.0 stuff is a big improvement over nfs v3. Anyhow I digress. Can’t wait to take it for a spin. [0] https://github.com/Barre/zerofs_nfsserve https://github.com/Barre/zerofs_nfsserve [1] https://github.com/buildbarn/bb-remote-execution/tree/master/pkg/filesystem/virtual/nfsv4 https://github.com/buildbarn/bb-remote-execution/tree/master...
- Eikon 1y ago
  BuildBarn's nfsv4 server looks great! I'm not too sure about NFSv4 for ZeroFS specifically, the benefits don't seem _that_ obvious to me, especially compared to something like 9P, where server-side locking in the ".L" version is native anyway. Especially considering that most ZeroFS users are mounting the fs on the same box as the server is running and so local locks are working. MacOS is a bit of a blind spot for 9P though, and a client there would be great.
akshayKMR 1y ago
Incredibly cool! It shows running Ubuntu and Postgres, and also supports full posix operations. Questions: - I see there is a diagram running multiple Postgres nodes backed by this store, very similar to horizontally distributed web-server. Doesn't Postgres use WAL replication? Or is it disabled and they are they running on same "views" of the filesystem? - What does this mean for services that handle geo-distribution on app layer? e.g. CockroachDB? Sorry if this sounds dumb.
williamstein 1y ago
They have a bunch of claims comparing this to JuiceFS at https://www.zerofs.net/zerofs-vs-juicefs https://www.zerofs.net/zerofs-vs-juicefs I am in no way affiliated with JuiceFS, but I have done a lot of benchmarking and testing of it, and the numbers claimed here for JuiceFS are suspicious (basically 5 ops/second with everything mostly broken).
- ChocolateGod 1y ago
  I have a juicefs setup and get operations in the 1000/s, not sure how they got such low numbers. JuiceFS also supports multiple concurrent clients making their own connection to the metadata and object storage, allowing near instant synchronization and better performance, where this seems to rely on a single service having a connection and everyone connecting through it with no support for clustering.
  - deleted 1y ago
    [deleted]
  - Eikon 1y ago
    I have no doubt that JuiceFS can perform “thousands of operations per second” across parallel clients. I don't think that's a useful benchmark because the use cases we are targeting are not embarrassingly parallel. Using a bunch of clients on any system hides the latency profile. You could even get your "thousands of operations per second" on a system where any operation takes 10 seconds to complete.
    - ChocolateGod 1y ago
      I'm referring to one client being able to do 1000s of operations a second. I've not experienced this bad performance with JuiceFS that the article is describing. This is a JuiceFS setup with 10TB of data, where JuiceFS was specifically chosen because of its minimal latency compared to raw object storage.
      - Eikon 1y ago
anorwell 1y ago
Seems like a really interesting project! I don't understand what's going on with latency vs durability here. The benchmarks [1] report ~1ms latency for sequential writes, but that's just not possible with S3. So presumably writes are not being confirmed to storage before confirming the write to the client. What is the durability model? The docs don't talk about intermediate storage. Slatedb does confirm writes to S3 by default, but I assume that's not happening? [1] https://www.zerofs.net/zerofs-vs-juicefs https://www.zerofs.net/zerofs-vs-juicefs
- Shakahs 1y ago
  SlateDB offers different durability levels for writes. By default writes are buffered locally and flushed to S3 when the buffer is full or the client invokes flush(). https://slatedb.io/docs/design/writes/ https://slatedb.io/docs/design/writes/
- Eikon 1y ago
  The durability profile before sync should be pretty close to a local filesystem. There’s (in-memory) buffering happening on writes, then when fsync is issued or when we exceed the in-memory threshold or we exceed a timeout, data is sync-ed.
  - anorwell 1y ago
    Thanks, makes sense. I found the benchmark src to see it's not fsyncing, so only some of the files will be durable by the time the benchmark is done. The benchmark docs might benefit from discussing this or benchmarking both cases? O_SYNC / fsync before file close is an important use case. edit: A quirk with the use of NFSv3 here is that there's no specific close op. So, if I understand right, ZeroFS' "close-to-open consistency" doesn't imply durability on close (and can't unless every NFS op is durable before returning), only on fsync. Whereas EFS and (I think?) azure files do have this property.
    - Eikon 1y ago
      There's an NFSv3 COMMIT operation, combined with a "durability" marker on writes. fsync could translate to COMMIT, but if writes are marked as "durable", COMMIT is not called by common clients, and if writes are marked as non-durable, COMMIT is called after every operation, which kind of defeats the point. When you use NFS with ZeroFS, you cannot really rely on "fsync". I'd recommend using 9P when that matters, which has proper semantics there. One property of ZeroFS is that any file you fsync actually syncs everything else too.
moltar 1y ago
Has anyone tried it as a cache dir in CI? I’m concerned about random reads looking pretty slow in the benchmarks.
billev2k 1y ago
I had to laugh out loud: "In practice, you'll encounter other constraints well before these theoretical limits, such as S3 provider limits, performance considerations with billions of objects, or simply running out of money."
aspenmayer 1y ago
How does this compare to rclone? https://rclone.org/ https://rclone.org/ https://github.com/rclone/rclone https://github.com/rclone/rclone
- sedawkgrep 1y ago
  At a glance, ZeroFS appears to be an overlay filesystem above a VFS. It looks like it abstracts the underlying filesystem configuration/complication and creates a simplified, unified configuration that could allow you to be more dynamic underneath. It also appears to support a great deal of IO and permission complexity, which makes sense as it refers to itself as a filesystem. It also purports to be extremely fast. In the end it looks to function like an actual filesystem with all that entails. rclone (my first time hearing about it) appears to be a much simpler solution with different design goals. It looks more like rsync in most instances, or a user-space link to a remote filesystem in others (like ftp or smbclient). The mount capability, while similar to zerofs, looks more like something much simpler and targeted for less-sophisticated needs. Not saying one's better than the other; merely that they really look to be serving different needs/goals while having some overlap.
  - aspenmayer 1y ago
    rclone is MIT license. ZeroFS is AGPL with commercial license available. https://www.zerofs.net/licensing https://www.zerofs.net/licensing
jiggawatts 1y ago
It looks bizarre to see that Azure Storage support was included. That's because there's no need to use fancy SlateDB+LSM tree code to simulate a block device on Azure Storage, it already natively provides "page blobs" that can be mounted as standard disks and used for boot disks, databases, or whatever. These are used as the backing storage for all Azure virtual machine disks and can also be used directly via the HTTP API. For example, SQL Server can store its database files on page blobs, without having to attach them as disks and formatting them as NTFS volumes. See: https://learn.microsoft.com/en-us/azure/storage/blobs/storage-blob-pageblob-overview https://learn.microsoft.com/en-us/azure/storage/blobs/storag...
- Eikon 1y ago
  What's so bizarre about it? Are we prevented from implementing features when the platform itself provides adjacent tech? Page blobs are 2x+ more expensive ($0.045/GB vs $0.02/GB) and would require Azure-specific code. We don't dream of maintaining separate implementations for each cloud provider. The approach works identically on S3, GCS, Azure, and local storage. If anything, ZeroFS helps you distance from vendor lock-in, especially if you start to run "usually managed" components on top, say databases. Speaking of Azure's "native" solutions - we benchmarked ZeroFS against Azure Files. ZeroFS is 35-41x faster for most operations and 38% cheaper. If Azure Files performs that poorly, I don't hold much hope for page blobs either: https://www.zerofs.net/zerofs-vs-azure-files https://www.zerofs.net/zerofs-vs-azure-files
  - jiggawatts 1y ago
    Fair points. Note that Azure Files is a significantly different offering than block storage, it is essentially a Windows file server cluster and uses the SMB v3.1.1 protocol. It's slow, but not that slow. It looks like you benchmarked against the HDD version of it, which few people use. Everyone who cares about its performance uses Azure Files Premium, which is the pure-SSD version of the service. They recently updated it to have flexible performance that can be cheaper than the older HDD storage: https://techcommunity.microsoft.com/blog/azurestorageblog/lower-costs-and-boost-flexibility-with-azure-files-provisioned-v2/4443621 https://techcommunity.microsoft.com/blog/azurestorageblog/lo... You're also mounting it from Linux, which is just... weird. Nobody (for some values of nobody) does this. The sole purpose for this service is to provide backwards-compatible file shares for Windows servers and desktops. Benchmarking cloud services can be tricky because they have odd quirks such as IOPS-per-GB ratios, so nearly empty storage can be much slower than you'd expect!
arch_rust 1y ago
Powered by deku! https://github.com/sharksforarms/deku https://github.com/sharksforarms/deku
- Eikon 1y ago
  Deku is great!
up2isomorphism 1y ago
For personal user, just don’t use things like S3. For business they won’t use things like this either. Just don’t understand what’s the point of creating things like this.
- Eikon 1y ago
  > For personal user, just don’t use things like S3 Why? > For business they won’t use things like this either. Again, why not?

64 points by Eikon 1y ago

Very cool to see NFS, NDB, or even P9 supported. Looking at those benchmarks, I think you must be using a local disk to sync writes before uploading to S3?

Eikon 1y ago
There’s no on-disk write cache.

jauntywundrkind 1y ago

Built stop the excellent SlateDB! Breaks files down into 256k chunks. Encrypted. Much much better posix compatibility than most FUSE alternatives. SlateDB has snapshots & clones, so that could be another great superpower of zerofs. Incredible performance figures, rocketing to probably the best way to use object storage in an fs like way. There's a whole series of comparisons, & they probably need a logarithmic scale given the scale of the lead slatedb has! https://www.zerofs.net/zerofs-vs-juicefs https://www.zerofs.net/zerofs-vs-juicefs Speaks 9p, NFS, or NBD. Some great demos of ZFS with l2arc caches giving a near local performance while having s3 persistence. Totally what I was thinking of when in the Immich someone mentioned wanting a way to run it on cheap object storage. https://news.ycombinator.com/item?id=45169036 https://news.ycombinator.com/item?id=45169036

ramses0 1y ago
The concept of "auto"-tiered, transparent storage is _really_ compelling! ...and the use of AGPL is really clever as an "enterprise poison pill". It still sucks that S3 is ~$20/mo/TB basically "in perpetuity", while random SSD drives are ~$80/TB and I'd feel comfortable effectively amortizing them out at ~$20/yr for local storage instead of $20/mo for S3. :-/
- Eikon 1y ago
  > It still sucks that S3 is ~$20/mo/TB basically "in perpetuity", while random SSD drives are ~$80/TB and I'd feel comfortable effectively amortizing them out at ~$20/yr for local storage instead of $20/mo for S3. :-/ ZeroFS will work with any S3 implementation that supports conditional writes. Even something a bit premium-priced like CloudFlare R2 goes for around $15/TB and has no egress costs. If you want to go for as cheap as possible, I'd throw minio or similar on a few cheap Hetzner storage servers https://www.hetzner.com/dedicated-rootserver/matrix-sx/ https://www.hetzner.com/dedicated-rootserver/matrix-sx/
  - aspenmayer 1y ago
    How does this compare to rclone, as I also asked in a top level comment? rclone can mount S3 as a union mount so it seems to have the same feature set? Have you benchmarked ZeroFS against rclone? https://news.ycombinator.com/item?id=45178548 https://news.ycombinator.com/item?id=45178548
    - Eikon 1y ago
      Unless I am missing something, rclone will probably compare similarly to https://www.zerofs.net/zerofs-vs-mountpoint-s3 https://www.zerofs.net/zerofs-vs-mountpoint-s3
      - aspenmayer 1y ago
        One big difference I see is that rclone is MIT licensed, whereas ZeroFS is licensed under AGPL with commercial license also available? https://www.zerofs.net/licensing https://www.zerofs.net/licensing

monkaiju 1y ago

Finally a way to get more than 2gb of local storage on digital ocean's app platform!

nodesocket 1y ago
2gb? They have storage optimized nvme droplets that at the high end support 4.6tb (though absurdly expensive at $2,000/mo).

visualphoenix 1y ago

Seems to be nfs v3 [0] - curious to test it out - the only userspace nfsv4 implementation I’m aware of is in buildbarn (golang) [1]. The example of their nfs v3 implementation disables locking. Still pretty cool to see all the ways the rust ecosystem is empowering stuff like this. I’m kinda surprised someone hasn’t integrated the buildbarn nfs v4 stuff into docker/podman - the virtiofs stuff is pretty bad on osx and the buildbarn nfs 4.0 stuff is a big improvement over nfs v3. Anyhow I digress. Can’t wait to take it for a spin. [0] https://github.com/Barre/zerofs_nfsserve https://github.com/Barre/zerofs_nfsserve [1] https://github.com/buildbarn/bb-remote-execution/tree/master/pkg/filesystem/virtual/nfsv4 https://github.com/buildbarn/bb-remote-execution/tree/master...

Eikon 1y ago
BuildBarn's nfsv4 server looks great! I'm not too sure about NFSv4 for ZeroFS specifically, the benefits don't seem _that_ obvious to me, especially compared to something like 9P, where server-side locking in the ".L" version is native anyway. Especially considering that most ZeroFS users are mounting the fs on the same box as the server is running and so local locks are working. MacOS is a bit of a blind spot for 9P though, and a client there would be great.

akshayKMR 1y ago

Incredibly cool! It shows running Ubuntu and Postgres, and also supports full posix operations. Questions: - I see there is a diagram running multiple Postgres nodes backed by this store, very similar to horizontally distributed web-server. Doesn't Postgres use WAL replication? Or is it disabled and they are they running on same "views" of the filesystem? - What does this mean for services that handle geo-distribution on app layer? e.g. CockroachDB? Sorry if this sounds dumb.

williamstein 1y ago

They have a bunch of claims comparing this to JuiceFS at https://www.zerofs.net/zerofs-vs-juicefs https://www.zerofs.net/zerofs-vs-juicefs I am in no way affiliated with JuiceFS, but I have done a lot of benchmarking and testing of it, and the numbers claimed here for JuiceFS are suspicious (basically 5 ops/second with everything mostly broken).

ChocolateGod 1y ago
I have a juicefs setup and get operations in the 1000/s, not sure how they got such low numbers. JuiceFS also supports multiple concurrent clients making their own connection to the metadata and object storage, allowing near instant synchronization and better performance, where this seems to rely on a single service having a connection and everyone connecting through it with no support for clustering.
- deleted 1y ago
  [deleted]
- Eikon 1y ago
  I have no doubt that JuiceFS can perform “thousands of operations per second” across parallel clients. I don't think that's a useful benchmark because the use cases we are targeting are not embarrassingly parallel. Using a bunch of clients on any system hides the latency profile. You could even get your "thousands of operations per second" on a system where any operation takes 10 seconds to complete.
  - ChocolateGod 1y ago
    I'm referring to one client being able to do 1000s of operations a second. I've not experienced this bad performance with JuiceFS that the article is describing. This is a JuiceFS setup with 10TB of data, where JuiceFS was specifically chosen because of its minimal latency compared to raw object storage.
    - Eikon 1y ago

anorwell 1y ago

Seems like a really interesting project! I don't understand what's going on with latency vs durability here. The benchmarks [1] report ~1ms latency for sequential writes, but that's just not possible with S3. So presumably writes are not being confirmed to storage before confirming the write to the client. What is the durability model? The docs don't talk about intermediate storage. Slatedb does confirm writes to S3 by default, but I assume that's not happening? [1] https://www.zerofs.net/zerofs-vs-juicefs https://www.zerofs.net/zerofs-vs-juicefs

Shakahs 1y ago
SlateDB offers different durability levels for writes. By default writes are buffered locally and flushed to S3 when the buffer is full or the client invokes flush(). https://slatedb.io/docs/design/writes/ https://slatedb.io/docs/design/writes/
Eikon 1y ago
The durability profile before sync should be pretty close to a local filesystem. There’s (in-memory) buffering happening on writes, then when fsync is issued or when we exceed the in-memory threshold or we exceed a timeout, data is sync-ed.
- anorwell 1y ago
  Thanks, makes sense. I found the benchmark src to see it's not fsyncing, so only some of the files will be durable by the time the benchmark is done. The benchmark docs might benefit from discussing this or benchmarking both cases? O_SYNC / fsync before file close is an important use case. edit: A quirk with the use of NFSv3 here is that there's no specific close op. So, if I understand right, ZeroFS' "close-to-open consistency" doesn't imply durability on close (and can't unless every NFS op is durable before returning), only on fsync. Whereas EFS and (I think?) azure files do have this property.
  - Eikon 1y ago
    There's an NFSv3 COMMIT operation, combined with a "durability" marker on writes. fsync could translate to COMMIT, but if writes are marked as "durable", COMMIT is not called by common clients, and if writes are marked as non-durable, COMMIT is called after every operation, which kind of defeats the point. When you use NFS with ZeroFS, you cannot really rely on "fsync". I'd recommend using 9P when that matters, which has proper semantics there. One property of ZeroFS is that any file you fsync actually syncs everything else too.

moltar 1y ago

Has anyone tried it as a cache dir in CI? I’m concerned about random reads looking pretty slow in the benchmarks.

billev2k 1y ago

I had to laugh out loud: "In practice, you'll encounter other constraints well before these theoretical limits, such as S3 provider limits, performance considerations with billions of objects, or simply running out of money."

aspenmayer 1y ago

How does this compare to rclone? https://rclone.org/ https://rclone.org/ https://github.com/rclone/rclone https://github.com/rclone/rclone

sedawkgrep 1y ago
At a glance, ZeroFS appears to be an overlay filesystem above a VFS. It looks like it abstracts the underlying filesystem configuration/complication and creates a simplified, unified configuration that could allow you to be more dynamic underneath. It also appears to support a great deal of IO and permission complexity, which makes sense as it refers to itself as a filesystem. It also purports to be extremely fast. In the end it looks to function like an actual filesystem with all that entails. rclone (my first time hearing about it) appears to be a much simpler solution with different design goals. It looks more like rsync in most instances, or a user-space link to a remote filesystem in others (like ftp or smbclient). The mount capability, while similar to zerofs, looks more like something much simpler and targeted for less-sophisticated needs. Not saying one's better than the other; merely that they really look to be serving different needs/goals while having some overlap.
- aspenmayer 1y ago
  rclone is MIT license. ZeroFS is AGPL with commercial license available. https://www.zerofs.net/licensing https://www.zerofs.net/licensing

jiggawatts 1y ago

It looks bizarre to see that Azure Storage support was included. That's because there's no need to use fancy SlateDB+LSM tree code to simulate a block device on Azure Storage, it already natively provides "page blobs" that can be mounted as standard disks and used for boot disks, databases, or whatever. These are used as the backing storage for all Azure virtual machine disks and can also be used directly via the HTTP API. For example, SQL Server can store its database files on page blobs, without having to attach them as disks and formatting them as NTFS volumes. See: https://learn.microsoft.com/en-us/azure/storage/blobs/storage-blob-pageblob-overview https://learn.microsoft.com/en-us/azure/storage/blobs/storag...

Eikon 1y ago
What's so bizarre about it? Are we prevented from implementing features when the platform itself provides adjacent tech? Page blobs are 2x+ more expensive ($0.045/GB vs $0.02/GB) and would require Azure-specific code. We don't dream of maintaining separate implementations for each cloud provider. The approach works identically on S3, GCS, Azure, and local storage. If anything, ZeroFS helps you distance from vendor lock-in, especially if you start to run "usually managed" components on top, say databases. Speaking of Azure's "native" solutions - we benchmarked ZeroFS against Azure Files. ZeroFS is 35-41x faster for most operations and 38% cheaper. If Azure Files performs that poorly, I don't hold much hope for page blobs either: https://www.zerofs.net/zerofs-vs-azure-files https://www.zerofs.net/zerofs-vs-azure-files
- jiggawatts 1y ago
  Fair points. Note that Azure Files is a significantly different offering than block storage, it is essentially a Windows file server cluster and uses the SMB v3.1.1 protocol. It's slow, but not that slow. It looks like you benchmarked against the HDD version of it, which few people use. Everyone who cares about its performance uses Azure Files Premium, which is the pure-SSD version of the service. They recently updated it to have flexible performance that can be cheaper than the older HDD storage: https://techcommunity.microsoft.com/blog/azurestorageblog/lower-costs-and-boost-flexibility-with-azure-files-provisioned-v2/4443621 https://techcommunity.microsoft.com/blog/azurestorageblog/lo... You're also mounting it from Linux, which is just... weird. Nobody (for some values of nobody) does this. The sole purpose for this service is to provide backwards-compatible file shares for Windows servers and desktops. Benchmarking cloud services can be tricky because they have odd quirks such as IOPS-per-GB ratios, so nearly empty storage can be much slower than you'd expect!

arch_rust 1y ago

Eikon 1y ago
Deku is great!

up2isomorphism 1y ago

For personal user, just don’t use things like S3. For business they won’t use things like this either. Just don’t understand what’s the point of creating things like this.

Eikon 1y ago
> For personal user, just don’t use things like S3 Why? > For business they won’t use things like this either. Again, why not?