4 ms·

We were doing this at Avere in 2015. The system built a POSIX filesystem out of S3 objects on the backend (including metadata) and then served it over NFS or SM

We were doing this at Avere in 2015. The system built a POSIX filesystem out of S3 objects on the backend (including metadata) and then served it over NFS or SMB from a cluster of cache nodes. Keeping metadata and data in separate data stores with different consistency models is a disaster waiting to happen - ask anyone who has run Lustre. Having fast caches with SSDs was the key to getting any kind of decent performance out of S3. The fun part was mounting a filesystem and running df and seeing an exabyte of capacity. They were acquired by Microsoft in 2018 and integrated into Azure.

daviesliu 6y ago
I saw Avere was recommended in GCP before, but never find the details, thanks for sharing that. It seems that Avere is close to ObjectiveFS, which also use S3 both for data and metadata. My guest is that Avere could require a cluster of nodes as the fast layer for write and synchronization. In JuiceFS, Redis is used for synchronization and persisting metadata. Local SSD could used for data read-caching, not for writing. JuiceFS is in production for 3+ years, we have not find much challenge that was difficult with this design, since it's borrowed from GFS, HDFS and MooseFS, have be proved in production for more than 10 years. I'd like to hear what's the challenge you were facing.
- mprovost 6y ago
  The nice thing about storing everything including metadata in S3 is that you have a consistent copy of your filesystem. How do you back up a filesystem that is split across Redis and S3 for example? I was able to blow away the cluster running it, spin up a new one, point it at the bucket and add the encryption key, and it would bring up a working filesystem. Just like plugging in a hard drive.
  - daviesliu 6y ago
    JuiceFS has a single node version working like this, it store in the metadata in SQLite and upload S3 in background, it's useful in some cases but not in general, you may lose recent metadata if the single node is broken. The difficulty of using S3 as metadata is that they is not way to persistent metadata under 1 ms, for example, created a symlink, it will take more than 20ms, or you may lose it. With en external persisted database, we could have the ACID for metadata operations. Also, the meta is the source of truth, whenever the object store is out of sync (losing or leaking a object), the whole file system is still consistent, rather than part of a file is corrupted, should be safer than put everything into S3. The database is the key, redis is our first choice, we will add support to other databases in future.
bogomipz 6y ago
>"Keeping metadata and data in separate data stores with different consistency models is a disaster waiting to happen - ask anyone who has run Lustre." Can you elaborate - is the issue corruption or performance? I've never used Lustre.
- mprovost 6y ago
  Lustre is all about performance, at the cost of being split across multiple distributed systems. So there are opportunities to have different components get out of sync - the metadata servers are usually the weak link. And the more moving parts the harder it is to debug when something goes wrong.

daviesliu 6y ago

I saw Avere was recommended in GCP before, but never find the details, thanks for sharing that. It seems that Avere is close to ObjectiveFS, which also use S3 both for data and metadata. My guest is that Avere could require a cluster of nodes as the fast layer for write and synchronization. In JuiceFS, Redis is used for synchronization and persisting metadata. Local SSD could used for data read-caching, not for writing. JuiceFS is in production for 3+ years, we have not find much challenge that was difficult with this design, since it's borrowed from GFS, HDFS and MooseFS, have be proved in production for more than 10 years. I'd like to hear what's the challenge you were facing.

mprovost 6y ago
The nice thing about storing everything including metadata in S3 is that you have a consistent copy of your filesystem. How do you back up a filesystem that is split across Redis and S3 for example? I was able to blow away the cluster running it, spin up a new one, point it at the bucket and add the encryption key, and it would bring up a working filesystem. Just like plugging in a hard drive.
- daviesliu 6y ago
  JuiceFS has a single node version working like this, it store in the metadata in SQLite and upload S3 in background, it's useful in some cases but not in general, you may lose recent metadata if the single node is broken. The difficulty of using S3 as metadata is that they is not way to persistent metadata under 1 ms, for example, created a symlink, it will take more than 20ms, or you may lose it. With en external persisted database, we could have the ACID for metadata operations. Also, the meta is the source of truth, whenever the object store is out of sync (losing or leaking a object), the whole file system is still consistent, rather than part of a file is corrupted, should be safer than put everything into S3. The database is the key, redis is our first choice, we will add support to other databases in future.

bogomipz 6y ago

>"Keeping metadata and data in separate data stores with different consistency models is a disaster waiting to happen - ask anyone who has run Lustre." Can you elaborate - is the issue corruption or performance? I've never used Lustre.

mprovost 6y ago
Lustre is all about performance, at the cost of being split across multiple distributed systems. So there are opportunities to have different components get out of sync - the metadata servers are usually the weak link. And the more moving parts the harder it is to debug when something goes wrong.