5 ms·

High-availability durable filesystem is a difficult problem to solve. It usually starts with NFS, which is a big huge single point of failure. Depending on the

High-availability durable filesystem is a difficult problem to solve. It usually starts with NFS, which is a big huge single point of failure. Depending on the nature of the application this might be good enough. But if it's not, you'll typically want cross-datacenter replication so if one rack goes down you don't lose all your data. So then you're looking at something like Glusterfs/MooseFS/Ceph. But the latencies involved with synchronously replicating to multiple datacenters can really kill your performance. For example, try git cloning a large project onto a Glusterfs mount with >20ms ping between nodes. It's brutal. Other products try to do asynchronous replication, EdgeFS is one I was looking at recently. This follows the general industry trend, like it or not, of "eventually consistent is consistent enough". Not much better than a cron job + rsync, in my opinion, but for some workloads it's good enough. If there's a partition you'll lose data. Eventually you just give up and realize that a perfectly synchronized geographically distributed POSIX filesystem is a pipe dream, you bite the bullet and re-write your app against S3 and call it a day.

hanikesn 6y ago
Quobyte claim to have solved the synchronized distributed POSIX filesystem (mod geographically ).
objectivefs 6y ago
ObjectiveFS takes a different approach to high-availability durable file system by using S3 as the storage backend and building a log structured file system on top. This leverages the durability and scalability of S3 while allowing e.g. git cloning/diffing of a large project (even cross region) to remain fast. To compare we have some small file benchmark results at https://objectivefs.com/howto/performance-amazon-efs-vs-objectivefs https://objectivefs.com/howto/performance-amazon-efs-vs-obje... .
fxtentacle 6y ago
> write your app against S3 and call it a day. ... and then later you notice that you need a special SLA and Amazon Redshift to guarantee that a read from S3 will return the same value as the last write. Even S3 is only eventually consistent and especially if a US user uploads images into your US bucket and then you serve the URLs to EU users, you might have loading problems. The correct solution, according to our support contact, is that we wait a second after uploading to S3 to make sure at least the metadata has replicated to the EU before we post the image url. That, or pay for Redshift to tell us how long to wait more precisely. Because contrary to what everyone expects, S3 requests to US buckets might be served by delayed replicas in other countries if the request comes from outside the US.
- Galanwe 6y ago
  > The correct solution, according to our support contact, is that we wait a second after uploading to S3 I'm shocked that you got this answer. This is definitely not how you are supposed to operate. If you need to ensure the sequantiality of a write followed by a read on S3, the idiomatic way is to enable versioning on your bucket, issue a write, and provide the version ID to whoever need to read after that write. Not only will that transparently ensure that you will not read deprecated data, but it will even ensure that you actually read the result of that particular write, and not any consecutive write that could have happened in between. This pattern is very easy to implement for sequential operations in a single function, like: version = s3.put_object(...) data = s3.get_object(version_id=version, ...) I agree that when you deal with various processes it can become messy. In practice it never bothered me too much though. I prefer having explicit synchronization through versions, rather than having a blocking write waiting for all caches to be synchronized. Also, this should only be necessary if you write to an already existing object. New keys will always trigger a cache invalidation from my understanding.
  - fxtentacle 6y ago
    Depending on your write throughput, versioning can be quite expensive. Also, we were already at a throughput where the s3 metadata nodes would sometimes go offline, so I'm not sure putting more work on them would have improved our overall situation.
    - Galanwe 6y ago
      Interesting, do you have more information on how you did monitors these S3 metadata nodes? Never heard of that. Why would an unversioned put be less costly than a versioned put? To me it should be pretty much the same. I would almost suspect unversioned puts are hacked around versioned ones. Out of curiosity what kind of workload did you perform? I have been moving terabytes of files (some very small 1kB files, some huge 150GB files) and never noticed a single blip of change in behavior from S3 (and certainly not anything going "off")
fock 6y ago
wait, S3 can do git now? I guess you are right in nearly everything... but starting with git on gluster and then jumping to selling apples as bananas (e.g. S3) doesn't really make an argument.
the8472 6y ago
> For example, try git cloning a large project onto a Glusterfs mount with >20ms ping between nodes. It's brutal. That may be true but also is due to applications often having very sequential IO patterns even when they don't need to be. I hope we'll get some convenience wrappers around io_uring that make batching of many small IO calls in a synchronous manner simple and easy for cases where you don't want to deal with async runtimes. E.g. bulk_statx() or fsync_many() would be prime candidates for batching.
harshaw 6y ago
EFS attempts to solve this problem, although the speed of light between data centers is still a minor challenge.
slaymaker1907 6y ago
You can minimize the effects of eventual consistency by distinguishing the distant replicas from the same-datacenter ones. Cross datacenter/region might be eventually consistent, but you will only see it rarely since the same-datacenter replicas are fully consistent with each other. The main downside is that you have to pick one datacenter as the main datacenter at any given time. This can change if the old one goes down or due to business needs, but you can't have two of them as the main datacenter at once for a given dataset.
- jmpman 6y ago
  Know of any commercial offerings with that characteristic?
geertj 6y ago
> It usually starts with NFS, which is a big huge single point of failure. NFS is just the protocol. Whether it's a single point of failure depends on the server-side implementation. In Amazon EFS it is not. (disclaimer: I'm a PM-T on the EFS team)
- ryanmarsh 6y ago
  Hi, my use case for EFS Lambda is burts of small writes. Could you add a perf mode that favors bursts of small writes? We did some perf tests and just couldn’t make it fast enough.
  - tw04 6y ago
    https://www.netapp.com/cloud-services/cloud-volumes-service-aws/ https://www.netapp.com/cloud-services/cloud-volumes-service-... You could try NetApp - it's significantly faster than EFS in my experience.
    - SteveNuts 6y ago
      This solution is at least 40k per year for just the disk controllers, so be prepared for that.
      - Proven 6y ago
        The small 24 drive all flash model (dual controller) is around $20k, based on a quick google search
      - scoot 6y ago
        $100/TB/mo for the cloud version. https://aws.amazon.com/marketplace/pp/B07MF4GHYW https://aws.amazon.com/marketplace/pp/B07MF4GHYW Depending on the use-case, Veritas InfoScale might be a better option. It can aggregate different tiers of storage, and present a multi-node shared-nothing file-system, with options for replication between AZs, regions, or different cloud providers. Priced by core not capacity. https://aws.amazon.com/marketplace/pp/B07CYGD14V https://aws.amazon.com/marketplace/pp/B07CYGD14V
Cixelyn 6y ago
> EdgeFS is one I was looking at recently Do you have any additional info? EdgeFS's github[1] doesn't work; does repo access require a Nexenta sales call? We're also looking into asynchronously replicated FSs, I think built-in caching + tiering is slightly nicer than cron + rsync; would love to know what other solutions you looked into. [1] https://github.com/Nexenta/edgefs https://github.com/Nexenta/edgefs
- rsync 6y ago
  Don't give up too easily on (something like cron+rsync). Things like cron+rsync fail in boring ways. Fancy things fail in fascinating ways.
  - asyncscrum 6y ago
    This guy rsyncs

hanikesn 6y ago

Quobyte claim to have solved the synchronized distributed POSIX filesystem (mod geographically ).

objectivefs 6y ago

ObjectiveFS takes a different approach to high-availability durable file system by using S3 as the storage backend and building a log structured file system on top. This leverages the durability and scalability of S3 while allowing e.g. git cloning/diffing of a large project (even cross region) to remain fast. To compare we have some small file benchmark results at https://objectivefs.com/howto/performance-amazon-efs-vs-objectivefs https://objectivefs.com/howto/performance-amazon-efs-vs-obje... .

fxtentacle 6y ago

> write your app against S3 and call it a day. ... and then later you notice that you need a special SLA and Amazon Redshift to guarantee that a read from S3 will return the same value as the last write. Even S3 is only eventually consistent and especially if a US user uploads images into your US bucket and then you serve the URLs to EU users, you might have loading problems. The correct solution, according to our support contact, is that we wait a second after uploading to S3 to make sure at least the metadata has replicated to the EU before we post the image url. That, or pay for Redshift to tell us how long to wait more precisely. Because contrary to what everyone expects, S3 requests to US buckets might be served by delayed replicas in other countries if the request comes from outside the US.

Galanwe 6y ago
> The correct solution, according to our support contact, is that we wait a second after uploading to S3 I'm shocked that you got this answer. This is definitely not how you are supposed to operate. If you need to ensure the sequantiality of a write followed by a read on S3, the idiomatic way is to enable versioning on your bucket, issue a write, and provide the version ID to whoever need to read after that write. Not only will that transparently ensure that you will not read deprecated data, but it will even ensure that you actually read the result of that particular write, and not any consecutive write that could have happened in between. This pattern is very easy to implement for sequential operations in a single function, like: version = s3.put_object(...) data = s3.get_object(version_id=version, ...) I agree that when you deal with various processes it can become messy. In practice it never bothered me too much though. I prefer having explicit synchronization through versions, rather than having a blocking write waiting for all caches to be synchronized. Also, this should only be necessary if you write to an already existing object. New keys will always trigger a cache invalidation from my understanding.
- fxtentacle 6y ago
  Depending on your write throughput, versioning can be quite expensive. Also, we were already at a throughput where the s3 metadata nodes would sometimes go offline, so I'm not sure putting more work on them would have improved our overall situation.
  - Galanwe 6y ago
    Interesting, do you have more information on how you did monitors these S3 metadata nodes? Never heard of that. Why would an unversioned put be less costly than a versioned put? To me it should be pretty much the same. I would almost suspect unversioned puts are hacked around versioned ones. Out of curiosity what kind of workload did you perform? I have been moving terabytes of files (some very small 1kB files, some huge 150GB files) and never noticed a single blip of change in behavior from S3 (and certainly not anything going "off")

fock 6y ago

wait, S3 can do git now? I guess you are right in nearly everything... but starting with git on gluster and then jumping to selling apples as bananas (e.g. S3) doesn't really make an argument.

the8472 6y ago

> For example, try git cloning a large project onto a Glusterfs mount with >20ms ping between nodes. It's brutal. That may be true but also is due to applications often having very sequential IO patterns even when they don't need to be. I hope we'll get some convenience wrappers around io_uring that make batching of many small IO calls in a synchronous manner simple and easy for cases where you don't want to deal with async runtimes. E.g. bulk_statx() or fsync_many() would be prime candidates for batching.

harshaw 6y ago

EFS attempts to solve this problem, although the speed of light between data centers is still a minor challenge.

slaymaker1907 6y ago

You can minimize the effects of eventual consistency by distinguishing the distant replicas from the same-datacenter ones. Cross datacenter/region might be eventually consistent, but you will only see it rarely since the same-datacenter replicas are fully consistent with each other. The main downside is that you have to pick one datacenter as the main datacenter at any given time. This can change if the old one goes down or due to business needs, but you can't have two of them as the main datacenter at once for a given dataset.

jmpman 6y ago
Know of any commercial offerings with that characteristic?

geertj 6y ago

> It usually starts with NFS, which is a big huge single point of failure. NFS is just the protocol. Whether it's a single point of failure depends on the server-side implementation. In Amazon EFS it is not. (disclaimer: I'm a PM-T on the EFS team)

ryanmarsh 6y ago
Hi, my use case for EFS Lambda is burts of small writes. Could you add a perf mode that favors bursts of small writes? We did some perf tests and just couldn’t make it fast enough.
- tw04 6y ago
  https://www.netapp.com/cloud-services/cloud-volumes-service-aws/ https://www.netapp.com/cloud-services/cloud-volumes-service-... You could try NetApp - it's significantly faster than EFS in my experience.
  - SteveNuts 6y ago
    This solution is at least 40k per year for just the disk controllers, so be prepared for that.
    - Proven 6y ago
      The small 24 drive all flash model (dual controller) is around $20k, based on a quick google search
    - scoot 6y ago
      $100/TB/mo for the cloud version. https://aws.amazon.com/marketplace/pp/B07MF4GHYW https://aws.amazon.com/marketplace/pp/B07MF4GHYW Depending on the use-case, Veritas InfoScale might be a better option. It can aggregate different tiers of storage, and present a multi-node shared-nothing file-system, with options for replication between AZs, regions, or different cloud providers. Priced by core not capacity. https://aws.amazon.com/marketplace/pp/B07CYGD14V https://aws.amazon.com/marketplace/pp/B07CYGD14V

Cixelyn 6y ago

> EdgeFS is one I was looking at recently Do you have any additional info? EdgeFS's github[1] doesn't work; does repo access require a Nexenta sales call? We're also looking into asynchronously replicated FSs, I think built-in caching + tiering is slightly nicer than cron + rsync; would love to know what other solutions you looked into. [1] https://github.com/Nexenta/edgefs https://github.com/Nexenta/edgefs

rsync 6y ago
Don't give up too easily on (something like cron+rsync). Things like cron+rsync fail in boring ways. Fancy things fail in fascinating ways.
- asyncscrum 6y ago
  This guy rsyncs