3 ms·

Thanks for the question! Mountpoint for Amazon S3 is a FUSE layer that doesn't support full POSIX semantics. For example, you can't use Mountpoint for Amazon S3

Thanks for the question! Mountpoint for Amazon S3 is a FUSE layer that doesn't support full POSIX semantics. For example, you can't use Mountpoint for Amazon S3 for random writes to existing files, appends, or renames. This means that you have to carefully instrument your application to understand whether or not it's compatible with Mountpoint, which can be error-prone. Regatta, on the other hand, provides full POSIX compatibility for the file interface, which means that it works out-of-the-box with all file based applications.

memco 2y ago
Does Regatta require a local disk sized for the entire file to support random writes? One problem I’ve seen is that we have set up instances with a modest local disk but then work with files for which we need to pull the whole file into a local cache modify some parts and then push the full result back into s3. It would be helpful to have a way to work with s3 as though it were posix without having to match the local disk size to the largest file we might need to process.
- huntaub 2y ago
  This is exactly the problem that we solve! You don't need any local disk on your EC2 instance in order to use Regatta or work with data in S3. Our high-speed caching layer plays the role as this local disk for you, so that you can work with data sets that are hundreds of TiBs, even if you only have a 20 GiB EBS volume on your instance.
  - Jayakumark 2y ago
    What is the acceptable latency , if we have to use this outside of Ec2 , lets say mounting S3 from on-prem/GCP/Azure ?
    - huntaub 2y ago
      Well, in my opinion, I want to deliver the lowest latency possible. I expect that we will have Regatta running in GCP and Azure within the next 6 months. I'd love to connect if there's a place on-prem that you're looking to use Regatta. Would you shoot an email to hleath [at] regattastorage.com, and we could chat about what you're looking for?
scottlamb 2y ago
> For example, you can't use Mountpoint for Amazon S3 for random writes to existing files, appends, or renames. Can you support these operations with the expected semantics and performance? If the application makes a one-byte change to a giant file and calls fdatasync, what happens? Do you re-upload the entire file to S3? How do you handle a rename? Applications commonly do this for atomic replacement on POSIX and expect three properties from this operation: * fast. * destination always points to either the original or new afterward (on success or failure); no scenario at which it's lost/truncated. * no extra storage used (on success or failure). Do you guarantee any of those? How? I don't see an obvious way from the S3 HTTP API. Given that POSIX API doesn't support things like arbitrary per-operation deadlines/timeouts, do you think it's suitable as a distributed filesystem API at all? Why?
- huntaub 2y ago
  The tl;dr of this is -- yes. We have a durable caching layer that we use to stage writes before we asynchronously replicate them to S3. This means that we are able to quickly (<1ms) perform operations like single-byte updates and renames and provide strong read-after-write consistency to other file system clients. Once the operation is stored in our durable cache, then we update your S3 bucket to match what the file system expects. This generally takes around a minute, but could take longer depending on the number of S3 operations a file operation translates to (for example, a directory rename requires that CopyObject each object in the directory in S3). I think that the POSIX API is to here to stay (like the S3 API). I agree that it would be better to have timeouts and deadlines, but I don't think that those make it impossible to provide a good distributed file system experience on POSIX (look at Amazon's EFS, Oracle's FSS, Google's FileStore, etc). It just makes the bar for availability higher.

memco 2y ago

Does Regatta require a local disk sized for the entire file to support random writes? One problem I’ve seen is that we have set up instances with a modest local disk but then work with files for which we need to pull the whole file into a local cache modify some parts and then push the full result back into s3. It would be helpful to have a way to work with s3 as though it were posix without having to match the local disk size to the largest file we might need to process.

huntaub 2y ago
This is exactly the problem that we solve! You don't need any local disk on your EC2 instance in order to use Regatta or work with data in S3. Our high-speed caching layer plays the role as this local disk for you, so that you can work with data sets that are hundreds of TiBs, even if you only have a 20 GiB EBS volume on your instance.
- Jayakumark 2y ago
  What is the acceptable latency , if we have to use this outside of Ec2 , lets say mounting S3 from on-prem/GCP/Azure ?
  - huntaub 2y ago
    Well, in my opinion, I want to deliver the lowest latency possible. I expect that we will have Regatta running in GCP and Azure within the next 6 months. I'd love to connect if there's a place on-prem that you're looking to use Regatta. Would you shoot an email to hleath [at] regattastorage.com, and we could chat about what you're looking for?

scottlamb 2y ago

> For example, you can't use Mountpoint for Amazon S3 for random writes to existing files, appends, or renames. Can you support these operations with the expected semantics and performance? If the application makes a one-byte change to a giant file and calls fdatasync, what happens? Do you re-upload the entire file to S3? How do you handle a rename? Applications commonly do this for atomic replacement on POSIX and expect three properties from this operation: * fast. * destination always points to either the original or new afterward (on success or failure); no scenario at which it's lost/truncated. * no extra storage used (on success or failure). Do you guarantee any of those? How? I don't see an obvious way from the S3 HTTP API. Given that POSIX API doesn't support things like arbitrary per-operation deadlines/timeouts, do you think it's suitable as a distributed filesystem API at all? Why?

huntaub 2y ago
The tl;dr of this is -- yes. We have a durable caching layer that we use to stage writes before we asynchronously replicate them to S3. This means that we are able to quickly (<1ms) perform operations like single-byte updates and renames and provide strong read-after-write consistency to other file system clients. Once the operation is stored in our durable cache, then we update your S3 bucket to match what the file system expects. This generally takes around a minute, but could take longer depending on the number of S3 operations a file operation translates to (for example, a directory rename requires that CopyObject each object in the directory in S3). I think that the POSIX API is to here to stay (like the S3 API). I agree that it would be better to have timeouts and deadlines, but I don't think that those make it impossible to provide a good distributed file system experience on POSIX (look at Amazon's EFS, Oracle's FSS, Google's FileStore, etc). It just makes the bar for availability higher.