4 ms·

I'm guessing any filesystem would work, when it's only writing on one server. "On the destination instance, the EBS volume is mounted with RO (Read-Only) permi

I'm guessing any filesystem would work, when it's only writing on one server. "On the destination instance, the EBS volume is mounted with RO (Read-Only) permissions. But the caveat to this is — we do not use a clustered file system on top of EBS. So the destination side can’t see the changes made to EBS volume instantaneously. "

mlyle 5y ago
Nah. Because not every intermediate state of a filesystem is self-consistent, and because caching can "tear" updates. So Amazon's multi-attach documentation says: > Standard file systems, such as XFS and EXT4, are not designed to be accessed simultaneously by multiple servers, such as EC2 instances. Using Multi-Attach with a standard file system can result in data corruption or loss, so this is not safe for production workloads. You can use a clustered file system to ensure data resiliency and reliability for production workloads. > Multi-Attach enabled volumes do not support I/O fencing. I/O fencing protocols control write access in a shared storage environment to maintain data consistency. Your applications must provide write ordering for the attached instances to maintain data consistency. Or from the Netflix document: > It ignores files with partial file system metadata. Cache populators can see incomplete file system metadata due to the way it mounts the filesystem, with “No Recovery” option. It proceeds with complete files, leaving incomplete files to be processed by a subsequent iteration. This is explained further in step 7. AKA you get random read errors, etc, and need to cope. Behavior of something like this can be expected to be very kernel version and filesystem dependent, even with the most defensive application access strategy.
- a-dub 5y ago
  might be kinda fun to build a "ringbufferfs" that uses multi attach ebs volumes as slow shared memory for pushing bits around via the ebs side channel. probably cheaper and more reliable too than provisioning a full size volume and relying on implementation quirks in existing single host filesystems and ebs.
- eloff 5y ago
  It sounds like one would have to code very defensively to make that work safely.
- profile53 5y ago
  > Nah. Because not every intermediate state of a filesystem is self-consistent, and because caching can "tear" updates. So Amazon's multi-attach documentation says: Pretty much every modern file system is tolerant to computer crashes via journaling. Mounting an in-use FS as read only is effectively the same as mounting a file system after a computer crash. Not self consistent, but close enough it can be repaired in memory (i.e. without touching the disk) via replaying the journal.
  - profile53 5y ago
    Oh, I just realized I misunderstood your point. Because the data on disk gets updated while the in memory structures on the read only side don’t, you’ll get garbage and have to swallow random errors. I bet it only works in very specific write-once type environments like Netflix, because otherwise updating a file would lead to reading garbage on the other side.
  - mlyle 5y ago
    > Pretty much every modern file system is tolerant to computer crashes via journaling. Mounting an in-use FS as read only is effectively the same as mounting a file system after a computer crash. Not self consistent, but close enough it can be repaired in memory (i.e. without touching the disk) via replaying the journal. Filesystems generally don't replay the journal to memory, but instead replay it to disk as part of going read-write. Conventional replay of the journal would make these issues even worse.
    - profile53 5y ago
      You're right. I knew ext4 is crash consistent even in a read only mode, but after looking it up, ext4 will ignore the read only mount request, replay the journal to disk, then mount in read only mode. So, it doesn't just replay the journal to an in memory structure when mounted read only.
      - mlyle 5y ago
        Yah, and replaying the journal while there are other writers is dangerous: it can undo their changes. So you either make the device fully read-only and make your filesystem cope with not being able to do the journal applies.. or you use something like lvm where you can just apply the journal to a snapshot / set of in-memory deltas. But it's worse than this and the above doesn't solve the problems. Say you've mounted already, the filesystem is changing under you, and assume no cache for simplicity (caching makes this even worse): For instance, say update A truncates a file and frees up some storage. update B allocates this free space to another file. Update C writes to it. A different system starts a read, and finds the metadata for the first file. Then it hesitates, and later reads the data from update C in the wrong file. Most filesystems are not written with this use case in mind-- mount readonly while someone else updates. It's something that I've tried doing with varying degrees of success in the past (sometimes perfect, sometimes with strangeness at the application level, sometimes with kernel panics).

mlyle 5y ago

Nah. Because not every intermediate state of a filesystem is self-consistent, and because caching can "tear" updates. So Amazon's multi-attach documentation says: > Standard file systems, such as XFS and EXT4, are not designed to be accessed simultaneously by multiple servers, such as EC2 instances. Using Multi-Attach with a standard file system can result in data corruption or loss, so this is not safe for production workloads. You can use a clustered file system to ensure data resiliency and reliability for production workloads. > Multi-Attach enabled volumes do not support I/O fencing. I/O fencing protocols control write access in a shared storage environment to maintain data consistency. Your applications must provide write ordering for the attached instances to maintain data consistency. Or from the Netflix document: > It ignores files with partial file system metadata. Cache populators can see incomplete file system metadata due to the way it mounts the filesystem, with “No Recovery” option. It proceeds with complete files, leaving incomplete files to be processed by a subsequent iteration. This is explained further in step 7. AKA you get random read errors, etc, and need to cope. Behavior of something like this can be expected to be very kernel version and filesystem dependent, even with the most defensive application access strategy.

a-dub 5y ago
might be kinda fun to build a "ringbufferfs" that uses multi attach ebs volumes as slow shared memory for pushing bits around via the ebs side channel. probably cheaper and more reliable too than provisioning a full size volume and relying on implementation quirks in existing single host filesystems and ebs.
eloff 5y ago
It sounds like one would have to code very defensively to make that work safely.
profile53 5y ago
> Nah. Because not every intermediate state of a filesystem is self-consistent, and because caching can "tear" updates. So Amazon's multi-attach documentation says: Pretty much every modern file system is tolerant to computer crashes via journaling. Mounting an in-use FS as read only is effectively the same as mounting a file system after a computer crash. Not self consistent, but close enough it can be repaired in memory (i.e. without touching the disk) via replaying the journal.
- profile53 5y ago
  Oh, I just realized I misunderstood your point. Because the data on disk gets updated while the in memory structures on the read only side don’t, you’ll get garbage and have to swallow random errors. I bet it only works in very specific write-once type environments like Netflix, because otherwise updating a file would lead to reading garbage on the other side.
- mlyle 5y ago
  > Pretty much every modern file system is tolerant to computer crashes via journaling. Mounting an in-use FS as read only is effectively the same as mounting a file system after a computer crash. Not self consistent, but close enough it can be repaired in memory (i.e. without touching the disk) via replaying the journal. Filesystems generally don't replay the journal to memory, but instead replay it to disk as part of going read-write. Conventional replay of the journal would make these issues even worse.
  - profile53 5y ago
    You're right. I knew ext4 is crash consistent even in a read only mode, but after looking it up, ext4 will ignore the read only mount request, replay the journal to disk, then mount in read only mode. So, it doesn't just replay the journal to an in memory structure when mounted read only.
    - mlyle 5y ago
      Yah, and replaying the journal while there are other writers is dangerous: it can undo their changes. So you either make the device fully read-only and make your filesystem cope with not being able to do the journal applies.. or you use something like lvm where you can just apply the journal to a snapshot / set of in-memory deltas. But it's worse than this and the above doesn't solve the problems. Say you've mounted already, the filesystem is changing under you, and assume no cache for simplicity (caching makes this even worse): For instance, say update A truncates a file and frees up some storage. update B allocates this free space to another file. Update C writes to it. A different system starts a read, and finds the metadata for the first file. Then it hesitates, and later reads the data from update C in the wrong file. Most filesystems are not written with this use case in mind-- mount readonly while someone else updates. It's something that I've tried doing with varying degrees of success in the past (sometimes perfect, sometimes with strangeness at the application level, sometimes with kernel panics).