3 ms·

I'm using a distributed filesystem for persisted files (config, assets and resources, etc - anything that'd need to be in a docker volume and persisted between

by 3np 6y ago

I'm using a distributed filesystem for persisted files (config, assets and resources, etc - anything that'd need to be in a docker volume and persisted between restarts) in my container workloads. This mostly works fine, except for sqlite applications that insist on using WAL - this completely breaks and results in corrupted database files. Maybe this could be an alternative for those!

teraflop 6y ago
"Near-realtime" isn't good enough for SQLite. It has a strong dependency on the consistency guarantees provided by a POSIX-like filesystem. The approach described in this page does make some attempt to provide consistency. The master generates a single stream of file updates, and all of the replicas consistently apply those changes in the same order. But rsync is not guaranteed to observe changes to files on the master in the same order that they were written. In particular, SQLite (by default) uses a rollback journal to recover from failures. While a transaction is in process and some parts of the database might have been partially updated, the journal stores the old contents of the modified pages. Since the old data is fsync'ed to the journal before the new data is written to the database file, the data on disk at any moment in time is recoverable. But there's no clean way for rsync to read an atomic snapshot of both the database file and the journal at the same instant in time. If it reads them at different times, a database change might be partially applied or partially rolled back, which has a high likelihood of corrupting the database.
jlokier 6y ago
This won't work for database files, and it won't work for any other application that requires a filesystem to behave consistently over a network either. The property you are looking for in a distributed filesystem is called strong consistency, or cache coherent (different name, same effect). If a distributed filesystem does genuinely provide that property, it should be ok for SQLite and other applications, because it means the filesystem behaves the same as if it were multiple processes on the same local system accessing it. Most distributed filesystems don't provide that guarantee though. It's a very nice guarantee, but it comes at a complexity cost, usually a performance cost (although there are clever ways to approach the perforance of a non-consistent system), and in particular it means the filesystems can't be used when the network connection is down. There is another property called durability, which you might also care about. That affects whether you get corruption when devices fail or are rebooted suddenly.
- 3np 6y ago
  I actually don't need strong consistency here - I'll only have a single instance of the container accessing the sqlite file at a single point, but they can be running on any host. As long as I can ensure replication is synchronized before process start when a container gets rescheduled we good. It goes over my head a bit exactly why (related to POSIX lock mechanisms?) but sqlite doesn't work reliable on NFS/ceph/gluster/networked filesystems, even when only accessed from a single host, distributed or not. https://sqlite.org/forum/forumpost/4b340b81eb https://sqlite.org/forum/forumpost/4b340b81eb
  - slaymaker1907 6y ago
    You could try implementing a small shim over the FS using Posix or Dokan which just goes to the other FS but make sure writes are synced correctly. It would probably be quite slow, but is sounds like you mostly care about having something that works.
  - jlokier 6y ago
    > It goes over my head a bit exactly why (related to POSIX lock mechanisms?) but sqlite doesn't work reliable on NFS/ceph/gluster/networked filesystems, even when only accessed from a single host, distributed or not. > https://sqlite.org/forum/forumpost/4b340b81eb https://sqlite.org/forum/forumpost/4b340b81eb I don't see anything in that forum thread which says SQLite is unreliable on sufficiently POSIX-ish network filesystems. Single host or not. The linked post says it will be slower to commit write transactions than is possible using other methods, but I don't see anything there saying it's unreliable. (What looks unreliable to me is Joelmo's proposal to remove the WAL lock for better performance. But the lock is there for a reliability reason, you can't just remove it. To get the boost in performance Joelmo would like requires significantly different techniques.)
- pbowyer 6y ago
  > Most distributed filesystems don't provide that guarantee though. It's a very nice guarantee, but it comes at a complexity cost, usually a performance cost (although there are clever ways to approach the perforance of a non-consistent system), and in particular it means the filesystems can't be used when the network connection is down. Which distributed file systems do provide this guarantee? I can't think of one, so hopeful to learn something new today.

by 3np 6y ago

teraflop 6y ago

"Near-realtime" isn't good enough for SQLite. It has a strong dependency on the consistency guarantees provided by a POSIX-like filesystem. The approach described in this page does make some attempt to provide consistency. The master generates a single stream of file updates, and all of the replicas consistently apply those changes in the same order. But rsync is not guaranteed to observe changes to files on the master in the same order that they were written. In particular, SQLite (by default) uses a rollback journal to recover from failures. While a transaction is in process and some parts of the database might have been partially updated, the journal stores the old contents of the modified pages. Since the old data is fsync'ed to the journal before the new data is written to the database file, the data on disk at any moment in time is recoverable. But there's no clean way for rsync to read an atomic snapshot of both the database file and the journal at the same instant in time. If it reads them at different times, a database change might be partially applied or partially rolled back, which has a high likelihood of corrupting the database.

jlokier 6y ago

This won't work for database files, and it won't work for any other application that requires a filesystem to behave consistently over a network either. The property you are looking for in a distributed filesystem is called strong consistency, or cache coherent (different name, same effect). If a distributed filesystem does genuinely provide that property, it should be ok for SQLite and other applications, because it means the filesystem behaves the same as if it were multiple processes on the same local system accessing it. Most distributed filesystems don't provide that guarantee though. It's a very nice guarantee, but it comes at a complexity cost, usually a performance cost (although there are clever ways to approach the perforance of a non-consistent system), and in particular it means the filesystems can't be used when the network connection is down. There is another property called durability, which you might also care about. That affects whether you get corruption when devices fail or are rebooted suddenly.

3np 6y ago
I actually don't need strong consistency here - I'll only have a single instance of the container accessing the sqlite file at a single point, but they can be running on any host. As long as I can ensure replication is synchronized before process start when a container gets rescheduled we good. It goes over my head a bit exactly why (related to POSIX lock mechanisms?) but sqlite doesn't work reliable on NFS/ceph/gluster/networked filesystems, even when only accessed from a single host, distributed or not. https://sqlite.org/forum/forumpost/4b340b81eb https://sqlite.org/forum/forumpost/4b340b81eb
- slaymaker1907 6y ago
  You could try implementing a small shim over the FS using Posix or Dokan which just goes to the other FS but make sure writes are synced correctly. It would probably be quite slow, but is sounds like you mostly care about having something that works.
- jlokier 6y ago
  > It goes over my head a bit exactly why (related to POSIX lock mechanisms?) but sqlite doesn't work reliable on NFS/ceph/gluster/networked filesystems, even when only accessed from a single host, distributed or not. > https://sqlite.org/forum/forumpost/4b340b81eb https://sqlite.org/forum/forumpost/4b340b81eb I don't see anything in that forum thread which says SQLite is unreliable on sufficiently POSIX-ish network filesystems. Single host or not. The linked post says it will be slower to commit write transactions than is possible using other methods, but I don't see anything there saying it's unreliable. (What looks unreliable to me is Joelmo's proposal to remove the WAL lock for better performance. But the lock is there for a reliability reason, you can't just remove it. To get the boost in performance Joelmo would like requires significantly different techniques.)
pbowyer 6y ago
> Most distributed filesystems don't provide that guarantee though. It's a very nice guarantee, but it comes at a complexity cost, usually a performance cost (although there are clever ways to approach the perforance of a non-consistent system), and in particular it means the filesystems can't be used when the network connection is down. Which distributed file systems do provide this guarantee? I can't think of one, so hopeful to learn something new today.