3 ms·

This won't work for database files, and it won't work for any other application that requires a filesystem to behave consistently over a network either. The pr

This won't work for database files, and it won't work for any other application that requires a filesystem to behave consistently over a network either. The property you are looking for in a distributed filesystem is called strong consistency, or cache coherent (different name, same effect). If a distributed filesystem does genuinely provide that property, it should be ok for SQLite and other applications, because it means the filesystem behaves the same as if it were multiple processes on the same local system accessing it. Most distributed filesystems don't provide that guarantee though. It's a very nice guarantee, but it comes at a complexity cost, usually a performance cost (although there are clever ways to approach the perforance of a non-consistent system), and in particular it means the filesystems can't be used when the network connection is down. There is another property called durability, which you might also care about. That affects whether you get corruption when devices fail or are rebooted suddenly.

3np 6y ago
I actually don't need strong consistency here - I'll only have a single instance of the container accessing the sqlite file at a single point, but they can be running on any host. As long as I can ensure replication is synchronized before process start when a container gets rescheduled we good. It goes over my head a bit exactly why (related to POSIX lock mechanisms?) but sqlite doesn't work reliable on NFS/ceph/gluster/networked filesystems, even when only accessed from a single host, distributed or not. https://sqlite.org/forum/forumpost/4b340b81eb https://sqlite.org/forum/forumpost/4b340b81eb
- slaymaker1907 6y ago
  You could try implementing a small shim over the FS using Posix or Dokan which just goes to the other FS but make sure writes are synced correctly. It would probably be quite slow, but is sounds like you mostly care about having something that works.
- jlokier 6y ago
  > It goes over my head a bit exactly why (related to POSIX lock mechanisms?) but sqlite doesn't work reliable on NFS/ceph/gluster/networked filesystems, even when only accessed from a single host, distributed or not. > https://sqlite.org/forum/forumpost/4b340b81eb https://sqlite.org/forum/forumpost/4b340b81eb I don't see anything in that forum thread which says SQLite is unreliable on sufficiently POSIX-ish network filesystems. Single host or not. The linked post says it will be slower to commit write transactions than is possible using other methods, but I don't see anything there saying it's unreliable. (What looks unreliable to me is Joelmo's proposal to remove the WAL lock for better performance. But the lock is there for a reliability reason, you can't just remove it. To get the boost in performance Joelmo would like requires significantly different techniques.)
pbowyer 6y ago
> Most distributed filesystems don't provide that guarantee though. It's a very nice guarantee, but it comes at a complexity cost, usually a performance cost (although there are clever ways to approach the perforance of a non-consistent system), and in particular it means the filesystems can't be used when the network connection is down. Which distributed file systems do provide this guarantee? I can't think of one, so hopeful to learn something new today.
- 3np 6y ago
  I was curious as well and search turned up chubaofs - it claims to have "strong replication consistency" https://github.com/chubaofs/chubaofs https://github.com/chubaofs/chubaofs
- jlokier 6y ago
  Heh, even Microsoft SMB provides some level of coherency guarantee combined with performance, using its read and write leases. NFSv4 has a similar concept. It's an old idea by now. I actually don't know of any distributed (multi-master, fully replicated) filesystems that are published with this property. I only know it's possible because I designed one that isn't published. The basic principle is similar to CPU MESI caching but with predicate-scopes suited to a filesystem rather than cache lines. (Predicate-scopes are similar to predicate-locks in databases). As you know, multi-core CPU systems remain fast despite sharing memory, incurring significant overhead only for the changes that need to be communicated between cores. Same applies on a network.

3np 6y ago

I actually don't need strong consistency here - I'll only have a single instance of the container accessing the sqlite file at a single point, but they can be running on any host. As long as I can ensure replication is synchronized before process start when a container gets rescheduled we good. It goes over my head a bit exactly why (related to POSIX lock mechanisms?) but sqlite doesn't work reliable on NFS/ceph/gluster/networked filesystems, even when only accessed from a single host, distributed or not. https://sqlite.org/forum/forumpost/4b340b81eb https://sqlite.org/forum/forumpost/4b340b81eb

slaymaker1907 6y ago
You could try implementing a small shim over the FS using Posix or Dokan which just goes to the other FS but make sure writes are synced correctly. It would probably be quite slow, but is sounds like you mostly care about having something that works.
jlokier 6y ago
> It goes over my head a bit exactly why (related to POSIX lock mechanisms?) but sqlite doesn't work reliable on NFS/ceph/gluster/networked filesystems, even when only accessed from a single host, distributed or not. > https://sqlite.org/forum/forumpost/4b340b81eb https://sqlite.org/forum/forumpost/4b340b81eb I don't see anything in that forum thread which says SQLite is unreliable on sufficiently POSIX-ish network filesystems. Single host or not. The linked post says it will be slower to commit write transactions than is possible using other methods, but I don't see anything there saying it's unreliable. (What looks unreliable to me is Joelmo's proposal to remove the WAL lock for better performance. But the lock is there for a reliability reason, you can't just remove it. To get the boost in performance Joelmo would like requires significantly different techniques.)

pbowyer 6y ago

> Most distributed filesystems don't provide that guarantee though. It's a very nice guarantee, but it comes at a complexity cost, usually a performance cost (although there are clever ways to approach the perforance of a non-consistent system), and in particular it means the filesystems can't be used when the network connection is down. Which distributed file systems do provide this guarantee? I can't think of one, so hopeful to learn something new today.

3np 6y ago
I was curious as well and search turned up chubaofs - it claims to have "strong replication consistency" https://github.com/chubaofs/chubaofs https://github.com/chubaofs/chubaofs
jlokier 6y ago
Heh, even Microsoft SMB provides some level of coherency guarantee combined with performance, using its read and write leases. NFSv4 has a similar concept. It's an old idea by now. I actually don't know of any distributed (multi-master, fully replicated) filesystems that are published with this property. I only know it's possible because I designed one that isn't published. The basic principle is similar to CPU MESI caching but with predicate-scopes suited to a filesystem rather than cache lines. (Predicate-scopes are similar to predicate-locks in databases). As you know, multi-core CPU systems remain fast despite sharing memory, incurring significant overhead only for the changes that need to be communicated between cores. Same applies on a network.