3 ms·

Is it 'Posix Compatible' or 'Posix' aka 'Posix compliant'? It's incredibly hard to make a distributed posix compatible filesystem since you run into CAP. I be

Is it 'Posix Compatible' or 'Posix' aka 'Posix compliant'? It's incredibly hard to make a distributed posix compatible filesystem since you run into CAP. I believe (but am not certain) you are caching locally in violation of Posix or you are signing up for arbitrarily long stalls and a ton of latency on every read/write. (I'm not certain because I'm not sure what Posix specifies wrt stale reads and other cache consistency requirements between sync's) It would be interesting to hear what the tradeoffs are here, but assuming they are explicit and can be designed around this seems very useful.

jeffbee 4y ago
It is not posix anything. It provides a compatibility layer that makes open, close, read, and write work but other than that does not provide the type of features that would allow you to deliver mail on it with qmail or whatever. It is incredibly misleading to advertise it that way. As you say there is no free lunch with distributed filesystems. Application programmers have to program their way around the fact that something like posix atomic writes with multiple writers is never going to work, and that the only way to get reasonable efficiency out of the thing is to defer work until the file is closed.
- remram 4y ago
  What specifically is missing?
daviesliu 4y ago
Agreed, it's very hard, that's why GFS and HDFS had give up some parts of POSIX compatibility. Per CAP, it's addressed by different meta engines (CP system, Redis, MySQL, TiKV) and also different object stores (AP system). When the meta engine is not available, the operation to JuiceFS will be blocked for a while and finally it returns EIO. When object store returns 404 (object not found), which means it's not consistent with the meta engine, it will be retried for a while, may return EIO if it's not recovered. The file format is carefully designed to workaround the consistency issue from object store and local cache. Any part of data is written into object store and local cache with unique ID, so you will not go stale data once the metadata is correct [1]. Within a mount point, JuiceFS provides read-after-write consistency. Across clusters, JuiceFS provides open-after-close consistency, which should be enough for most of the applications, also provide good balance between consistency and performance. [1] https://juicefs.com/docs/community/architecture/#how-juicefs-store-files https://juicefs.com/docs/community/architecture/#how-juicefs...

jeffbee 4y ago

It is not posix anything. It provides a compatibility layer that makes open, close, read, and write work but other than that does not provide the type of features that would allow you to deliver mail on it with qmail or whatever. It is incredibly misleading to advertise it that way. As you say there is no free lunch with distributed filesystems. Application programmers have to program their way around the fact that something like posix atomic writes with multiple writers is never going to work, and that the only way to get reasonable efficiency out of the thing is to defer work until the file is closed.

remram 4y ago
What specifically is missing?

daviesliu 4y ago

Agreed, it's very hard, that's why GFS and HDFS had give up some parts of POSIX compatibility. Per CAP, it's addressed by different meta engines (CP system, Redis, MySQL, TiKV) and also different object stores (AP system). When the meta engine is not available, the operation to JuiceFS will be blocked for a while and finally it returns EIO. When object store returns 404 (object not found), which means it's not consistent with the meta engine, it will be retried for a while, may return EIO if it's not recovered. The file format is carefully designed to workaround the consistency issue from object store and local cache. Any part of data is written into object store and local cache with unique ID, so you will not go stale data once the metadata is correct [1]. Within a mount point, JuiceFS provides read-after-write consistency. Across clusters, JuiceFS provides open-after-close consistency, which should be enough for most of the applications, also provide good balance between consistency and performance. [1] https://juicefs.com/docs/community/architecture/#how-juicefs-store-files https://juicefs.com/docs/community/architecture/#how-juicefs...