I use CephFS. Blob storage is easier than POSIX file systems: You have server-client state. The concept of opened files, directories, and their states. Locks.

I use CephFS. Blob storage is easier than POSIX file systems: You have server-client state. The concept of opened files, directories, and their states. Locks. The ability for multiple writers to write to the same file while still providing POSIX guarantees. All of those need to correctly handle failure of both the client and the server. CephFS implements that with a Metadata server that has lots of logica and needs plenty of RAM. A distributed file system like CephFS is more convenient than S3 in multiple ways, and I agree it's preferable for most use cases. But it's undoubtedly more complex to build.