A few observations here o managing state is hard. Kubernetes makes it look simple because we have been moving the state from apps to databases/messaging queues

A few observations here o managing state is hard. Kubernetes makes it look simple because we have been moving the state from apps to databases/messaging queues o storing data durably at scale, with speed and consistently is a CAP problem o distributed block storage is slow, complex and resource hungry o distributed storage with metadata almost always has a metadata speed problem (again CAP) o managing your own high performance storage system is bankruptingly hard Ceph/rook is almost certainly not the answer. Ceph has been pushed with openstack, which is a horridly mess of complexity. Ceph is slow hard to mange and eats resources. if you are running k8s on real steel, then use a SAN. if you're on AWS/google use the block primitives provided. Firstly, there is no general storage backend that is a good fit for all workloads. Some things need low latency (either direct attached, or local SAN) some can cope with single instances of small EBS volumes. Some need shared block sorage, some need shared posix. With AWS, there is no share block storage publicly available. Mapping volumes to random containers is pretty simple. Failing that there is EFS, which despite terrible metadata performance, kicks out a boat load of data. If you app can store all its state in one large file, this might be for you. Google has the same, although I've not tried their new EFS/NFS service.