4 ms·

One important aspect that should be mentioned is that OpenEBS is essentially a block storage system (think virtual hard drives, which need to be formatted with

by fh973 6y ago

One important aspect that should be mentioned is that OpenEBS is essentially a block storage system (think virtual hard drives, which need to be formatted with dfs/ext4) and by its nature can't expose the same files to multiple pods. Ceph(FS) and other distributed file systems allow you to access your data from any number of machines. Also they provide faster and more robust failover as they don't have to unplug the virtual hard drive and repair the local file system.

antonvs 6y ago
> by its nature can't expose the same files to multiple pods. I don't know OpenEBS, but this isn't true of block storage systems in general. Both AWS and Google allow mounting virtual disks to multiple pods or nodes if all but one are mounted read-only. Even multi-node writing is possible - see https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/ebs-volumes-multi.html https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/ebs-volu... : > Amazon EBS Multi-Attach enables you to attach a single Provisioned IOPS SSD (io1) volume to up to 16 Nitro-based instances that are in the same Availability Zone. You can attach multiple Multi-Attach enabled volumes to an instance or set of instances. Each instance to which the volume is attached has full read and write permission to the shared volume. Google offers this too: https://cloud.google.com/sdk/gcloud/reference/beta/compute/disks/create#--multi-writer https://cloud.google.com/sdk/gcloud/reference/beta/compute/d... Of course this does have constraints compare to a true distributed filesystem, which the AWS page describes.
- fh973 6y ago
  You are able to attach the block devices, but local file systems do not support being mounted on more than one host. For that you need to use a cluster file system, but that's a different beast.
  - antonvs 6y ago
    The multi-reader scenario definitely supports multiple readers of the same file system - we use it both in and out of Kubernetes on GCP. Kubernetes has specific support for this with its ReadOnlyMany access mode: https://kubernetes.io/docs/concepts/storage/persistent-volumes/#access-modes https://kubernetes.io/docs/concepts/storage/persistent-volum... However, I do see on that page that AWS EBS doesn't support this mode, so perhaps that's what you're thinking of.
    - hardwaresofton 6y ago
      Would you mind going into detail on your setup? What do you use to do this outside and inside GCP?
      - antonvs 6y ago
        Both scenarios I was referring to are on GCP, but one is on k8s (GKE) and one uses ordinary GCP VMs. In both cases we're using the multi-reader mounting feature of GCP persistent disks, all that changes is how that's specified. Outside GCP, we do have an on-premise product that can deploy on any k8s cluster. In that case, the default is for the customer to provide an NFS mount for the data in question. We use the same k8s ReadOnlyMany access type for that.

by fh973 6y ago

antonvs 6y ago

> by its nature can't expose the same files to multiple pods. I don't know OpenEBS, but this isn't true of block storage systems in general. Both AWS and Google allow mounting virtual disks to multiple pods or nodes if all but one are mounted read-only. Even multi-node writing is possible - see https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/ebs-volumes-multi.html https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/ebs-volu... : > Amazon EBS Multi-Attach enables you to attach a single Provisioned IOPS SSD (io1) volume to up to 16 Nitro-based instances that are in the same Availability Zone. You can attach multiple Multi-Attach enabled volumes to an instance or set of instances. Each instance to which the volume is attached has full read and write permission to the shared volume. Google offers this too: https://cloud.google.com/sdk/gcloud/reference/beta/compute/disks/create#--multi-writer https://cloud.google.com/sdk/gcloud/reference/beta/compute/d... Of course this does have constraints compare to a true distributed filesystem, which the AWS page describes.

fh973 6y ago
You are able to attach the block devices, but local file systems do not support being mounted on more than one host. For that you need to use a cluster file system, but that's a different beast.
- antonvs 6y ago
  The multi-reader scenario definitely supports multiple readers of the same file system - we use it both in and out of Kubernetes on GCP. Kubernetes has specific support for this with its ReadOnlyMany access mode: https://kubernetes.io/docs/concepts/storage/persistent-volumes/#access-modes https://kubernetes.io/docs/concepts/storage/persistent-volum... However, I do see on that page that AWS EBS doesn't support this mode, so perhaps that's what you're thinking of.
  - hardwaresofton 6y ago
    Would you mind going into detail on your setup? What do you use to do this outside and inside GCP?
    - antonvs 6y ago
      Both scenarios I was referring to are on GCP, but one is on k8s (GKE) and one uses ordinary GCP VMs. In both cases we're using the multi-reader mounting feature of GCP persistent disks, all that changes is how that's specified. Outside GCP, we do have an on-premise product that can deploy on any k8s cluster. In that case, the default is for the customer to provide an NFS mount for the data in question. We use the same k8s ReadOnlyMany access type for that.