4 ms·

Has anyone used ceph in production? If so, are there are any advanatages of using Ceph over Aws S3? It used to be pretty popular during openstack days. Lately,

Has anyone used ceph in production? If so, are there are any advanatages of using Ceph over Aws S3? It used to be pretty popular during openstack days. Lately, I havent herd much about it.

raincom 6y ago
It's advantage is being on-prem. So, it goes with the on-prem k8s like Openshift of Redhat/IBM. Who are the likely customers for on-prem solutions? Insurance, banks, healthcare, etc. That's where you see these solutions.
- ghaff 6y ago
  You can also run those (OpenShift/Ceph/Rook/etc.) both on-prem and on various public clouds which gives you a degree of portability and compatibility across platforms.
  - raincom 6y ago
    If something works on-prem, it also works on clouds: after all, just get compute, and run. However, that's not how Redhat sells Openshift.
    - ghaff 6y ago
      >However, that's not how Redhat sells Openshift. Really? There are also managed offerings on various clouds but, to the best of my knowledge (I work at Red Hat but not specifically on OpenShift), the fact that you can run it yourself on different clouds is a selling point if that's what you want to do.
    - yipbub 6y ago
      Running the same large scale cluster on Openshift (with a managed lifecycle) on a different cloud, is just a matter of: `openshift-installer create install-config --dir=install_1` edit the deployment size, and any small vendor specific tweaks like instance size ~5 lines `openshift-installer create cluster --dir=install_1` You'll get a different StorageClass from which you can make Persistent Volume Claims from, easy scaling of nodes using `kubectl scale machineset` or an autoscaler. Even with bare metal setup, which requires setting up some DNS wildcards and DHCP, everything down to the layers(I think they're layers) of the OS filesystem can be managed via a K8s custom resource (backed by ignition). You can create network attachment definitions for multiple nics, you can specify the contents of a config file via a bunch of MachineConfigs. And you can just not do all of this until you need to, you can go straight ahead to creating Deployments, and load balanced services. With KubeVirt, you can use k8s resources and virtctl to manage virtual machines. Vendor specificness is highly minimized by a management layer.
  - wmf 6y ago
    Running your own storage in public clouds doesn't make much sense though. Native object storage is generally a lot better than running your own Ceph/Minio, native block storage is better than running your own Ceph, etc.
antongribok 6y ago
I run ~100-200 PB of Ceph object and block storage. I've run Ceph in production at my current and previous employers, both Fortune 100 companies since 2013. Ceph is used at Fidelity, Bloomberg, OVH, Digital Ocean, CERN, Walmart, and some Canadian and European universities, and last but not least at DreamHost.
manigandham 6y ago
Digital Ocean Spaces is an S3 alternative that is powered by Ceph.

Has anyone used ceph in production? If so, are there are any advanatages of using Ceph over Aws S3? It used to be pretty popular during openstack days. Lately, I havent herd much about it.

raincom 6y ago

It's advantage is being on-prem. So, it goes with the on-prem k8s like Openshift of Redhat/IBM. Who are the likely customers for on-prem solutions? Insurance, banks, healthcare, etc. That's where you see these solutions.

ghaff 6y ago
You can also run those (OpenShift/Ceph/Rook/etc.) both on-prem and on various public clouds which gives you a degree of portability and compatibility across platforms.
- raincom 6y ago
  If something works on-prem, it also works on clouds: after all, just get compute, and run. However, that's not how Redhat sells Openshift.
  - ghaff 6y ago
    >However, that's not how Redhat sells Openshift. Really? There are also managed offerings on various clouds but, to the best of my knowledge (I work at Red Hat but not specifically on OpenShift), the fact that you can run it yourself on different clouds is a selling point if that's what you want to do.
  - yipbub 6y ago
    Running the same large scale cluster on Openshift (with a managed lifecycle) on a different cloud, is just a matter of: `openshift-installer create install-config --dir=install_1` edit the deployment size, and any small vendor specific tweaks like instance size ~5 lines `openshift-installer create cluster --dir=install_1` You'll get a different StorageClass from which you can make Persistent Volume Claims from, easy scaling of nodes using `kubectl scale machineset` or an autoscaler. Even with bare metal setup, which requires setting up some DNS wildcards and DHCP, everything down to the layers(I think they're layers) of the OS filesystem can be managed via a K8s custom resource (backed by ignition). You can create network attachment definitions for multiple nics, you can specify the contents of a config file via a bunch of MachineConfigs. And you can just not do all of this until you need to, you can go straight ahead to creating Deployments, and load balanced services. With KubeVirt, you can use k8s resources and virtctl to manage virtual machines. Vendor specificness is highly minimized by a management layer.
- wmf 6y ago
  Running your own storage in public clouds doesn't make much sense though. Native object storage is generally a lot better than running your own Ceph/Minio, native block storage is better than running your own Ceph, etc.

antongribok 6y ago

I run ~100-200 PB of Ceph object and block storage. I've run Ceph in production at my current and previous employers, both Fortune 100 companies since 2013. Ceph is used at Fidelity, Bloomberg, OVH, Digital Ocean, CERN, Walmart, and some Canadian and European universities, and last but not least at DreamHost.

manigandham 6y ago

Digital Ocean Spaces is an S3 alternative that is powered by Ceph.