4 ms·

Does this cause problems in practice? I've never used Ceph in production, but it doesn't seem unreasonable to expect the sysadmin to keep clock skew in check.

by walrus 12y ago

cmurf 12y ago
Ceph isn't ready for production. GlusterFS is, and it also behaves more like a file system, while Ceph is more object storage. They serve different use cases but GlusterFS is more general purpose I think. No MDS required, scales fairly linearly. They're using it at Facebook with XFS and Btrfs underneath depending on workload. At home you can build little ARM gluster clusters, so it kinda even scales down! http://events.linuxfoundation.org/sites/events/files/slides/Vault%20CephFS%20fsck.pdf http://events.linuxfoundation.org/sites/events/files/slides/...
- nigwil_ 12y ago
  "Ceph isn't ready for production." - this is not the case. Ceph object-store known as RADOS is production ready and supported by Red Hat [1], in addition the block-device service known as RBD which is an access method to RADOS is also production ready and supported. RBD provides many SAN like features like snapshots, cloning, and soon mirroring. The CephFS (POSIX filesystem access method) is not yet a supported product but is used by several sites in production. Ceph also provides S3/Swift object-store gateway (radosgw) so existing applications can access the object store with a compatible API. Ceph provides unified storage: providing object, block and filesystem access from a single cluster, so Ceph is the more general purpose technology. Ceph MDS provides metadata service for the CephFS POSIX-filesystem but is not needed if you're not using that feature. For development and testing Ceph will run on a single VM, or more realistically on 3 small VMs so the cluster characteristics can be explored. [1] alternatively you can buy a production grade fully supported appliance product from Fujitsu based on Ceph known as ETERNUS CD10000. For most deployments Ceph runs well on commodity hardware and the Ceph community provides excellent support too.
- mdnormy 12y ago
  > Ceph isn't ready for production DreamHost would beg to differ. They've been using it for their DreamObject[1] for few years. Granted that they have have better visibility earlier because Sage Weil, but I've seen Ceph deployment for production in the wild since mid-2014. [1] https://www.dreamhost.com/cloud/storage/ https://www.dreamhost.com/cloud/storage/
- cmurf 12y ago
  The supplied link suggests it's not production ready, but it's the wrong link. This is the correct link. John Spray, a senior software engineer at Red Hat, "Some people are already using it in production; we're terrified of this. It's really not ready yet." http://www.zdnet.com/article/linux-storage-futures/ http://www.zdnet.com/article/linux-storage-futures/ EDIT: And I'm not suggestion it isn't badass, just that the idea Ceph as a first suggestion for general purpose RAID replacement is probably not a good idea just yet, and that there's an alternative.
  - mtanski 12y ago
    cmurf. What the article and John is talking about is CephFS (which we do use). As you can see that is in the heading of the quote that you took. What John is not talking about is the Object Store (S3 compatible) or RBD (rados block device, EBS like block storage). There's countless people using both Object Store and RBD in production today. Both of these are the most deployed solutions for OpenStack (for storage or object storage). For example the Bloomberg folks use in production. We use CephFS, and contributed to it earlier, and helped find a lot of bugs in the kernel and the MDS. Last time we uncounted a bug was over a year ago. We currently only use one MDS with a hot spare (backup node actively replaying the log). What's really missing today from CephFS is the fsck and restore utilities and I believe people are working on them as we speak.
duaneb 12y ago
Clock skew is notoriously difficult to actually fix, especially with a network partition.

Does this cause problems in practice? I've never used Ceph in production, but it doesn't seem unreasonable to expect the sysadmin to keep clock skew in check.

by walrus 12y ago

Does this cause problems in practice? I've never used Ceph in production, but it doesn't seem unreasonable to expect the sysadmin to keep clock skew in check.

cmurf 12y ago

Ceph isn't ready for production. GlusterFS is, and it also behaves more like a file system, while Ceph is more object storage. They serve different use cases but GlusterFS is more general purpose I think. No MDS required, scales fairly linearly. They're using it at Facebook with XFS and Btrfs underneath depending on workload. At home you can build little ARM gluster clusters, so it kinda even scales down! http://events.linuxfoundation.org/sites/events/files/slides/Vault%20CephFS%20fsck.pdf http://events.linuxfoundation.org/sites/events/files/slides/...

nigwil_ 12y ago
"Ceph isn't ready for production." - this is not the case. Ceph object-store known as RADOS is production ready and supported by Red Hat [1], in addition the block-device service known as RBD which is an access method to RADOS is also production ready and supported. RBD provides many SAN like features like snapshots, cloning, and soon mirroring. The CephFS (POSIX filesystem access method) is not yet a supported product but is used by several sites in production. Ceph also provides S3/Swift object-store gateway (radosgw) so existing applications can access the object store with a compatible API. Ceph provides unified storage: providing object, block and filesystem access from a single cluster, so Ceph is the more general purpose technology. Ceph MDS provides metadata service for the CephFS POSIX-filesystem but is not needed if you're not using that feature. For development and testing Ceph will run on a single VM, or more realistically on 3 small VMs so the cluster characteristics can be explored. [1] alternatively you can buy a production grade fully supported appliance product from Fujitsu based on Ceph known as ETERNUS CD10000. For most deployments Ceph runs well on commodity hardware and the Ceph community provides excellent support too.
mdnormy 12y ago
> Ceph isn't ready for production DreamHost would beg to differ. They've been using it for their DreamObject[1] for few years. Granted that they have have better visibility earlier because Sage Weil, but I've seen Ceph deployment for production in the wild since mid-2014. [1] https://www.dreamhost.com/cloud/storage/ https://www.dreamhost.com/cloud/storage/
cmurf 12y ago
The supplied link suggests it's not production ready, but it's the wrong link. This is the correct link. John Spray, a senior software engineer at Red Hat, "Some people are already using it in production; we're terrified of this. It's really not ready yet." http://www.zdnet.com/article/linux-storage-futures/ http://www.zdnet.com/article/linux-storage-futures/ EDIT: And I'm not suggestion it isn't badass, just that the idea Ceph as a first suggestion for general purpose RAID replacement is probably not a good idea just yet, and that there's an alternative.
- mtanski 12y ago
  cmurf. What the article and John is talking about is CephFS (which we do use). As you can see that is in the heading of the quote that you took. What John is not talking about is the Object Store (S3 compatible) or RBD (rados block device, EBS like block storage). There's countless people using both Object Store and RBD in production today. Both of these are the most deployed solutions for OpenStack (for storage or object storage). For example the Bloomberg folks use in production. We use CephFS, and contributed to it earlier, and helped find a lot of bugs in the kernel and the MDS. Last time we uncounted a bug was over a year ago. We currently only use one MDS with a hot spare (backup node actively replaying the log). What's really missing today from CephFS is the fsck and restore utilities and I believe people are working on them as we speak.

duaneb 12y ago

Clock skew is notoriously difficult to actually fix, especially with a network partition.