4 ms·

> Block level is the lowest form of addressing bytes on devices. Filesystems are an abstraction on top of block devices. Object stores are an abstraction on fil

> Block level is the lowest form of addressing bytes on devices. Filesystems are an abstraction on top of block devices. Object stores are an abstraction on filesystems. I don't agree with this, but I think you may be confused because "Object Storage" can mean several different things. "Object Store" in Ceph (as in RADOS - Reliable Autonomous Distributed Object Store) basically means key-value store. I typically say "blob store" instead to avoid the confusion with more sophisticated systems. It is exposed through a S3-like API. As far as I know, this layer of CEPH is pretty good, and you need a layer like this in most distributed systems anyway. Ceph provides something called RBD, RADOS Block Device, which exposes a Block Device interface and is implemented on top of RADOS blob storage. It is useful for VM disks and has decent performance because it makes heavy use of the cache. Some people use filesystems on top of RBD, but as far as I know CephFS itself does not sit on top of RBD. It is not as widely used as RBD because it is pretty recent (first release in 2016). The data is stored in RADOS and the metadata (which is the hardest part in a distributed filesystem) is dealt with by a Metadata Server cluster (MDS). This sounds like a typical distributed filesystem architecture to me, similar to GFS (the MDS replaces the GFS master and RADOS is used instead of chunk servers). People tend to have a lot of issues with Ceph, but I think this is because: 1) It is used in reasonably large scale production settings where you are going to have issues anyway ; 2) It is not as easy to understand and fine-tune as it should be ; 3) Some people expect it to solve all their issues magically with perfect performance... 4) Some people use filesystems on top of RBD when they should have used CephFS or even direct interfaces to RADOS when possible. But in general, I think Ceph is an example of a decently architectured complex distributed system.

manigandham 8y ago
Sure, and key/value systems are at the similar level of object stores, meaning they are abstractions on filesystems (which are abstractions on block devices). This is the hierarchy. Using Ceph for block and file access is like using AWS S3 to emulate block devices and filesystems. It'll work, and there is software for it, but it will never be very good. And Ceph is far from S3.
- noahdesu 8y ago
  > It'll work, and there is software for it, but it will never be very good What are some examples of distributed file systems and block devices that _are_ very good?
gregsfortytwo 8y ago
Just for future reference, RADOS is actually not very S3-like. It is an object store; it does map from object names to buckets of bytes. But unlike S3 and many similar object stores or key-value DBs, RADOS allows you to do file-like operations: you can append, write to random offsets in the object, overwrite pieces of it but not the whole object, etc. (That's all in addition to some stunningly-complex stuff like injecting custom code to do specific kinds of transactional read-writes on the OSD [storage node] itself.) That's all key to RBD being useful, or indeed CephFS itself. There are systems that map a filesystem layer on top of S3, but they have trouble because there aren't good ways to overwrite random small pieces of an S3 object. With RADOS, there are! :)

manigandham 8y ago

Sure, and key/value systems are at the similar level of object stores, meaning they are abstractions on filesystems (which are abstractions on block devices). This is the hierarchy. Using Ceph for block and file access is like using AWS S3 to emulate block devices and filesystems. It'll work, and there is software for it, but it will never be very good. And Ceph is far from S3.

noahdesu 8y ago
> It'll work, and there is software for it, but it will never be very good What are some examples of distributed file systems and block devices that _are_ very good?

gregsfortytwo 8y ago

Just for future reference, RADOS is actually not very S3-like. It is an object store; it does map from object names to buckets of bytes. But unlike S3 and many similar object stores or key-value DBs, RADOS allows you to do file-like operations: you can append, write to random offsets in the object, overwrite pieces of it but not the whole object, etc. (That's all in addition to some stunningly-complex stuff like injecting custom code to do specific kinds of transactional read-writes on the OSD [storage node] itself.) That's all key to RBD being useful, or indeed CephFS itself. There are systems that map a filesystem layer on top of S3, but they have trouble because there aren't good ways to overwrite random small pieces of an S3 object. With RADOS, there are! :)