4 ms·

Dangit, I trust the CoreOS team more/better than a lot of people in the space. Torus would have been so useful. At the other end of the spectrum though, maybe

Dangit, I trust the CoreOS team more/better than a lot of people in the space. Torus would have been so useful. At the other end of the spectrum though, maybe this is reasonable? As a developer, my first thoughts for "I want my own S3" is not etcd (strong consistency) but projects like https://github.com/minio/minio https://github.com/minio/minio , or even using eventually consistent SQLite replication / synchronization tools https://github.com/gundb/sqlite https://github.com/gundb/sqlite . So that makes me ask about rook.io too, what layer of the "stack" is it trying to fit into? Obviously pretty low, but that also seems unnecessary (and part of why I suspect Torus is stopping).

jacques_chester 10y ago
At a glance, Torus was intended to be a distributed file system. As I understand it, distributed file systems are easier than distributed block systems, but harder than distributed blob systems. A blob system is all-or-nothing. You create or replace the entire blob at once. This makes bookkeeping and replication much easier for the implementer. A filesystem supports much richer semantics, including the ability to seek parts of files and modify small regions of files. You need a lot more mechanics to maintain consistency across a network. A block store is difficult because you're trying to work at very high speed on very small units of state wooshing back and forth willy-nilly. You don't get to rely on any of the higher semantics provided by a filesystem or blobstore, since you're pretending to be a magical harddrive. I am often wrong in these matters, as an interested outsider, so I'd be happy to receive correction.
- umamukkara 10y ago
  Disclosure: I work for OpenEBS project Torus was intending to write distributed block storge that is container native. Metadata management using key value (KV / etcd) method is increases the complexity and not new. Ceph tried it. OpenEBS uses a novel approch, linux sparse files for managing the blocks of a volume. Fork of Rancher longhorn. The issue of managing the large scale distributed block storage metadata is solved easily throught he management of the files (not blocks). https://blog.openebs.io/torus-from-coreos-steps-aside-as-cloud-native-storage-platform-what-now-2375e7f5b145#.fslj3rwsg https://blog.openebs.io/torus-from-coreos-steps-aside-as-clo...
  - fh973 10y ago
    What I don't get around the new efforts around container-native storage: if you decide to build a new container-native storage system, why would you aim for block storage and not file storage? Block storage is not exactly a great fit for containers as you can't access its file systems from multiple hosts and fail-over is a hassle (forced remount, fsck).
    - umamukkara 10y ago
      Container native storage has two aspects. One, the storage that container uses, (Like Docker uses Device Mapper, Overlay2 etc.) and two, the persistent storage that the applications inside containers need. Both of them need to be truly container native. Portworx is attempting LCFS to provide a container native storage for containers itself https://github.com/portworx/lcfs https://github.com/portworx/lcfs. So, you are right. You would need a container-native storage (file system for Docker) for running containers. OpenEBS is targeting providing containerized storage (persistent+block storage) for applications in containers. OpenEBS builds a storage volume as a container and presents the volume-container as part of the K8s POD. This way the the storage persistence problems are resolved by K8S orchestration ingelligence written for application PODs. Ofcouse, OpenEBS containers will use LCFS when it is ready.
    - wmf 10y ago
      File systems are much harder than block storage due to the large amount of metadata and the consistency/durability requirements for metadata. So a lot of systems just build block storage and format ext4 on top or put NFS+ZFS on top (but that's not very distributed).
- fh973 10y ago
  Working on a distributed file system (Quobyte), I'd say the order of complexity is file > block > object. Why? Because a drop-in replacement for a local file system also needs to be good at high-performance concurrent block IO, otherwise you wouldn't be able to run applications like databases.
  - jacques_chester 10y ago
    So am I wrong in thinking FS semantics can give some hints on how to optimise activity? That is: a pure block store must treat every operation with identical care, as it has no insight into the higher-level meaning. But a filesystem can, for example, give consideration to whether it's a write, read, create or appending op; where in the file it's happening; the order of operations; metadata vs file contents and so on. Again, I am only distantly familiar with any of this.
    - zzzcpan 10y ago
      "block store must treat every operation with identical care" Yes, and because of that it's easier to do a block storage, than a filesystem. But sure, filesystems have enough semantics to potentially scale across multiple data centers and probably be performant enough for many apps to tolerate those latencies. However, the level of complexity would be similar to that of Google's Spanner.
      - jacques_chester 10y ago
        Thankyou.

jacques_chester 10y ago

At a glance, Torus was intended to be a distributed file system. As I understand it, distributed file systems are easier than distributed block systems, but harder than distributed blob systems. A blob system is all-or-nothing. You create or replace the entire blob at once. This makes bookkeeping and replication much easier for the implementer. A filesystem supports much richer semantics, including the ability to seek parts of files and modify small regions of files. You need a lot more mechanics to maintain consistency across a network. A block store is difficult because you're trying to work at very high speed on very small units of state wooshing back and forth willy-nilly. You don't get to rely on any of the higher semantics provided by a filesystem or blobstore, since you're pretending to be a magical harddrive. I am often wrong in these matters, as an interested outsider, so I'd be happy to receive correction.

umamukkara 10y ago
Disclosure: I work for OpenEBS project Torus was intending to write distributed block storge that is container native. Metadata management using key value (KV / etcd) method is increases the complexity and not new. Ceph tried it. OpenEBS uses a novel approch, linux sparse files for managing the blocks of a volume. Fork of Rancher longhorn. The issue of managing the large scale distributed block storage metadata is solved easily throught he management of the files (not blocks). https://blog.openebs.io/torus-from-coreos-steps-aside-as-cloud-native-storage-platform-what-now-2375e7f5b145#.fslj3rwsg https://blog.openebs.io/torus-from-coreos-steps-aside-as-clo...
- fh973 10y ago
  What I don't get around the new efforts around container-native storage: if you decide to build a new container-native storage system, why would you aim for block storage and not file storage? Block storage is not exactly a great fit for containers as you can't access its file systems from multiple hosts and fail-over is a hassle (forced remount, fsck).
  - umamukkara 10y ago
    Container native storage has two aspects. One, the storage that container uses, (Like Docker uses Device Mapper, Overlay2 etc.) and two, the persistent storage that the applications inside containers need. Both of them need to be truly container native. Portworx is attempting LCFS to provide a container native storage for containers itself https://github.com/portworx/lcfs https://github.com/portworx/lcfs. So, you are right. You would need a container-native storage (file system for Docker) for running containers. OpenEBS is targeting providing containerized storage (persistent+block storage) for applications in containers. OpenEBS builds a storage volume as a container and presents the volume-container as part of the K8s POD. This way the the storage persistence problems are resolved by K8S orchestration ingelligence written for application PODs. Ofcouse, OpenEBS containers will use LCFS when it is ready.
  - wmf 10y ago
    File systems are much harder than block storage due to the large amount of metadata and the consistency/durability requirements for metadata. So a lot of systems just build block storage and format ext4 on top or put NFS+ZFS on top (but that's not very distributed).
fh973 10y ago
Working on a distributed file system (Quobyte), I'd say the order of complexity is file > block > object. Why? Because a drop-in replacement for a local file system also needs to be good at high-performance concurrent block IO, otherwise you wouldn't be able to run applications like databases.
- jacques_chester 10y ago
  So am I wrong in thinking FS semantics can give some hints on how to optimise activity? That is: a pure block store must treat every operation with identical care, as it has no insight into the higher-level meaning. But a filesystem can, for example, give consideration to whether it's a write, read, create or appending op; where in the file it's happening; the order of operations; metadata vs file contents and so on. Again, I am only distantly familiar with any of this.
  - zzzcpan 10y ago
    "block store must treat every operation with identical care" Yes, and because of that it's easier to do a block storage, than a filesystem. But sure, filesystems have enough semantics to potentially scale across multiple data centers and probably be performant enough for many apps to tolerate those latencies. However, the level of complexity would be similar to that of Google's Spanner.
    - jacques_chester 10y ago
      Thankyou.