6 ms·

A few major things we wanted to accomplish: Backed by etcd. Today etcd is a well tested and widely used consistent store with users like Kubernetes, Flannel (n

by philips 10y ago

A few major things we wanted to accomplish: Backed by etcd. Today etcd is a well tested and widely used consistent store with users like Kubernetes, Flannel (now Canal), Fleet, SkyDNS and many others. Building something like etcd requires tons of testing and etcd is becoming the solid go to for this category of distributed problems. Easy to work on code base. Building an OSS community around a complex technology is really hard. It shouldn't be underestimated that a code base that can be built on OS X, Linux, and Windows will have an advantage in getting people involved and maturing the project. Designed for other storage application. Torus is backed by a distributed storage interface that is exposed via gRPC (with the opportunity for language bindings in lots of languages). Our hope is to let people build write-ahead log systems, object systems, or filesystems on top of this abstraction. Flexibility built into the storage layer. Over time we want people to implement different hash maps or explicit configuration of storage layout. This is an abstraction that we have today and intend to build out over time with feedback from users. Happy to answer more questions. This is a young project and we look forward to working with people and hearing about what they want out of a storage system. Overall, we want to build something with folks that is easy to work on and backs itself with technologies that are getting good operational understanding in the "Cloud Native" space.

koolba 10y ago
> Backed by etcd. Today etcd is a well tested and widely used consistent store with users like Kubernetes, Flannel (now Canal), Fleet, SkyDNS and many others. Building something like etcd requires tons of testing and etcd is becoming the solid go to for this category of distributed problems. Being built atop something else doesn't mean it inherits all the stability of the underlying system; it's an upper bound not a lower bound.
xfactor973 10y ago
It sounds like etcd is in the I/O path. Is that correct?
- barakm 10y ago
  It's in the sync() path, but not in the I/O path.
  - notacoward 10y ago
    Why is it in the sync path? That seems unnecessary, as well as bad for both performance and correctness (another thing in any path means another potential for partial failure).
    - grogers 10y ago
      This is not a great analogy since their first application is block storage, but think about a journaling file system. Typically file data is not journaled, but metadata is. By having a consistent view of the metadata, the entire filesystem (as far as you can interact with it) is consistent. That consistent journal is the same primitive that etcd is providing.
      - notacoward 10y ago
        I don't think the two cases are analogous. First, block devices don't have any metadata to speak of. Second, filesystems have journals to "cover up" for the main-store operations being asynchronous and/or non-atomic. However, sync() is completely synchronous by definition - hence the name - so such "covering up" would be superfluous. There must be some metadata that's being written at etcd because it needs to be read from there later, but there's nothing in block-device semantics to require any such thing. Thinking about it further, I think I can guess at what's going on here. The key observation is that there's no sync() at the block device level. It's a filesystem operation; block devices don't see it. Sure, there are queue flags and FUA and such, but those are different (and I'm not sure any of those exist in NBD). Where is this sync() path? I'm guessing it's internal on the data servers, to deal with data that's being buffered there. With both replication and erasure coding, correct recovery requires exact knowledge of what has been fully written where, and that's the kind of metadata I suspect is being put in etcd. There's not even necessarily anything wrong with it, unless updating that information only on sync() means that supposedly durable writes since the last (unpredictable to the client) sync could be lost on failure. I hope that's not the case. Maybe I'll find time, in the midst of my work on an actual production-level distributed filesystem, to look at the code and see if my guess is correct.
vidarh 10y ago
I've managed to lose data with Etcd, and have regularly had issues with membership issues requiring maintenance. Meanwhile I've had Glusterfs volumes remain available for 5-6 years without maintenance at all. To me at least, having it "backed by etcd" is a big red flag, not a feature.
- fintler 10y ago
  I've managed to lose data with Etcd How did you manage to lose data?
  - vidarh 10y ago
    I don't remember the details, and to be clear this was not with the most recent version of Etcd at all - it was quite a while ago. I think and hope that whatever the problem was is no longer an issue. It has certainly gotten a lot better. My point is mainly that it is way too young to be something to trust important data to. We didn't either - what we lost was a cluster configuration that we could recreate from a combination of backups and redoing a handful of operations, but at the time it was scary to see, and prompted us to be very careful about what we put into Etcd going forward.
  - stevelandis 10y ago
    Use etcd in production and you will lose data. Bounce nodes, simulate power loss and you will lose data. Have you actually used it?
- nickpsecurity 10y ago
  I agree. I was also thinking of Ceph when I wrote my reply to Cantrill. Esp Ceph on XFS given maturity of both plus enormous effort that's gone into XFS. Sector/Sphere with UDT is also kick-ass. It took both of these a long time to get to where they were fighting all kinds of unforseen issues. They also tried to build on proven components that themselves were battletested for years. So, this company is trading away stuff like that for custom components and Etcd? And for a component focusing on integrity and availability? Huh?

by philips 10y ago

koolba 10y ago

> Backed by etcd. Today etcd is a well tested and widely used consistent store with users like Kubernetes, Flannel (now Canal), Fleet, SkyDNS and many others. Building something like etcd requires tons of testing and etcd is becoming the solid go to for this category of distributed problems. Being built atop something else doesn't mean it inherits all the stability of the underlying system; it's an upper bound not a lower bound.

xfactor973 10y ago

It sounds like etcd is in the I/O path. Is that correct?

barakm 10y ago
It's in the sync() path, but not in the I/O path.
- notacoward 10y ago
  Why is it in the sync path? That seems unnecessary, as well as bad for both performance and correctness (another thing in any path means another potential for partial failure).
  - grogers 10y ago
    This is not a great analogy since their first application is block storage, but think about a journaling file system. Typically file data is not journaled, but metadata is. By having a consistent view of the metadata, the entire filesystem (as far as you can interact with it) is consistent. That consistent journal is the same primitive that etcd is providing.
    - notacoward 10y ago
      I don't think the two cases are analogous. First, block devices don't have any metadata to speak of. Second, filesystems have journals to "cover up" for the main-store operations being asynchronous and/or non-atomic. However, sync() is completely synchronous by definition - hence the name - so such "covering up" would be superfluous. There must be some metadata that's being written at etcd because it needs to be read from there later, but there's nothing in block-device semantics to require any such thing. Thinking about it further, I think I can guess at what's going on here. The key observation is that there's no sync() at the block device level. It's a filesystem operation; block devices don't see it. Sure, there are queue flags and FUA and such, but those are different (and I'm not sure any of those exist in NBD). Where is this sync() path? I'm guessing it's internal on the data servers, to deal with data that's being buffered there. With both replication and erasure coding, correct recovery requires exact knowledge of what has been fully written where, and that's the kind of metadata I suspect is being put in etcd. There's not even necessarily anything wrong with it, unless updating that information only on sync() means that supposedly durable writes since the last (unpredictable to the client) sync could be lost on failure. I hope that's not the case. Maybe I'll find time, in the midst of my work on an actual production-level distributed filesystem, to look at the code and see if my guess is correct.

vidarh 10y ago

I've managed to lose data with Etcd, and have regularly had issues with membership issues requiring maintenance. Meanwhile I've had Glusterfs volumes remain available for 5-6 years without maintenance at all. To me at least, having it "backed by etcd" is a big red flag, not a feature.

fintler 10y ago
I've managed to lose data with Etcd How did you manage to lose data?
- vidarh 10y ago
  I don't remember the details, and to be clear this was not with the most recent version of Etcd at all - it was quite a while ago. I think and hope that whatever the problem was is no longer an issue. It has certainly gotten a lot better. My point is mainly that it is way too young to be something to trust important data to. We didn't either - what we lost was a cluster configuration that we could recreate from a combination of backups and redoing a handful of operations, but at the time it was scary to see, and prompted us to be very careful about what we put into Etcd going forward.
- stevelandis 10y ago
  Use etcd in production and you will lose data. Bounce nodes, simulate power loss and you will lose data. Have you actually used it?
nickpsecurity 10y ago
I agree. I was also thinking of Ceph when I wrote my reply to Cantrill. Esp Ceph on XFS given maturity of both plus enormous effort that's gone into XFS. Sector/Sphere with UDT is also kick-ass. It took both of these a long time to get to where they were fighting all kinds of unforseen issues. They also tried to build on proven components that themselves were battletested for years. So, this company is trading away stuff like that for custom components and Etcd? And for a component focusing on integrity and availability? Huh?