7 ms·

I don't understand why this is being used over say librados + librbd from Ceph. This seems like an awful lot of work to get the same functionality that Gluster

I don't understand why this is being used over say librados + librbd from Ceph. This seems like an awful lot of work to get the same functionality that Gluster/Ceph already have. Is there something I'm missing here?

bogomipz 10y ago
I kind of see this as a general trend where CoreOS is concerned they reimplement everything in house, even when open source solution might already exist - rkt, etcd, now this.
- zsmith928 10y ago
  what is the existing alternative for storage in distributed container environments?
  - deleted 10y ago
    [deleted]
  - gsmethells 10y ago
    Ceph, OpenStack Swift, GlusterFS, OrangeFS, Lustre. https://en.wikipedia.org/wiki/Clustered_file_system#Distributed_file_systems https://en.wikipedia.org/wiki/Clustered_file_system#Distribu... https://en.wikipedia.org/wiki/Object_storage https://en.wikipedia.org/wiki/Object_storage
    - pinewurst 10y ago
      If Lustre is the answer, you're probably asking the wrong question. Unless that question involves short life-span, massively parallel swap-outs from one weapons simulation to another. I write this not as storage religion (of which there's far too much), but to warn away those who haven't experienced the many kinds of data (and stomach lining) loss that come with being a Lustre admin.
      - sitkack 10y ago
        https://www.csc.fi/web/blog/post/-/blogs/the-largest-unplanned-outage-in-years-and-how-we-survived-it https://www.csc.fi/web/blog/post/-/blogs/the-largest-unplann...
    - nickpsecurity 10y ago
      Add this to your list: http://sector.sourceforge.net/ http://sector.sourceforge.net/ I swore I added it to the Wikipedia links. I'll have to do it again, I guess.
    - objectivefs 10y ago
      ObjectiveFS[1] is another option if you have access to an object store (e.g. S3 or GCS) from your containers. [1] https://objectivefs.com https://objectivefs.com
  - domsj 10y ago
    Have a look at Open vStorage (http://www.openvstorage.com http://www.openvstorage.com). Some highlights: open-source, core is battle proven for more than 7 years, performance, scales across datacenters , unlimited snapshots, ... (disclaimer: I'm an engineer there...)
- amelius 10y ago
  Competition is good, even in open-source land. By the way, I'm still hoping for an open-source competitor for elasticsearch.
  - runako 10y ago
    > I'm still hoping for an open-source competitor for elasticsearch I thought Elasticsearch was open-sourced, and the GitHub repo says Apache v2. What am I missing here?
    - amelius 10y ago
      Nothing. It is open-source. But it needs an open-source competitor. There's too little happening at the open-source side of search.
      - sciurus 10y ago
        Huh? It has a major competitor: Solr. I'm not sure what you mean unless you object because they both use lucene. https://lucene.apache.org/solr/ https://lucene.apache.org/solr/
      - brazzledazzle 10y ago
        What about Solr? They're both based on Lucene but I imagine it qualifies.
- alrs 10y ago
  See also: Piston Cloud.
- __jal 10y ago
  Yeah, I've been smelling NIH for quite a while. And it generally seems that, when they say "composable modular tools", one should read "includes dependencies on the rest of our stack". This isn't intended as harshness. It is just that every time a tool from COS seems interesting enough to look in to, it turns out that I wouldn't be investing in a tool, it would be investing in an ecosystem that duplicates a ton of what I'm already managing.
  - Perceptes 10y ago
    Kubernetes is a big exception to this, it seems. Before Kubernetes was announced, CoreOS was actively developing fleet. After Kubernetes, fleet development was largely halted and is now just a "distributed init system for low-level cluster components," e.g. a way to bootstrap Kubernetes itself. I asked the lead developer of fleet if they would have created it if Kubernetes had existed at the time, and he said no. Kubernetes has turned into a big part of the CoreOS pitch, and they've contributed a lot to the project. Giving up a project you started in house because you recognized that another organization's project fit the bill better is very far from NIH.
- vidarh 10y ago
  Rkt had good reasoning behind it, and I think it's reasonable to assume that a lot of the recent changes to Docker came because they forced Docker's hand. For Etcd I'm of two minds. I like its simplicity. On the other hand, I have had far more problems with Etcd than I've ever had with e.g. Consul (though I dislike Consuls "and the kitchen sink" approach), up to and including data loss and more than once having to re-initialise clusters. I'm a couple of years of no further problems at least a way from trusting Etcd with my data. For this? They have to have very compelling stories for why they couldn't e.g. modify Ceph or Gluster to achieve whatever it is that they want. And it's no clear what they want. Gluster for example is built as a stack of pluggable translators that is relatively easy to extend. Even if they have a credible and compelling story for why we need something else, it will take years of production use by others before I'd trust any data to a new distributed file system. Gluster and Ceph have those years. I've had a Gluster volume running without maintenance or data loss for more than five years. Even then it took several years before I truly started to trust it.
  - ideal0227 10y ago
    Have you ever thought the etcd issues was just because how you operate it? Or maybe you were running an old version of it? Have you reported to upstream? If there is a very common issue that you can meet frequently, it should have been fixed or you should have reported it.
    - vidarh 10y ago
      I'm quite sure the etcd issues was "just because" of how we operated it. But the point is that I have had these issues with Etcd, using the default CoreOS configuration of Etcd, and I've not had these issues with Gluster despite having Gluster volumes running many times as long. Gluster had many issues too when it was a young project, but they were fixed many years ago. I don't doubt that Etcd too will become solid enough for me to trust, but looking at part of the docs [1] we still have "gems" such as "Permanent Loss of Quorum Requires New Cluster" (you don't lose your data to that one, but it's painful nevertheless). And yes, I've run into that one. Not "permanent" as in "we could never recover the machines in question", but "permanent" as in "we're not going to be able to get this sorted fast enough, so let's fail everything over". Everything else we run can handle that scenario easily. We know it means consistency issues, but if you run into that situation we have taken the decision that they are acceptable and will be resolved later if necessary. The last thing I need in a scenario like that is for some critical component to just refuse to run without requiring lots of manual intervention. All we'd like would be to be able to say "ok, drop all the other members and pretend all is well, and yes we know what we're asking". There are a number of gotcha's like that with Etcd that sounds find until they bite you in the ass. They'll resolve these things eventually, but storage is the very last place I'd be willing to take risks with it. So until they have years of track record of running flawlessly, with less hassle than Gluster or Ceph, and substantial advantages, I'll see it as the dangerous choice. [1] https://coreos.com/etcd/docs/latest/runtime-reconf-design.html https://coreos.com/etcd/docs/latest/runtime-reconf-design.ht...
deleted 10y ago
[deleted]
philips 10y ago
A few major things we wanted to accomplish: Backed by etcd. Today etcd is a well tested and widely used consistent store with users like Kubernetes, Flannel (now Canal), Fleet, SkyDNS and many others. Building something like etcd requires tons of testing and etcd is becoming the solid go to for this category of distributed problems. Easy to work on code base. Building an OSS community around a complex technology is really hard. It shouldn't be underestimated that a code base that can be built on OS X, Linux, and Windows will have an advantage in getting people involved and maturing the project. Designed for other storage application. Torus is backed by a distributed storage interface that is exposed via gRPC (with the opportunity for language bindings in lots of languages). Our hope is to let people build write-ahead log systems, object systems, or filesystems on top of this abstraction. Flexibility built into the storage layer. Over time we want people to implement different hash maps or explicit configuration of storage layout. This is an abstraction that we have today and intend to build out over time with feedback from users. Happy to answer more questions. This is a young project and we look forward to working with people and hearing about what they want out of a storage system. Overall, we want to build something with folks that is easy to work on and backs itself with technologies that are getting good operational understanding in the "Cloud Native" space.
- koolba 10y ago
  > Backed by etcd. Today etcd is a well tested and widely used consistent store with users like Kubernetes, Flannel (now Canal), Fleet, SkyDNS and many others. Building something like etcd requires tons of testing and etcd is becoming the solid go to for this category of distributed problems. Being built atop something else doesn't mean it inherits all the stability of the underlying system; it's an upper bound not a lower bound.
- xfactor973 10y ago
  It sounds like etcd is in the I/O path. Is that correct?
  - barakm 10y ago
    It's in the sync() path, but not in the I/O path.

bogomipz 10y ago

I kind of see this as a general trend where CoreOS is concerned they reimplement everything in house, even when open source solution might already exist - rkt, etcd, now this.

zsmith928 10y ago
what is the existing alternative for storage in distributed container environments?
- deleted 10y ago
  [deleted]
- gsmethells 10y ago
  Ceph, OpenStack Swift, GlusterFS, OrangeFS, Lustre. https://en.wikipedia.org/wiki/Clustered_file_system#Distributed_file_systems https://en.wikipedia.org/wiki/Clustered_file_system#Distribu... https://en.wikipedia.org/wiki/Object_storage https://en.wikipedia.org/wiki/Object_storage
  - pinewurst 10y ago
    If Lustre is the answer, you're probably asking the wrong question. Unless that question involves short life-span, massively parallel swap-outs from one weapons simulation to another. I write this not as storage religion (of which there's far too much), but to warn away those who haven't experienced the many kinds of data (and stomach lining) loss that come with being a Lustre admin.
    - sitkack 10y ago
      https://www.csc.fi/web/blog/post/-/blogs/the-largest-unplanned-outage-in-years-and-how-we-survived-it https://www.csc.fi/web/blog/post/-/blogs/the-largest-unplann...
  - nickpsecurity 10y ago
    Add this to your list: http://sector.sourceforge.net/ http://sector.sourceforge.net/ I swore I added it to the Wikipedia links. I'll have to do it again, I guess.
  - objectivefs 10y ago
    ObjectiveFS[1] is another option if you have access to an object store (e.g. S3 or GCS) from your containers. [1] https://objectivefs.com https://objectivefs.com
- domsj 10y ago
  Have a look at Open vStorage (http://www.openvstorage.com http://www.openvstorage.com). Some highlights: open-source, core is battle proven for more than 7 years, performance, scales across datacenters , unlimited snapshots, ... (disclaimer: I'm an engineer there...)
amelius 10y ago
Competition is good, even in open-source land. By the way, I'm still hoping for an open-source competitor for elasticsearch.
- runako 10y ago
  > I'm still hoping for an open-source competitor for elasticsearch I thought Elasticsearch was open-sourced, and the GitHub repo says Apache v2. What am I missing here?
  - amelius 10y ago
    Nothing. It is open-source. But it needs an open-source competitor. There's too little happening at the open-source side of search.
    - sciurus 10y ago
      Huh? It has a major competitor: Solr. I'm not sure what you mean unless you object because they both use lucene. https://lucene.apache.org/solr/ https://lucene.apache.org/solr/
    - brazzledazzle 10y ago
      What about Solr? They're both based on Lucene but I imagine it qualifies.
alrs 10y ago
See also: Piston Cloud.
__jal 10y ago
Yeah, I've been smelling NIH for quite a while. And it generally seems that, when they say "composable modular tools", one should read "includes dependencies on the rest of our stack". This isn't intended as harshness. It is just that every time a tool from COS seems interesting enough to look in to, it turns out that I wouldn't be investing in a tool, it would be investing in an ecosystem that duplicates a ton of what I'm already managing.
- Perceptes 10y ago
  Kubernetes is a big exception to this, it seems. Before Kubernetes was announced, CoreOS was actively developing fleet. After Kubernetes, fleet development was largely halted and is now just a "distributed init system for low-level cluster components," e.g. a way to bootstrap Kubernetes itself. I asked the lead developer of fleet if they would have created it if Kubernetes had existed at the time, and he said no. Kubernetes has turned into a big part of the CoreOS pitch, and they've contributed a lot to the project. Giving up a project you started in house because you recognized that another organization's project fit the bill better is very far from NIH.
vidarh 10y ago
Rkt had good reasoning behind it, and I think it's reasonable to assume that a lot of the recent changes to Docker came because they forced Docker's hand. For Etcd I'm of two minds. I like its simplicity. On the other hand, I have had far more problems with Etcd than I've ever had with e.g. Consul (though I dislike Consuls "and the kitchen sink" approach), up to and including data loss and more than once having to re-initialise clusters. I'm a couple of years of no further problems at least a way from trusting Etcd with my data. For this? They have to have very compelling stories for why they couldn't e.g. modify Ceph or Gluster to achieve whatever it is that they want. And it's no clear what they want. Gluster for example is built as a stack of pluggable translators that is relatively easy to extend. Even if they have a credible and compelling story for why we need something else, it will take years of production use by others before I'd trust any data to a new distributed file system. Gluster and Ceph have those years. I've had a Gluster volume running without maintenance or data loss for more than five years. Even then it took several years before I truly started to trust it.
- ideal0227 10y ago
  Have you ever thought the etcd issues was just because how you operate it? Or maybe you were running an old version of it? Have you reported to upstream? If there is a very common issue that you can meet frequently, it should have been fixed or you should have reported it.
  - vidarh 10y ago
    I'm quite sure the etcd issues was "just because" of how we operated it. But the point is that I have had these issues with Etcd, using the default CoreOS configuration of Etcd, and I've not had these issues with Gluster despite having Gluster volumes running many times as long. Gluster had many issues too when it was a young project, but they were fixed many years ago. I don't doubt that Etcd too will become solid enough for me to trust, but looking at part of the docs [1] we still have "gems" such as "Permanent Loss of Quorum Requires New Cluster" (you don't lose your data to that one, but it's painful nevertheless). And yes, I've run into that one. Not "permanent" as in "we could never recover the machines in question", but "permanent" as in "we're not going to be able to get this sorted fast enough, so let's fail everything over". Everything else we run can handle that scenario easily. We know it means consistency issues, but if you run into that situation we have taken the decision that they are acceptable and will be resolved later if necessary. The last thing I need in a scenario like that is for some critical component to just refuse to run without requiring lots of manual intervention. All we'd like would be to be able to say "ok, drop all the other members and pretend all is well, and yes we know what we're asking". There are a number of gotcha's like that with Etcd that sounds find until they bite you in the ass. They'll resolve these things eventually, but storage is the very last place I'd be willing to take risks with it. So until they have years of track record of running flawlessly, with less hassle than Gluster or Ceph, and substantial advantages, I'll see it as the dangerous choice. [1] https://coreos.com/etcd/docs/latest/runtime-reconf-design.html https://coreos.com/etcd/docs/latest/runtime-reconf-design.ht...

deleted 10y ago

[deleted]

philips 10y ago

A few major things we wanted to accomplish: Backed by etcd. Today etcd is a well tested and widely used consistent store with users like Kubernetes, Flannel (now Canal), Fleet, SkyDNS and many others. Building something like etcd requires tons of testing and etcd is becoming the solid go to for this category of distributed problems. Easy to work on code base. Building an OSS community around a complex technology is really hard. It shouldn't be underestimated that a code base that can be built on OS X, Linux, and Windows will have an advantage in getting people involved and maturing the project. Designed for other storage application. Torus is backed by a distributed storage interface that is exposed via gRPC (with the opportunity for language bindings in lots of languages). Our hope is to let people build write-ahead log systems, object systems, or filesystems on top of this abstraction. Flexibility built into the storage layer. Over time we want people to implement different hash maps or explicit configuration of storage layout. This is an abstraction that we have today and intend to build out over time with feedback from users. Happy to answer more questions. This is a young project and we look forward to working with people and hearing about what they want out of a storage system. Overall, we want to build something with folks that is easy to work on and backs itself with technologies that are getting good operational understanding in the "Cloud Native" space.

koolba 10y ago
> Backed by etcd. Today etcd is a well tested and widely used consistent store with users like Kubernetes, Flannel (now Canal), Fleet, SkyDNS and many others. Building something like etcd requires tons of testing and etcd is becoming the solid go to for this category of distributed problems. Being built atop something else doesn't mean it inherits all the stability of the underlying system; it's an upper bound not a lower bound.
xfactor973 10y ago
It sounds like etcd is in the I/O path. Is that correct?
- barakm 10y ago
  It's in the sync() path, but not in the I/O path.