17 ms·

Torus: A distributed storage system by CoreOS

267 points by philips 10y ago

philips 10y ago
The CoreOS team is excited to make this initial release and start collaborating with folks that want to tackle distributed storage. tl;dr this is a new OSS distributed storage project that is written in Go and backed by etcd for consistency. The first project built on top of Torus is a network block device that can be mounted into containers for persistent storage. It also includes integrations out of the box for Kubernetes "flex volumes". Get involved! :) https://github.com/coreos/torus https://github.com/coreos/torus
- paulsutter 10y ago
  What's the current largest-scale deployment?
  - lobster_johnson 10y ago
    It's described as a prototype, so I'm pretty sure there's no deployment at scale at all.
- stevelandis 10y ago
  Brandon, a few months ago I met you at a meetup and you did not think this problem needed to be solved, and containers were for ephemeral apps... just an AMA request, what changed hearts?
CSDude 10y ago
Seems really nice. Would love to see it how it compares to Ceph, we were considering Ceph for our container storage. For starters, it seems much more simpler to deploy.
- Inflatablewoman 10y ago
  In my old firm we managed to get ceph running on coreos using ceph in docker. The storage was exposed using a s3 gateway. https://github.com/ceph/ceph-docker https://github.com/ceph/ceph-docker
  - ymse 10y ago
    Docker supports native RBD since version 1.8. Which should be both faster and more reliable than using the S3 gateway (though that has other benefits and becoming pretty good too).
- kinvaris 10y ago
  I work at a company where object storage is something they have worked on for many years to get it stable. You can download it for free (it's Open Source)! Open vStorage is very easy to deploy through Ansible! Try it :)! https://www.openvstorage.com/ https://www.openvstorage.com/ https://github.com/openvstorage https://github.com/openvstorage
xfactor973 10y ago
I don't understand why this is being used over say librados + librbd from Ceph. This seems like an awful lot of work to get the same functionality that Gluster/Ceph already have. Is there something I'm missing here?
- bogomipz 10y ago
  I kind of see this as a general trend where CoreOS is concerned they reimplement everything in house, even when open source solution might already exist - rkt, etcd, now this.
  - zsmith928 10y ago
    what is the existing alternative for storage in distributed container environments?
    - deleted 10y ago
      [deleted]
    - gsmethells 10y ago
      Ceph, OpenStack Swift, GlusterFS, OrangeFS, Lustre. https://en.wikipedia.org/wiki/Clustered_file_system#Distributed_file_systems https://en.wikipedia.org/wiki/Clustered_file_system#Distribu... https://en.wikipedia.org/wiki/Object_storage https://en.wikipedia.org/wiki/Object_storage
      - pinewurst 10y ago
        If Lustre is the answer, you're probably asking the wrong question. Unless that question involves short life-span, massively parallel swap-outs from one weapons simulation to another. I write this not as storage religion (of which there's far too much), but to warn away those who haven't experienced the many kinds of data (and stomach lining) loss that come with being a Lustre admin.
        10y ago
amq 10y ago
I just hope the performance is visibly better than of gluster. Especially for small files.
- zsmith928 10y ago
  are there any performance metrics for Tarus yet?
koolba 10y ago
So is this the building blocks for an in-house EBS?
- philips 10y ago
  Yes, that is the idea. Torus block is like EBS. Torus "library" could be used by other applications like an append only log, object store, etc.
  - stevelandiss 10y ago
    How do you say that with conviction? I looked at the code and it has no bearings of being able to build a fault tolerant elastic block storage system.
gsmethells 10y ago
Is the goal to provide POSIX file access as NFS and AFS do? The mention of object storage as a future direction feels like the early "the sky's the limit" euphoria of a new project that has yet to adopt specific end goals.
- barakm 10y ago
  More than "sky's the limit" -- early versions had POSIX access, though it was terribly messy. We know the architecture can support it, it's just a matter of learning from the mistakes and building something worth supporting.
- philips 10y ago
  The project doesn't provide a POSIX filesystem directly. But, instead provides a block device (think AWS EBS). The initial goals that are in this release: 1. An abstraction for replicated storage between machines in a cluster. 2. A block device "application" on top that an ext4 filesystem could be run on. This is like an EBS. In the future someone might build other applications like "object storage" or filesystems and we would love to get the feedback on the API to do that. But, that isn't in the initial goals and we will be focusing on the storage and block layers for the time being.
  - gsmethells 10y ago
    Will each disk in the system be weighted so that different size disks can be used in the system over time? OpenStack Swift offers this and is useful to not lock-in disk size at design time.
    - philips 10y ago
      Yes, see the admin documentation: https://github.com/coreos/torus/blob/master/Documentation/admin-guide.md#modify-my-cluster https://github.com/coreos/torus/blob/master/Documentation/ad...
agentultra 10y ago
Where's the formal specification of the system? If we want these distributed systems to be reliable how can we be sure our algorithms and processes work if we do not first model them? I'd be happy to work on a specification with the community.
- philips 10y ago
  We haven't gotten that far down the path. We would love the help. I believe there are some issues related to documenting the architecture and failure domains for the v0.1.1 release: https://github.com/coreos/torus/milestones/v0.1.1 https://github.com/coreos/torus/milestones/v0.1.1 This is one of those classic "release too early or release too late" sort of things that got cut in preference to get early feedback and community participation.
  - subway 10y ago
    Coming from the company that raised hell about Docker's lack of a spec, this is an interesting approach.
alrs 10y ago
Storing container images is a great use case for an object store, not a block store. They should get sucked down whole to ephemeral storage on the compute nodes. Mounting containers to a distributed block system is an anti-pattern. This is going to go poorly. Ceph has had some really smart people working on distributed block for a lot of years, and they still have significant issues. It's not because they're dumb, it's because performant, scalable, and available distributed block is either hard or impossible.
- sagichmal 10y ago
  This is about providing data volumes to containers, not about hosting container images or mounting containers themselves.
sysexit 10y ago
I love CoreOS and they've done some super impressive engineering. But really, a new storage system? Rewriting in Go and using etcd for central state management makes things easier, but this is still a hard problem. Some things that need to be solved sooner or later: data replication so that N faults of X entities are protected against (X can be disks, enclosures, racks, data centers, regions, ..), recovery from failed disks, scrubbing, data management, backups, some kind of storage orchestration and centralized management. If you look at Ceph which IMHO represents the state of the art in software defined storage, it took many years to get it to a point that it was usable. I hate to be cynical but in this case I would be surprised if CoreOS can pull this of. I wish them all the best though and would be happy for them if I am wrong.
- agentultra 10y ago
  > this is still a hard problem Which is why I'm rather surprised that these systems continue to be released without any formal specification or model checking. It's hard to get these things right... worth the effort, IMO, to create formal specifications.
- ams6110 10y ago
  It's like any time we take a step forward in one area we have to reinvent the last 50 years of computing to support it. Persistent storage is a "hard problem" Really?
  - vidarh 10y ago
    Persistent storage isn't a hard problem. Distributed, well performing, scalable and consistent storage is a hard problem.
    - dap 10y ago
      Actually, persistent storage is fairly hard in itself. Look at what ZFS does to ensure data integrity in the face of phantom writes, dropped writes, bad controllers, and other implicit, non-fatal failures.
dap 10y ago
> At its core, Torus is a library with an interface that appears as a traditional file, allowing for storage manipulation through well-understood basic file operations. Coordinated and checkpointed through etcd’s consensus process, this distributed file can be exposed to user applications in multiple ways. Today, Torus supports exposing this file as block-oriented storage via a Network Block Device (NBD). We also expect that in the future other storage systems, such as object storage, will be built on top of Torus as collections of these distributed files, coordinated by etcd. Am I understanding correctly that this is a file-based API? Distributing a POSIX filesystem effectively is very challenging, particularly since most applications that use them aren't written with CAP in mind; they don't expect a lot of basic operations to block for extended periods and fail in surprising ways, and they very often perform poorly when operations that are locally quick end up much slower over a network. To be concrete: > Today’s Torus release includes manifests using this feature to demonstrate running the PostgreSQL database server atop Kubernetes flex volumes, backed by Torus storage. It will be interesting to see how well this performs and how it behaves in the face of single-node failures and network congestion.
- philips 10y ago
  It isn't a POSIX filesystem API. It is like a single file or a big distributed tape.
  - dap 10y ago
    Virtualized, network block devices have all the same problems I described -- even worse, because the abstraction coneys even less about what an application is trying to do.
    - ibotty 10y ago
      No. The difference is, that with network block devices you are usually only allow accessing the block device once. That's an easier problem than mapping POSIX file system semantics!
bcantrill 10y ago
It's hard to take this seriously: storage is an excruciatingly hard problem, yet this cheerful description of a nascent and aspirational effort seems blissfully unaware of how difficult it is to even just reliably get bits to and from stable storage, let alone string that into a distributed system that must make CAP tradeoffs. There is not so much of a whisper as to what the data path actually looks like other than "the design includes the ability to support [...] Reed-Solomon error correction in the near future" -- and the fact that such an empty system hails itself as pioneering an unsolved problem in storage is galling in its ignorance of prior work (much of it open source). Take it from someone who has been involved in both highly durable local filesystems[1] and highly available object storage systems[2][3]: this is such a hard, nasty problem with so many dark, hidden and dire failure modes, that it takes years of running in production to get these systems to the level of reliability and operability that the data path demands. Given that (according to the repo, though not the breathless blog entry) its creators "do not recommend its use in production", Torus is -- in the famous words of Wolfgang Pauli -- not even wrong. [1] http://dtrace.org/blogs/bmc/2008/11/10/fishworks-now-it-can-be-told/ http://dtrace.org/blogs/bmc/2008/11/10/fishworks-now-it-can-... [2] http://dtrace.org/blogs/bmc/2013/06/25/manta-from-revelation-to-product/ http://dtrace.org/blogs/bmc/2013/06/25/manta-from-revelation... [3] http://dtrace.org/blogs/dap/2013/07/03/fault-tolerance-in-manta/ http://dtrace.org/blogs/dap/2013/07/03/fault-tolerance-in-ma...
- nickpsecurity 10y ago
  I totally agree with you. I also liked how they said the motivation is to make Google infrastructure for everyone else. How did Google do this? They basically imitated and improved on clustered filesystems developed in HPC. There were a lot of lessons to learn for emerging cloud market in all tooling done in HPC. Some was FOSS. Whereas, many companies seem to be doing the opposite in their work on these cloud filesystems. They don't build on proven, already-OSS components that have been battle-tested for a long time. They lack features and wisdom from prior deployments. They duplicate effort. They are also prone to using popular components, languages, whatever with implicitly-higher risk due to fact they were never built for fault-tolerant systems. Actually, many of them often assume something will watch and help them out in time of failure. If it's high-assurance or fault-tolerance, my manta is "tried and true beats novel and new." Just repurpose what's known to work while improving its capabilities and code quality. Knocks out risks you know about plus others you don't since they were never documents. Has that been your experience, too? Should be a maxim in IT given how often problem plays out.
ceocoder 10y ago
Congrats on the relaase! How does this compare to GlusterFS (loved it, was free - I used it before they got acquired by RedHat) and Isilon (expensive but fast and reliable). Is there an upper limit on number of nodes(?) or total size?
nwmcsween 10y ago
So a userspace fs, I'm guessing this will use fuse to actually expose a POSIX fs? Small sync writes will absolutely kill performance and will require all sorts of hacks like glusterfs has had to implement due to the amount of context switches. CoreOS really should have added whatever needed to ceph, filesystems are not something you just hack together overnight.
- betawaffle 10y ago
  We had a POSIX interface early on (via FUSE), but decided to expose a a block storage interface first instead. This is not a "filesystem", it's a storage abstraction, and we've spent more than 6 months on it. Seems like a lot of folks are quickly jumping to conclusions, which is expected from "the internet", but I would have hoped for better from HN.
  - zzzcpan 10y ago
    > but I would have hoped for better from HN Don't worry about it. Some guys just mistake their experience with traditional storage systems as meaningful here.
  - nwmcsween 10y ago
    6 months is overnight in terms of a filesystem, there are literally millions of dollars of work that was done with glusterfs and probably a magnitude less man hours. I understand coreos wants.to innovate but what does torus bring to the table besides negatives and NIH?
nzoschke 10y ago
I'm always in awe and grateful for the teams like CoreOS that tackle these tough problems. But when it comes to container storage I'm confused. The 12 factor app methodologies work very well for every service I have written and supported. Processes/containers get ephemeral storage. When you need persistence you delegate to a stateful service, generally Postgres and S3. Id much rather write my apps around these simple and well understood constraints then depend on magic file systems.
- ecnahc515 10y ago
  The average 12 factor app doesn't really need this, but as you mentioned, sometimes you delegate to Postgres, or S3, and today those are available as hosted products via Amazon, Google, Heroku, etc. However, when your running in your own datacenter, you don't have the luxury of using Amazon's hosted products, so Torus exists to help provide some building blocks to build your own S3, or potentially even RDS.
joshuak 10y ago
Good... Good... Let the hate flow through you. An open source OS company just blogged about a new open source project that they are putting resources into. They did not release a commercial product that competes with any existing storage solution. How exactly would one expect a new project to be announced? Tell me more about how far away from done you think they are. Sorry if that's a bit sarcastic, but seriously wouldn't it be nice if these comments where more along the lines of: "Interesting new project, here's some issues we've run into vis a vis reliable storage that you should look out for..."
- abrookewood 10y ago
  Yes, I have to agree. There's a little too much negativity for my liking, especially from people who have a vested interest in downplaying the efforts of others.
- stevelandis 10y ago
  IMO it is because this can derail the adoption of containers. I think an immature project like this announced so prematurely does more harm than good. And I appreciate people that have experience with this bringing up the real issues.
- pjmlp 10y ago
  I for one find it very positive, a storage engine implemented in Go, for what I could gather in a quick Github glance. For me, this is surely systems work.
- mmf 10y ago
  I absolutely agree, all those negative comments sound like rants from grumpy old guys. I have worked in HPC distributed storage and the problems the CoreOS people are trying to solve is a valid and present one: deploying flexible distributed storage today is an absolute pain. Plus all the arguments about local storage failure being hard are moot because this is a distributed solution and the hardship of local storage reliability can be abstracted away almost completely.
zxcvcxz 10y ago
It's threads like these where you start to wonder just how many marketing teams are arguing with each other in the comments. I take the FUD with a grain of salt: there is a lot of financial incentive to create FUD, where there is financial incentive and little to no regulations a market will naturally arise. It's going to get worse as the bubble pops and companies become more and more desperate.
- vox_mollis 10y ago
  Indeed. The upvoting of bcantrill seems to be devoid of understanding that he has been waging a personal vendetta against CoreOS for some time now, and his comments should be understood in that context.
- notacoward 10y ago
  I don't think it's marketing teams. There are at least a half dozen CoreOS employees/contributors commenting here, plus a couple of known allies. On the "other side" there's Bryan, me, and one person who seems to be an ex-employee. AFAIK none of them are in marketing, or even in coordination with marketing. Certainly, I often get flak from my company's official mouthpieces for saying things that conflict with their preferred talking points. They wish they could control what I say. Mostly I think this is a matter of people naturally standing up for their friends and colleagues, which is a wonderful thing, vs. people who have specific concerns about The Right Way to do either technical or non-technical things. No coordination or collusion is necessary. You can see exactly the same thing happen for every company or project that's discussed here, or on Twitter, or wherever. Try criticizing a YC portfolio company some time. Not all of the people arguing with you will admit their affiliations, and of course the downvotes are all anonymous anyway. Just remember, the more skin someone has in the game, the more tempted they'll be to cross that vague line into astroturf. All the rest of us can do is admit our affiliations and biases, and hope that people will get their heads out of the ad hominem gutter enough to reach rational conclusions about the facts being presented.
js4all 10y ago
Despite all the negative sentiment here, I am super excited about this. I use CoreOS heavily and really like how everything just works. Running Kubernetes on it, is the first cluster solution for me that works without configuration orgies and is robust against machine outages. Torus seems to be the missing piece. For now we use local volumes with sidecar containers for r/o storage and nfs volumes for r/w storage. All other solutions are not practical. GCE and EBS are only single mount. iSCSI is unsupported in the cloud. Leaving only Ceph and Glusterfs, both mentioned here, but needing heavy configuration.
- lisivka 10y ago
  Gluster does not need heavy configuration: just add peers and create and start volume - 3 commands. Not harder than lvm. Of course, gluster has tons of options to fine tune cluster for various kinds of loads, which is not bad, but confusing for newbies. But if you are newbie, why you need to change defaults?
  - js4all 10y ago
    That's interesting. When I was looking at it, it seemed overwhelming. Do you have some pointers for the basics?
- elsonrodriguez 10y ago
  There's lots of work being done in Ceph-Docker to make ceph easier to configure. On top of that, there's work going on to get Ceph easily deployable on Kubernetes. https://github.com/cornelius-keller/ceph-docker/tree/kubernetes/examples/kubernetes https://github.com/cornelius-keller/ceph-docker/tree/kuberne...
  - js4all 10y ago
    Many thanks, that looks doable. I'll give it a try.
- gregsfortytwo 10y ago
  Torus is also definitely single mount.
hartror 10y ago
No mention of Jepsen tests, I look forward to a @aphyr call me maybe talk/blog post detailing all the issues.
jsmthrowaway 10y ago
I love that I know who this is at CoreOS by the choice of username. Pick better throwaways if you're going to blow up a thread like this. (ideal0227 upthread is also a CoreOS employee and one of the etcd developers.) On topic, yes. It is quite trivial to lose data with etcd and pretty much everybody I know who runs it has experienced problems. Try backing it up and restoring. etcd is compsci great yet operationally burdensome; it is extremely difficult to operate, tends to make assumptions and explain to you how to operate your fleet, and the developers are not very receptive to these and other operational concerns. This stems from CoreOS culture, to be quite clear -- CoreOS seeks to eliminate operations as a discipline and replace it with software, and tends to disregard or devalue operational concerns as a result. Launching down the storage path with Go software and disregarding several decades of operational experience in the industry on how to do distributed storage correctly (not to mention the "looking down" on etcd-related infrastructure decisions from this anonymous employee) should indicate to you that I'm not making this up, despite how it may appear. 90% of folks I know on etcd, I'd estimate, have (a) reverted to Zookeeper or (b) moved on to Consul. It is the single piece of software that is holding Kubernetes back, too, and it's fairly obvious from roadmap direction that Kubernetes is now the etcd customer. Plan accordingly.
- alexandre_m 10y ago
  Cockroachdb is built on top of etcd. So far seems to work very well for their product. I think this whole thread is a bit harsh and unfair to CoreOs and their solutions.
  - gyuho 10y ago
    Disclaimer: I work on etcd at CoreOS. CockroachDB is built on top of etcd/raft package, not etcd itself.
  - vidarh 10y ago
    I think that reflects the complexity of a storage product more than CoreOS itself. I use CoreOS. I like it. I love the update mechanism, and the tight focus on running as much as possible in containers. But I also have spent enough time using it to run across certain warts, and Etcd clusters refusing to start without manual intervention etc. have been a frequent enough problem that when they go after something incredibly hard such as distributed storage, and it's relying on Etcd, I see that as a scary combination.
anonbanker 10y ago
How is this different than Tahoe-LAFS?
mrmondo 10y ago
Watch out, next we'll be seeing CoreOS writing their own encryption protocols. As someone that's designed and built storage clusters from the ground up - take it from me: storage is not as easy as it seems if you care about your data and performance at scale. The number of people I've seen using CoreOS and then moving away from it is quite alarming, I feel like CoreOS will become the Ubuntu of the container world.
- branzo 10y ago
  +1 for the assertion about the destiny of CoreOS.
- jsmthrowaway 10y ago
  They're all-in on Kubernetes, which is quite apparent given (a) who's funding them and (b) product direction, including this one. They don't have the resources or backing to go after Docker; notice that changed? Docker has something like 15x the funding and is pulling away in a lot of ways, so CoreOS hitched on the Kubernetes wagon. Better bet for destiny: Google is letting them build stuff off the books by just funding them, and at some point they'll get quietly bought with all their stuff added to Kubernetes, and that'll be that.
teajunky 10y ago
This reminds me of the SheepDog storage architecture (https://sheepdog.github.io/sheepdog/ https://sheepdog.github.io/sheepdog/) SheepDog provides safe distributed block devices and uses corosync or Zookeeper instead of etcd.
gregsfortytwo 10y ago
Okay, so from reading the developer docs at https://github.com/coreos/torus/Documentation https://github.com/coreos/torus/Documentation, I'm inferring that: * Single reader/writer. If you try and set up more, Bad Things happen (even if it's single-writer, you don't get read-after-write consistency). * It sure sounds like network partitions will allow all kinds of badness. * If copy 1 goes down, you can keep operating on copy 2, then lose copy 2, have copy 1 come back up, and then warp back in time? Maybe that's prevented because of append-only and you just lose the data entirely because it was only replicating to one node due to the "temporary" failure. Most egregiously, the Architecture description of an Inode implies that persisting a write-to-disk requires persisting the "INode". INodes are persisted in etcd. Which means your entire cluster's write-to-disk throughput is limited to what you can push through a Raft consensus algorithm. Look, there are all kinds of reasons one could legitimately decide that none of the existing scalable block storage systems satisfy your use case. Maybe containers really are different enough from VMs. But the blog post claims that there just aren't any solutions; the research papers cited in the Documentation page are mostly old and are about systems in a very different part of this sub-space; and what developer documentation exists does not encourage me that this is a good idea. Granted, I've been working on Ceph for 7 years and am a bit of a snob as a result.
- zzzcpan 10y ago
  They accept data loss, yes, and haven't figured out consistency yet. But given that it's an early prototype I would expect them to evolve and change things a lot in a couple of years, possibly throwing away raft and etcd. Good work on Ceph, by the way. I've been following your work since it was a PhD if I remember correctly.
  - gregsfortytwo 10y ago
    "haven't figured out consistency yet" is the issue. Seemingly very small decisions have huge impacts that you don't expect, and the state of the art is far enough along that you don't get better than the existing solutions except by exploiting newly-identified workload characteristics (or acceptable ways of losing data, loosening consistency, etc based on the workload) that you've planned out to make use of ahead of time. One example: Both Gluter and Ceph have erasure-coded storage. Gluster's looks just like the replicated storage, only it involves more nodes and less overhead. Ceph's is severely limited in comparison: it's append-only, it doesn't allow use of Ceph's omap kv store, "object class" embedded code, etc. The reason is because distributed EC is subject to the same kind of problem as the RAID5 write hole: if a Gluster client submits an overwrite to 3 of a 4+2 replica group and then crashes, the overwritten data is unrecoverable and the newly-written data never made it. Torus won't hit that particular issue because it is log-structured to begin with, which has all kinds of advantages. But garbage collection is really hard! Much harder than seems remotely reasonable! Getting good coalescing and read performance is really hard! Much harder than seems remotely reasonable! There's one big existing storage log-structured distributed storage system which has discussed this publicly: Microsoft Azure. They have a few papers out which hint at the contortions they went through to make block devices work performantly — and Azure writes first to 3-replica and then destages to the log! They still had performance issues! https://github.com/coreos/torus/blob/master/Documentation/research-links.md https://github.com/coreos/torus/blob/master/Documentation/re... points to a bunch of HDFS research and replacements; HDFS is designed for the opposite (large files with high bandwidth and nobody-cares latency) of what I presume Torus is targeting (high IO efficiency, low latency). Mostly the same for the Google papers they cite. There's no mention of Azure's storage system papers, nor of Ceph, nor anything about the not-paper-publishing-but-blogging stuff from Gluster or sheepdog; nor from academic research into VM storage systems (there's tons about this!). Can they fix a bunch of this? Sure. But the desires they list in eg https://github.com/coreos/torus/blob/master/Documentation/architecture.md https://github.com/coreos/torus/blob/master/Documentation/ar... go towards making things worse, not better. They aren't talking about how etcd can be in the allocator path but not the persistence path[1] and how every mount should run a repair on the data to deal with out-of-date headers. They talk about adding in filesystems, but not any way of supporting read-after-write (which is impossible with the primitives they describe so far, and really hard in a log-structured system without synchronous communication of some kind). They discuss network partitions between the storage nodes, and between the client and etcd; they don't discuss clients keeping access to the disks but losing it to etcd. [1] Using etcd for allocation would be a reasonable choice, but putting it in the persistence path is now. Right now a database in your container would require two separate write streams to do an fsync: 1) data write. It doesn't say in docs and I didn't look to see if replication is client or server-driven, but assuming sanity the network traffic is client->server1->server2->server1->client, with a write-to-disk happening before the server2->server1 step. 2) etcd write. Client->etcd master->etcd slaves (2+)->etcd master->client, with a disk write to each etcd process' disk before the reply to etcd master. This is a busy-neighbor, long-tail latency disaster waiting to happen.
kingofhawks 10y ago
Is it possible that Facebook haystack will be open source?
frozenice 10y ago
Funny, just this week I was looking for an easy solution to cluster storage for containers. I found SXCluster and posted it here https://news.ycombinator.com/item?id=11812566 https://news.ycombinator.com/item?id=11812566 - not directly geared towards containers, but very easy to setup (no need for etcd and the like).
JulieONeilEMC 10y ago
You could also check out EMC's Unity midrange storage at the EMC Store. http://bit.ly/1SIZ9N7 http://bit.ly/1SIZ9N7

267 points by philips 10y ago

The CoreOS team is excited to make this initial release and start collaborating with folks that want to tackle distributed storage. tl;dr this is a new OSS distributed storage project that is written in Go and backed by etcd for consistency. The first project built on top of Torus is a network block device that can be mounted into containers for persistent storage. It also includes integrations out of the box for Kubernetes "flex volumes". Get involved! :) https://github.com/coreos/torus https://github.com/coreos/torus

paulsutter 10y ago
What's the current largest-scale deployment?
- lobster_johnson 10y ago
  It's described as a prototype, so I'm pretty sure there's no deployment at scale at all.
stevelandis 10y ago
Brandon, a few months ago I met you at a meetup and you did not think this problem needed to be solved, and containers were for ephemeral apps... just an AMA request, what changed hearts?

CSDude 10y ago

Seems really nice. Would love to see it how it compares to Ceph, we were considering Ceph for our container storage. For starters, it seems much more simpler to deploy.

Inflatablewoman 10y ago
In my old firm we managed to get ceph running on coreos using ceph in docker. The storage was exposed using a s3 gateway. https://github.com/ceph/ceph-docker https://github.com/ceph/ceph-docker
- ymse 10y ago
  Docker supports native RBD since version 1.8. Which should be both faster and more reliable than using the S3 gateway (though that has other benefits and becoming pretty good too).
kinvaris 10y ago
I work at a company where object storage is something they have worked on for many years to get it stable. You can download it for free (it's Open Source)! Open vStorage is very easy to deploy through Ansible! Try it :)! https://www.openvstorage.com/ https://www.openvstorage.com/ https://github.com/openvstorage https://github.com/openvstorage

xfactor973 10y ago

I don't understand why this is being used over say librados + librbd from Ceph. This seems like an awful lot of work to get the same functionality that Gluster/Ceph already have. Is there something I'm missing here?

bogomipz 10y ago
I kind of see this as a general trend where CoreOS is concerned they reimplement everything in house, even when open source solution might already exist - rkt, etcd, now this.
- zsmith928 10y ago
  what is the existing alternative for storage in distributed container environments?
  - deleted 10y ago
    [deleted]
  - gsmethells 10y ago
    Ceph, OpenStack Swift, GlusterFS, OrangeFS, Lustre. https://en.wikipedia.org/wiki/Clustered_file_system#Distributed_file_systems https://en.wikipedia.org/wiki/Clustered_file_system#Distribu... https://en.wikipedia.org/wiki/Object_storage https://en.wikipedia.org/wiki/Object_storage
    - pinewurst 10y ago
      If Lustre is the answer, you're probably asking the wrong question. Unless that question involves short life-span, massively parallel swap-outs from one weapons simulation to another. I write this not as storage religion (of which there's far too much), but to warn away those who haven't experienced the many kinds of data (and stomach lining) loss that come with being a Lustre admin.
      - 10y ago

amq 10y ago

I just hope the performance is visibly better than of gluster. Especially for small files.

zsmith928 10y ago
are there any performance metrics for Tarus yet?

koolba 10y ago

So is this the building blocks for an in-house EBS?

philips 10y ago
Yes, that is the idea. Torus block is like EBS. Torus "library" could be used by other applications like an append only log, object store, etc.
- stevelandiss 10y ago
  How do you say that with conviction? I looked at the code and it has no bearings of being able to build a fault tolerant elastic block storage system.

gsmethells 10y ago

Is the goal to provide POSIX file access as NFS and AFS do? The mention of object storage as a future direction feels like the early "the sky's the limit" euphoria of a new project that has yet to adopt specific end goals.

barakm 10y ago
More than "sky's the limit" -- early versions had POSIX access, though it was terribly messy. We know the architecture can support it, it's just a matter of learning from the mistakes and building something worth supporting.
philips 10y ago
The project doesn't provide a POSIX filesystem directly. But, instead provides a block device (think AWS EBS). The initial goals that are in this release: 1. An abstraction for replicated storage between machines in a cluster. 2. A block device "application" on top that an ext4 filesystem could be run on. This is like an EBS. In the future someone might build other applications like "object storage" or filesystems and we would love to get the feedback on the API to do that. But, that isn't in the initial goals and we will be focusing on the storage and block layers for the time being.
- gsmethells 10y ago
  Will each disk in the system be weighted so that different size disks can be used in the system over time? OpenStack Swift offers this and is useful to not lock-in disk size at design time.
  - philips 10y ago
    Yes, see the admin documentation: https://github.com/coreos/torus/blob/master/Documentation/admin-guide.md#modify-my-cluster https://github.com/coreos/torus/blob/master/Documentation/ad...

agentultra 10y ago

Where's the formal specification of the system? If we want these distributed systems to be reliable how can we be sure our algorithms and processes work if we do not first model them? I'd be happy to work on a specification with the community.

philips 10y ago
We haven't gotten that far down the path. We would love the help. I believe there are some issues related to documenting the architecture and failure domains for the v0.1.1 release: https://github.com/coreos/torus/milestones/v0.1.1 https://github.com/coreos/torus/milestones/v0.1.1 This is one of those classic "release too early or release too late" sort of things that got cut in preference to get early feedback and community participation.
- subway 10y ago
  Coming from the company that raised hell about Docker's lack of a spec, this is an interesting approach.

alrs 10y ago

Storing container images is a great use case for an object store, not a block store. They should get sucked down whole to ephemeral storage on the compute nodes. Mounting containers to a distributed block system is an anti-pattern. This is going to go poorly. Ceph has had some really smart people working on distributed block for a lot of years, and they still have significant issues. It's not because they're dumb, it's because performant, scalable, and available distributed block is either hard or impossible.

sagichmal 10y ago
This is about providing data volumes to containers, not about hosting container images or mounting containers themselves.

sysexit 10y ago

I love CoreOS and they've done some super impressive engineering. But really, a new storage system? Rewriting in Go and using etcd for central state management makes things easier, but this is still a hard problem. Some things that need to be solved sooner or later: data replication so that N faults of X entities are protected against (X can be disks, enclosures, racks, data centers, regions, ..), recovery from failed disks, scrubbing, data management, backups, some kind of storage orchestration and centralized management. If you look at Ceph which IMHO represents the state of the art in software defined storage, it took many years to get it to a point that it was usable. I hate to be cynical but in this case I would be surprised if CoreOS can pull this of. I wish them all the best though and would be happy for them if I am wrong.

agentultra 10y ago
> this is still a hard problem Which is why I'm rather surprised that these systems continue to be released without any formal specification or model checking. It's hard to get these things right... worth the effort, IMO, to create formal specifications.
ams6110 10y ago
It's like any time we take a step forward in one area we have to reinvent the last 50 years of computing to support it. Persistent storage is a "hard problem" Really?
- vidarh 10y ago
  Persistent storage isn't a hard problem. Distributed, well performing, scalable and consistent storage is a hard problem.
  - dap 10y ago
    Actually, persistent storage is fairly hard in itself. Look at what ZFS does to ensure data integrity in the face of phantom writes, dropped writes, bad controllers, and other implicit, non-fatal failures.

dap 10y ago

> At its core, Torus is a library with an interface that appears as a traditional file, allowing for storage manipulation through well-understood basic file operations. Coordinated and checkpointed through etcd’s consensus process, this distributed file can be exposed to user applications in multiple ways. Today, Torus supports exposing this file as block-oriented storage via a Network Block Device (NBD). We also expect that in the future other storage systems, such as object storage, will be built on top of Torus as collections of these distributed files, coordinated by etcd. Am I understanding correctly that this is a file-based API? Distributing a POSIX filesystem effectively is very challenging, particularly since most applications that use them aren't written with CAP in mind; they don't expect a lot of basic operations to block for extended periods and fail in surprising ways, and they very often perform poorly when operations that are locally quick end up much slower over a network. To be concrete: > Today’s Torus release includes manifests using this feature to demonstrate running the PostgreSQL database server atop Kubernetes flex volumes, backed by Torus storage. It will be interesting to see how well this performs and how it behaves in the face of single-node failures and network congestion.

philips 10y ago
It isn't a POSIX filesystem API. It is like a single file or a big distributed tape.
- dap 10y ago
  Virtualized, network block devices have all the same problems I described -- even worse, because the abstraction coneys even less about what an application is trying to do.
  - ibotty 10y ago
    No. The difference is, that with network block devices you are usually only allow accessing the block device once. That's an easier problem than mapping POSIX file system semantics!

bcantrill 10y ago

It's hard to take this seriously: storage is an excruciatingly hard problem, yet this cheerful description of a nascent and aspirational effort seems blissfully unaware of how difficult it is to even just reliably get bits to and from stable storage, let alone string that into a distributed system that must make CAP tradeoffs. There is not so much of a whisper as to what the data path actually looks like other than "the design includes the ability to support [...] Reed-Solomon error correction in the near future" -- and the fact that such an empty system hails itself as pioneering an unsolved problem in storage is galling in its ignorance of prior work (much of it open source). Take it from someone who has been involved in both highly durable local filesystems[1] and highly available object storage systems[2][3]: this is such a hard, nasty problem with so many dark, hidden and dire failure modes, that it takes years of running in production to get these systems to the level of reliability and operability that the data path demands. Given that (according to the repo, though not the breathless blog entry) its creators "do not recommend its use in production", Torus is -- in the famous words of Wolfgang Pauli -- not even wrong. [1] http://dtrace.org/blogs/bmc/2008/11/10/fishworks-now-it-can-be-told/ http://dtrace.org/blogs/bmc/2008/11/10/fishworks-now-it-can-... [2] http://dtrace.org/blogs/bmc/2013/06/25/manta-from-revelation-to-product/ http://dtrace.org/blogs/bmc/2013/06/25/manta-from-revelation... [3] http://dtrace.org/blogs/dap/2013/07/03/fault-tolerance-in-manta/ http://dtrace.org/blogs/dap/2013/07/03/fault-tolerance-in-ma...

nickpsecurity 10y ago
I totally agree with you. I also liked how they said the motivation is to make Google infrastructure for everyone else. How did Google do this? They basically imitated and improved on clustered filesystems developed in HPC. There were a lot of lessons to learn for emerging cloud market in all tooling done in HPC. Some was FOSS. Whereas, many companies seem to be doing the opposite in their work on these cloud filesystems. They don't build on proven, already-OSS components that have been battle-tested for a long time. They lack features and wisdom from prior deployments. They duplicate effort. They are also prone to using popular components, languages, whatever with implicitly-higher risk due to fact they were never built for fault-tolerant systems. Actually, many of them often assume something will watch and help them out in time of failure. If it's high-assurance or fault-tolerance, my manta is "tried and true beats novel and new." Just repurpose what's known to work while improving its capabilities and code quality. Knocks out risks you know about plus others you don't since they were never documents. Has that been your experience, too? Should be a maxim in IT given how often problem plays out.

ceocoder 10y ago

Congrats on the relaase! How does this compare to GlusterFS (loved it, was free - I used it before they got acquired by RedHat) and Isilon (expensive but fast and reliable). Is there an upper limit on number of nodes(?) or total size?

nwmcsween 10y ago

So a userspace fs, I'm guessing this will use fuse to actually expose a POSIX fs? Small sync writes will absolutely kill performance and will require all sorts of hacks like glusterfs has had to implement due to the amount of context switches. CoreOS really should have added whatever needed to ceph, filesystems are not something you just hack together overnight.

betawaffle 10y ago
We had a POSIX interface early on (via FUSE), but decided to expose a a block storage interface first instead. This is not a "filesystem", it's a storage abstraction, and we've spent more than 6 months on it. Seems like a lot of folks are quickly jumping to conclusions, which is expected from "the internet", but I would have hoped for better from HN.
- zzzcpan 10y ago
  > but I would have hoped for better from HN Don't worry about it. Some guys just mistake their experience with traditional storage systems as meaningful here.
- nwmcsween 10y ago
  6 months is overnight in terms of a filesystem, there are literally millions of dollars of work that was done with glusterfs and probably a magnitude less man hours. I understand coreos wants.to innovate but what does torus bring to the table besides negatives and NIH?

nzoschke 10y ago

I'm always in awe and grateful for the teams like CoreOS that tackle these tough problems. But when it comes to container storage I'm confused. The 12 factor app methodologies work very well for every service I have written and supported. Processes/containers get ephemeral storage. When you need persistence you delegate to a stateful service, generally Postgres and S3. Id much rather write my apps around these simple and well understood constraints then depend on magic file systems.

ecnahc515 10y ago
The average 12 factor app doesn't really need this, but as you mentioned, sometimes you delegate to Postgres, or S3, and today those are available as hosted products via Amazon, Google, Heroku, etc. However, when your running in your own datacenter, you don't have the luxury of using Amazon's hosted products, so Torus exists to help provide some building blocks to build your own S3, or potentially even RDS.

joshuak 10y ago

Good... Good... Let the hate flow through you. An open source OS company just blogged about a new open source project that they are putting resources into. They did not release a commercial product that competes with any existing storage solution. How exactly would one expect a new project to be announced? Tell me more about how far away from done you think they are. Sorry if that's a bit sarcastic, but seriously wouldn't it be nice if these comments where more along the lines of: "Interesting new project, here's some issues we've run into vis a vis reliable storage that you should look out for..."

abrookewood 10y ago
Yes, I have to agree. There's a little too much negativity for my liking, especially from people who have a vested interest in downplaying the efforts of others.
stevelandis 10y ago
IMO it is because this can derail the adoption of containers. I think an immature project like this announced so prematurely does more harm than good. And I appreciate people that have experience with this bringing up the real issues.
pjmlp 10y ago
I for one find it very positive, a storage engine implemented in Go, for what I could gather in a quick Github glance. For me, this is surely systems work.
mmf 10y ago
I absolutely agree, all those negative comments sound like rants from grumpy old guys. I have worked in HPC distributed storage and the problems the CoreOS people are trying to solve is a valid and present one: deploying flexible distributed storage today is an absolute pain. Plus all the arguments about local storage failure being hard are moot because this is a distributed solution and the hardship of local storage reliability can be abstracted away almost completely.

zxcvcxz 10y ago

It's threads like these where you start to wonder just how many marketing teams are arguing with each other in the comments. I take the FUD with a grain of salt: there is a lot of financial incentive to create FUD, where there is financial incentive and little to no regulations a market will naturally arise. It's going to get worse as the bubble pops and companies become more and more desperate.

vox_mollis 10y ago
Indeed. The upvoting of bcantrill seems to be devoid of understanding that he has been waging a personal vendetta against CoreOS for some time now, and his comments should be understood in that context.
notacoward 10y ago
I don't think it's marketing teams. There are at least a half dozen CoreOS employees/contributors commenting here, plus a couple of known allies. On the "other side" there's Bryan, me, and one person who seems to be an ex-employee. AFAIK none of them are in marketing, or even in coordination with marketing. Certainly, I often get flak from my company's official mouthpieces for saying things that conflict with their preferred talking points. They wish they could control what I say. Mostly I think this is a matter of people naturally standing up for their friends and colleagues, which is a wonderful thing, vs. people who have specific concerns about The Right Way to do either technical or non-technical things. No coordination or collusion is necessary. You can see exactly the same thing happen for every company or project that's discussed here, or on Twitter, or wherever. Try criticizing a YC portfolio company some time. Not all of the people arguing with you will admit their affiliations, and of course the downvotes are all anonymous anyway. Just remember, the more skin someone has in the game, the more tempted they'll be to cross that vague line into astroturf. All the rest of us can do is admit our affiliations and biases, and hope that people will get their heads out of the ad hominem gutter enough to reach rational conclusions about the facts being presented.

js4all 10y ago

Despite all the negative sentiment here, I am super excited about this. I use CoreOS heavily and really like how everything just works. Running Kubernetes on it, is the first cluster solution for me that works without configuration orgies and is robust against machine outages. Torus seems to be the missing piece. For now we use local volumes with sidecar containers for r/o storage and nfs volumes for r/w storage. All other solutions are not practical. GCE and EBS are only single mount. iSCSI is unsupported in the cloud. Leaving only Ceph and Glusterfs, both mentioned here, but needing heavy configuration.

lisivka 10y ago
Gluster does not need heavy configuration: just add peers and create and start volume - 3 commands. Not harder than lvm. Of course, gluster has tons of options to fine tune cluster for various kinds of loads, which is not bad, but confusing for newbies. But if you are newbie, why you need to change defaults?
- js4all 10y ago
  That's interesting. When I was looking at it, it seemed overwhelming. Do you have some pointers for the basics?
elsonrodriguez 10y ago
There's lots of work being done in Ceph-Docker to make ceph easier to configure. On top of that, there's work going on to get Ceph easily deployable on Kubernetes. https://github.com/cornelius-keller/ceph-docker/tree/kubernetes/examples/kubernetes https://github.com/cornelius-keller/ceph-docker/tree/kuberne...
- js4all 10y ago
  Many thanks, that looks doable. I'll give it a try.
gregsfortytwo 10y ago
Torus is also definitely single mount.

hartror 10y ago

No mention of Jepsen tests, I look forward to a @aphyr call me maybe talk/blog post detailing all the issues.

jsmthrowaway 10y ago

I love that I know who this is at CoreOS by the choice of username. Pick better throwaways if you're going to blow up a thread like this. (ideal0227 upthread is also a CoreOS employee and one of the etcd developers.) On topic, yes. It is quite trivial to lose data with etcd and pretty much everybody I know who runs it has experienced problems. Try backing it up and restoring. etcd is compsci great yet operationally burdensome; it is extremely difficult to operate, tends to make assumptions and explain to you how to operate your fleet, and the developers are not very receptive to these and other operational concerns. This stems from CoreOS culture, to be quite clear -- CoreOS seeks to eliminate operations as a discipline and replace it with software, and tends to disregard or devalue operational concerns as a result. Launching down the storage path with Go software and disregarding several decades of operational experience in the industry on how to do distributed storage correctly (not to mention the "looking down" on etcd-related infrastructure decisions from this anonymous employee) should indicate to you that I'm not making this up, despite how it may appear. 90% of folks I know on etcd, I'd estimate, have (a) reverted to Zookeeper or (b) moved on to Consul. It is the single piece of software that is holding Kubernetes back, too, and it's fairly obvious from roadmap direction that Kubernetes is now the etcd customer. Plan accordingly.

alexandre_m 10y ago
Cockroachdb is built on top of etcd. So far seems to work very well for their product. I think this whole thread is a bit harsh and unfair to CoreOs and their solutions.
- gyuho 10y ago
  Disclaimer: I work on etcd at CoreOS. CockroachDB is built on top of etcd/raft package, not etcd itself.
- vidarh 10y ago
  I think that reflects the complexity of a storage product more than CoreOS itself. I use CoreOS. I like it. I love the update mechanism, and the tight focus on running as much as possible in containers. But I also have spent enough time using it to run across certain warts, and Etcd clusters refusing to start without manual intervention etc. have been a frequent enough problem that when they go after something incredibly hard such as distributed storage, and it's relying on Etcd, I see that as a scary combination.

anonbanker 10y ago

How is this different than Tahoe-LAFS?

mrmondo 10y ago

Watch out, next we'll be seeing CoreOS writing their own encryption protocols. As someone that's designed and built storage clusters from the ground up - take it from me: storage is not as easy as it seems if you care about your data and performance at scale. The number of people I've seen using CoreOS and then moving away from it is quite alarming, I feel like CoreOS will become the Ubuntu of the container world.

branzo 10y ago
+1 for the assertion about the destiny of CoreOS.
jsmthrowaway 10y ago
They're all-in on Kubernetes, which is quite apparent given (a) who's funding them and (b) product direction, including this one. They don't have the resources or backing to go after Docker; notice that changed? Docker has something like 15x the funding and is pulling away in a lot of ways, so CoreOS hitched on the Kubernetes wagon. Better bet for destiny: Google is letting them build stuff off the books by just funding them, and at some point they'll get quietly bought with all their stuff added to Kubernetes, and that'll be that.

teajunky 10y ago

This reminds me of the SheepDog storage architecture (https://sheepdog.github.io/sheepdog/ https://sheepdog.github.io/sheepdog/) SheepDog provides safe distributed block devices and uses corosync or Zookeeper instead of etcd.

gregsfortytwo 10y ago

Okay, so from reading the developer docs at https://github.com/coreos/torus/Documentation https://github.com/coreos/torus/Documentation, I'm inferring that: * Single reader/writer. If you try and set up more, Bad Things happen (even if it's single-writer, you don't get read-after-write consistency). * It sure sounds like network partitions will allow all kinds of badness. * If copy 1 goes down, you can keep operating on copy 2, then lose copy 2, have copy 1 come back up, and then warp back in time? Maybe that's prevented because of append-only and you just lose the data entirely because it was only replicating to one node due to the "temporary" failure. Most egregiously, the Architecture description of an Inode implies that persisting a write-to-disk requires persisting the "INode". INodes are persisted in etcd. Which means your entire cluster's write-to-disk throughput is limited to what you can push through a Raft consensus algorithm. Look, there are all kinds of reasons one could legitimately decide that none of the existing scalable block storage systems satisfy your use case. Maybe containers really are different enough from VMs. But the blog post claims that there just aren't any solutions; the research papers cited in the Documentation page are mostly old and are about systems in a very different part of this sub-space; and what developer documentation exists does not encourage me that this is a good idea. Granted, I've been working on Ceph for 7 years and am a bit of a snob as a result.

zzzcpan 10y ago
They accept data loss, yes, and haven't figured out consistency yet. But given that it's an early prototype I would expect them to evolve and change things a lot in a couple of years, possibly throwing away raft and etcd. Good work on Ceph, by the way. I've been following your work since it was a PhD if I remember correctly.
- gregsfortytwo 10y ago
  "haven't figured out consistency yet" is the issue. Seemingly very small decisions have huge impacts that you don't expect, and the state of the art is far enough along that you don't get better than the existing solutions except by exploiting newly-identified workload characteristics (or acceptable ways of losing data, loosening consistency, etc based on the workload) that you've planned out to make use of ahead of time. One example: Both Gluter and Ceph have erasure-coded storage. Gluster's looks just like the replicated storage, only it involves more nodes and less overhead. Ceph's is severely limited in comparison: it's append-only, it doesn't allow use of Ceph's omap kv store, "object class" embedded code, etc. The reason is because distributed EC is subject to the same kind of problem as the RAID5 write hole: if a Gluster client submits an overwrite to 3 of a 4+2 replica group and then crashes, the overwritten data is unrecoverable and the newly-written data never made it. Torus won't hit that particular issue because it is log-structured to begin with, which has all kinds of advantages. But garbage collection is really hard! Much harder than seems remotely reasonable! Getting good coalescing and read performance is really hard! Much harder than seems remotely reasonable! There's one big existing storage log-structured distributed storage system which has discussed this publicly: Microsoft Azure. They have a few papers out which hint at the contortions they went through to make block devices work performantly — and Azure writes first to 3-replica and then destages to the log! They still had performance issues! https://github.com/coreos/torus/blob/master/Documentation/research-links.md https://github.com/coreos/torus/blob/master/Documentation/re... points to a bunch of HDFS research and replacements; HDFS is designed for the opposite (large files with high bandwidth and nobody-cares latency) of what I presume Torus is targeting (high IO efficiency, low latency). Mostly the same for the Google papers they cite. There's no mention of Azure's storage system papers, nor of Ceph, nor anything about the not-paper-publishing-but-blogging stuff from Gluster or sheepdog; nor from academic research into VM storage systems (there's tons about this!). Can they fix a bunch of this? Sure. But the desires they list in eg https://github.com/coreos/torus/blob/master/Documentation/architecture.md https://github.com/coreos/torus/blob/master/Documentation/ar... go towards making things worse, not better. They aren't talking about how etcd can be in the allocator path but not the persistence path[1] and how every mount should run a repair on the data to deal with out-of-date headers. They talk about adding in filesystems, but not any way of supporting read-after-write (which is impossible with the primitives they describe so far, and really hard in a log-structured system without synchronous communication of some kind). They discuss network partitions between the storage nodes, and between the client and etcd; they don't discuss clients keeping access to the disks but losing it to etcd. [1] Using etcd for allocation would be a reasonable choice, but putting it in the persistence path is now. Right now a database in your container would require two separate write streams to do an fsync: 1) data write. It doesn't say in docs and I didn't look to see if replication is client or server-driven, but assuming sanity the network traffic is client->server1->server2->server1->client, with a write-to-disk happening before the server2->server1 step. 2) etcd write. Client->etcd master->etcd slaves (2+)->etcd master->client, with a disk write to each etcd process' disk before the reply to etcd master. This is a busy-neighbor, long-tail latency disaster waiting to happen.

kingofhawks 10y ago

Is it possible that Facebook haystack will be open source?

frozenice 10y ago

Funny, just this week I was looking for an easy solution to cluster storage for containers. I found SXCluster and posted it here https://news.ycombinator.com/item?id=11812566 https://news.ycombinator.com/item?id=11812566 - not directly geared towards containers, but very easy to setup (no need for etcd and the like).

JulieONeilEMC 10y ago

You could also check out EMC's Unity midrange storage at the EMC Store. http://bit.ly/1SIZ9N7 http://bit.ly/1SIZ9N7