13 ms·
Jepsen: Etcd 3.4.3
- hprotagonist 7y agoThe corresponding jepsen post: http://jepsen.io/analyses/etcd-3.4.3 http://jepsen.io/analyses/etcd-3.4.3 I do no work at all in this area, but i love these reports. They're examples of well-written, clear, "engineer-mind" reports that we would all do well to emulate.
- dang 7y ago(This comment was originally posted to https://news.ycombinator.com/item?id=22191925 https://news.ycombinator.com/item?id=22191925. We merged the threads.)
- thom 7y agoI completely agree. I have also always appreciated the very honest and reflective ethics policy that underlies the work: https://jepsen.io/ethics https://jepsen.io/ethics
- cpitman 7y agoI agree they are great. I kind of miss the older style of reports though, with memes all over the place!
- jinqueeny 7y agoFor the impatient: “The etcd key-value store is a distributed database based on the Raft consensus algorithm. In our 2014 analysis, we found that etcd 0.4.1 exhibited stale reads by default. We returned to etcd, now at version 3.4.3, to investigate its safety properties in detail. We found that key-value operations appear to be strict serializable, and that watches deliver every change to a key in order. However, etcd locks are fundamentally unsafe, and those risks were exacerbated by a bug which failed to check lease validity after waiting for a lock.”
- nif2ee 7y agoHow many nodes can etcd handle without having noticeable decay in performance? their FAQ says 7 but did somebody use it in some other distributed app other than k8s with more nodes? Assume that most of the operations are get and watch (i.e. write/read <<< 1.0), how big of a cluster in terms of number nodes can we scale up to?
- candiddevmike 7y agoNot sure why youre looking at so many nodes for a cluster. Most of the time scaling the number of etcd nodes does little to help performance, instead focus on giving the nodes plenty of IOPS.
- jzoch 7y agoIf you run a 3 availability zone architecture and want to survive AZ + 1 failures you need more nodes. 9 nodes is the minimum to guarentee that survival while still maintaing a quorum
- sethev 7y agoDo you think you're actually gaining availability with an AZ + 1 setup?
- danaur 7y agoComon, could you be constructive? Of course they do. Now whats your gripe with that?
- sethev 7y agoI was more curious on why they thought that would increase availability. It didn't seem like it would be constructive to speculate about why they chose that. AWS doesn't guarantee that failures within an AZ are independent of each other, so it's not clear how you would estimate what availability you'd gain with this. Losing everything in an AZ + 1 instance sounds like a very unusual and specific scenario to design for.
- dang 7y agoThere's also https://etcd.io/blog/jepsen-343-results/ https://etcd.io/blog/jepsen-343-results/, via https://news.ycombinator.com/item?id=22191925 https://news.ycombinator.com/item?id=22191925.
- deleted 7y ago[deleted]
- candiddevmike 7y agoThis is a powerful validation for etcd and its status as a mission critical backend. You don't see a lot of positive Jespen results these days!
- shaklee3 7y agoI didn't think it was too positive. He found some documentation bugs, as well as what seems like really bad locking bugs.
- nl 7y agoHave you seen Jepsen reports before? Silent data loss isn't uncommon. https://aphyr.com/posts/284-call-me-maybe-mongodb https://aphyr.com/posts/284-call-me-maybe-mongodb ("Mostly, it just throws those writes away entirely: no rollback files, no nothing. I don’t really know why.") https://aphyr.com/posts/317-call-me-maybe-elasticsearch https://aphyr.com/posts/317-call-me-maybe-elasticsearch ("When the cluster comes back together, one primary blithely overwrites the other’s state.") etc etc
- senderista 7y agoIt is not possible to implement distributed locking correctly in the general case. Fencing tokens only work if your data store supports RMW operations, in which case you could just implement locking inside the data store itself.
- continuations 7y agoI mostly see etcd being used to store metadata & configuration data for distributed systems. Can etcd be also used as a general distributed database like FoundationDB or ScyllaDB? If so how does it compare to those other optiions?
- philips 7y agoetcd is used as the primary data store of applications like Kubernetes or CoreDNS. But, many use it for configuration like M3 or Vitess. You can see different users in the docs[1]. There are different databases like CockroachDB or Dgraph that use etcd's raft libraries but build more application focused (SQL, graphql, etc) APIs[2]. [1]: https://github.com/etcd-io/etcd/blob/master/Documentation/integrations.md https://github.com/etcd-io/etcd/blob/master/Documentation/in... [2]: https://github.com/etcd-io/etcd/tree/master/raft#notable-users https://github.com/etcd-io/etcd/tree/master/raft#notable-use...
- justnoise 7y agoI don't think etcd was designed with the idea of competing with larger distributed databases. For example, the maximum database size limit in etcd (as of v3.3) 10GB. This works for an application like Kubernetes where you're storing less than a million records but likely isn't something you want backing your wildly successful django application.
- gregwebs 7y agoEtcd is designed to be able to cache it's entire working set in memory. https://etcd.io/docs/v3.3.12/dev-guide/limit/ https://etcd.io/docs/v3.3.12/dev-guide/limit/ https://github.com/etcd-io/etcd/blob/master/Documentation/op-guide/hardware.md#memory https://github.com/etcd-io/etcd/blob/master/Documentation/op... TiKV, a distributed kv store that can store many terabytes of data, actually uses etcd internally for metadata storage.
- fulafel 7y agoMost databases fit in memory though.
- Gepsens 7y agoFor me, etcd is the go to for dynamic configuration management.
- jzelinskie 7y agoCongrats on the community blog launch! <3 >We would love to see someone in the etcd community integrate the etcd Jepsen tests directly into the existing etcd release pipeline. I consider this to a be an issue of higher priority than any of the bugs they just found, because this will ensure preventable bugs don't crop up in the future. It's shocking to me that Jepson goes through all the effort and than very few projects build a permanent pipeline for it. It's debatable these bugs would've existed if a Jepson pipeline had been consistently in use from the 0.4.x days. I'm sure it's no simple task, but neither is a lot of the existing testing infrastructure for etcd.
- aphyr 7y ago> It's debatable these bugs would've existed if a Jepson pipeline had been consistently in use from the 0.4.x days. I don't think it would have helped: the Jepsen tests I wrote in 2014 only checked single-key gets, puts, and CaS operations; the problems we found in this report were in watches and locks.
- jzelinskie 7y agoThat's a good point that I hadn't remembered (first-class locks weren't totally baked back then), but I think the intent of my original comment holds true: we should have valued the Jepson test suite more and continuously leveraged and improved our usage of it, rather than doing one-off tests every now and then. As a result, the community has no idea if there were regressions between now and then. I will admit what I don't know: I have no idea if this was actually feasible or desirable for us or you at the time, but I'd feel more comfortable if every release of etcd, Zookeeper, etc... had some kind of Jepson stamp of approval on it in terms of API coverage. I'm pretty sure CockroachDB has tried this[0], but it's been a few years and I don't know how it turned out for them long term. [0]: https://www.cockroachlabs.com/blog/diy-jepsen-testing-cockroachdb/ https://www.cockroachlabs.com/blog/diy-jepsen-testing-cockro...
- irfansharif 7y agoCockroachDB runs the Jepsen test suite nightly. We've been following along Aphyr's recent test additions (`multi-register` for instance, which immediately caught [0]), porting them over when appropriate. We definitely have work to be done incorporating the more DDL focused tests that tripped up YugaByte. [0]: https://github.com/cockroachdb/cockroach/pull/40600 https://github.com/cockroachdb/cockroach/pull/40600
- kbenson 7y agoThis is, apparently, not correct. Asking for revision 0 causes etcd to stream updates beginning with whatever revision the server has now, plus one, rather than the first revision. Asking for revision 1 yields all changes. This behavior was not documented. Whoops, looks suspiciously like someone tested the revision integer for truthiness to see if something was passed.
- Nullabillity 7y agoWorse, this is apparently an intentional choice.
- lvh 7y agoNope. It's because protobufs can't differentiate between nil/unset and a zero value in this field: https://github.com/etcd-io/etcd/blob/53f15caf73b9285d6043009fa64c925d5a8f573c/etcdserver/etcdserverpb/rpc.proto#L684 https://github.com/etcd-io/etcd/blob/53f15caf73b9285d6043009...
- m0zg 7y agoNot all protobufs. Protobuf 2 can do it just fine. The decision to not have emptiness support is the dumbest part of Proto 3 IMO.
- dilap 7y agoI don't think it's that crazy a decision, since it was also kind of crazy to pay the cost of "was this field set" for every single field, when the vast majority of the time you're not going to use it. But it can require a bit more more thought in your protocol design. So for example in this case, the options would be - design things such that 0 is ok to mean "most current" (so start revisions at 1) (this is hard after the fact, but if you know from day 0 that missing values for int types will be 0, you can design everything to start at 1) (Edit: maybe this is how etcd works?) - explicitly break out revision into a message type (so you can notice if it's not provided) - use something like "-1" to mean "now", so that 0 isn't overloaded etc... (You could argue maybe the right call for proto3 was to have a flag on a field saying if you want to be able to notice if it was provided or not. Best of all worlds, at cost of a bit of complexity.)
- kodablah 7y ago> This is, apparently, not correct. Asking for revision 0 causes etcd to stream updates beginning with whatever revision the server has now, plus one, rather than the first revision. Asking for revision 1 yields all changes. This behavior was not documented. I had worked on an alternative etcd impl and had to workaround this assumption as well. It is technically documented in the proto[0], and numeric 0 is of course "unset" or "default" in proto3 land. One thing I would like to see tested is nested transactions where one txn child mutates something then the second sibling txn child uses that something. I've found that implementation is lacking. 0 - https://github.com/etcd-io/etcd/blob/53f15caf73b9285d6043009fa64c925d5a8f573c/etcdserver/etcdserverpb/rpc.proto#L684 https://github.com/etcd-io/etcd/blob/53f15caf73b9285d6043009...
- aphyr 7y agoThis is also one of the things I suggested that'd be nice to have in the etcd API, along with an AST for boolean operations over guard expressions, more flexible comparators etc. You can emulate these with a sequence of independent transactions, but it'd be nice if the micro-transaction system were a tad more general.
- jascii 7y agoGoing off the name I thought it would be something to manage Unix configurations (/etc Daemon)
- klysm 7y agoIt’s kind of used for things of that nature in a distributed context
- shaklee3 7y agoAs usual, Kyle provides an awesome write-up. As much as I miss the old, funny, prose, the level of detail is still unmatched.
- senderista 7y agoThe elementary confusions apparent in the quoted documentation do not inspire confidence in the design or implementation.
- jupp0r 7y agoIf you look at other reports by Jepsen (elasticsearch and mongodb are real gems), you might find that this one found very few very minor bugs and documentation problems. Etcd is pretty solid for what it does. edit: seems like my recollection of the mongodb one is the 2.4.x one [1] and the later ones are much better. Also mongo included the Jepsen test suite into their CI. [1] https://aphyr.com/posts/284-call-me-maybe-mongodb https://aphyr.com/posts/284-call-me-maybe-mongodb