armon

9 ms·

31.
by armon 12y ago
We did take a look at D(TLS) when implementing Serf and the associated encryption mechanism. D(TLS) is much better suited when you are doing point-to-point duplex communication. The gossip algorithm instead is doing peer-to-peer (N-to-N ins
32.
by armon 12y ago
I'm also curious how this compares to Celery with rate limiting. One advantage of Celery is that you can use an AMQP broker which is probably better suited than Redis if you care about durability and at-least-once delivery.
33.
by armon 12y ago
We cover why this is not done in the article, namely location transparency (our files aren't always in S3), and partial-failures. A simple client-side uploader like this doesn't handle cases like the upload failing part way throug
34.
by armon 12y ago
Full disclosure, I'm a HashiCorp employee, but another system to look at is Consul: http://www.consul.io . It is compared to ZooKeeper more extensively here: http://www.consul.io/intro/vs/zookeeper.
35.
by armon 12y ago
My previous job was at an advertising firm, and we used HyperLogLogs for almost all of our real-time analytics infrastructure. They are incredibly space and time efficient. Each "counter" fits into about a single page of memory, a
36.
by armon 12y ago
Unless you develop natively on Linux, Docker must still run inside of a VM, so there is no real battery savings.
37.
by armon 12y ago
I think that Synapse already provides a plugin architecture. I t should be trivial to use the Consul HTTP API as a Synapse plugin. All this does is bridge the information Consul has and setup an HAProxy instance locally. If you want to go t
38.
by armon 12y ago
The two offer very different trade offs and have different architectures. Serf is an AP system, meaning it trades of consistency for availability. 90% of your nodes can be down and a Serf cluster will continue to operate. It does offer a pr
39.
by armon 12y ago
To reiterate a bit on what hardwaresofton has already said, yes you could use Cassandra. However, Cassandra like ZooKeeper is just a building block. It doesn't actually provide service discovery, health checking, etc. You could use it
40.
by armon 12y ago
SkyDNS is compared to Consul here: http://www.consul.io/intro/vs/skydns.html
41.
by armon 12y ago
That can probably be worded better. Serf operates at a node level abstraction, while Consul operates at a service level. In fact, Serf is used in Consul to power the discovery of other nodes.
42.
by armon 12y ago
You are right, the Consul site is very heavily based on the Serf site. It's actually almost entirely just a re-skin of the same site. The design is a bit cleaner however, which does make it nicer to read.
43.
by armon 12y ago
The value is base64 encoded, since JSON doesn't play nicely with binary values
44.
by armon 12y ago
We didn't manage to finish it in time, but it should be released in the next two weeks or so!
45.
by armon 13y ago
By this measure Memcached offers a greater feature set, an infinitely more robust parser, request concurrency, and doesn't really on a garbage collector. The very fact that this "db" relies on HTTP requests already makes me c
46.
by armon 13y ago
Serf doesn't actually use Docker in anyway, just as clarification. It does work really well within it, but there is no dependency either way.
47.
by armon 13y ago
There were many potentially other solutions to this issue. The clients could have buffered writes locally, and done batch updates to MongoDB instead of incrementing each time. Using a local SQLite, I can do "update counts set val=val+1
48.
by armon 13y ago
I've also worked in university research group settings, as well as a few startups in SV. I'd say there is a much stronger sense of the "whole" in these smaller groups, where everybody feels personally responsible for the
49.
by armon 13y ago
I've worked at Amazon, both in the warehouse management division as well as AWS, and the whole "bar raiser" phenomenon is mostly just marketing fluff. There are a handful of engineers who don't mind interviewing (or at l
50.
by armon 13y ago
We built a similar decorator/context manager for profiling, but it is really much more useful if you can export the data to statsite + graphite so that you can graph and view it on an on going basis. The insight into the runtime is muc
51.
by armon 13y ago
Riak Core provides a superset of the features of Serf. Riak Core uses gossip to manage membership, but it also provides quorums for coordination, and is based around the notion of a hash ring and virtual nodes. You could instead using Serf
52.
by armon 13y ago
I think we are maybe just working with different definitions. High Availability for Serf means that it can continue to handle changes in topology and deliver user events in the face of node failures and network problems. However, it is inev
53.
by armon 13y ago
You could build something like this on Serf, but it would take a little creativity. You might do something like this: 1) Send a "pre-deploy" event. Handler scripts use random number generator to decide which group they are in (&qu
54.
by armon 13y ago
In writing that section, we tried to provide "just the facts". If there is anything that seems wrong or misleading in any way, we'd like to know so that the page can be corrected. It is not our intention to say "Serf is
55.
by armon 13y ago
I'd highly recommend taking a look at this page: http://www.serfdom.io/docs/internals/gossip.html . One of the great attributes of the gossip protocol is it is very robust to intermittent network failures. Und
56.
by armon 13y ago
Dynamic roles is a fairly hard problem, which we hope to solve by building a different tool on top of Serf in the near future. If you want multiple roles that are static, it can be simulated by just providing a comma separated value as the
57.
by armon 13y ago
Yes, the availability of the system is not tied to any given node(s). Any node (or group of nodes) can continue to operate in the face of failure.
58.
by armon 13y ago
The Serf executable is actually just a wrapper around the `serf` library. That library is designed to be embedded in Go applications. Documentation for the library is available here: http://godoc.org/github.com/hashicor
59.
by armon 13y ago
We designed the `serf` library to be able to be easily embedded, so hopefully it can be of some use. Unfortunately Serf does not make use of any sort of NAT traversal currently. We've open sourced the project hoping to get the communit
60.
by armon 13y ago
We have a comparison against Chef and Puppet, which may be relevant to Salt here: http://www.serfdom.io/intro/vs-chef-puppet.html . Not sure about Salt's search, but Serf is designed to run much more often than con

More ›