jolynch

8 ms·

31.
by jolynch 10y ago
> in any serious enterprise that would not be tolerated I think that's a bit of a true scotsman fallacy. We use a lot of software we didn't write, and a lot of it has bugs. The languages that we write code in have bugs (e.g. Py
32.
by jolynch 10y ago
I respectfully disagree. I'm all for root cause analysis and taking the time to fix things upstream, but I also think that it's easy to say that and hard to actually do it. Yelp doesn't make more money and our infra isn'
33.
by jolynch 10y ago
(I help maintain SmartStack) I think it's really interesting that "what we've already got setup" is such a big driver in which systems we pick. For example, in 2013 Yelp already had hardened Zookeeper setups and Consul d
34.
by jolynch 10y ago
Nice, now that v3 includes some of these crucial features I might even be able to merge the etcd provider for Synapse! https://github.com/airbnb/synapse/pull/58
35.
by jolynch 10y ago
People don't use books to kill, maim and disfigure other people, they use guns. There are zero logical reasons to prohibit a database that maps gun metadata to the gun record (search by name is actually detrimental to proper investigat
36.
by jolynch 10y ago
It mostly came down to flexibility. The original form of our monitoring did do basically what you suggest, "one node failure is ok, two is not", but we found that was not good enough. That approach in our experience was: 1. Noisy.
37.
by jolynch 10y ago
https://aphyr.com/posts/329-jepsen-rethinkdb-2-1-5
38.
by jolynch 10y ago
To be honest most of my exposure to RethinkDB is from Aphyr's article on it. We already run basically every alternative he mentions as superior for particular use cases. For example we run Zookeeper / replicated SQL for inter-key
39.
by jolynch 10y ago
We've had a few internal discussions about ScyllaDB. I think that the performance numbers look attractive, but we are concerned about maintainability. In particular we've invested fairly heavily in configuration management, toolin
40.
by jolynch 10y ago
A deep dive into how data is stored on Cassandra rings and how to use this knowledge to give your operators notice before cluster issues become site issues. Any feedback is super welcome. If folks like this we may consider working on upstre
41.
Monitoring Cassandra at Scale(engineeringblog.yelp.com)
97 points by jolynch 10y ago|11 comments
42.
by jolynch 10y ago
You can down servers or backends over the stats socket, but you can't add them (adding servers requires a reload). The DNS resolution work that is going into 1.6 releases may lead to a workable approach where you over allocate servers
43.
by jolynch 11y ago
Sysdig makes me giddy like dtrace used to. I also sometimes feel a bit ... challenged ... by translating the questions I have (e.g. why did this arbitrary program start using a lot of memory and then OOM) into actual sysdig chisel invocatio
44.
by jolynch 11y ago
I think the latest number is looking more like 6M Mpps SYN ( http://marc.info/?l=linux-netdev&m=144431864418007&w=2 )! This is also awesome because it fixes the issue with SO_REUSEPORT where the 3WHS gets screwed up b
45.
by jolynch 11y ago
I mean, I wouldn't encourage anyone to use HAProxy for microservice load balancing unless they're running it on every node, at which point clients are always talking to localhost and everything is egress. That being said, you can
46.
by jolynch 11y ago
Has anyone actually done this at scale and seen how it works? We've talked about it a few times because Consul looks really cool but it doesn't appear to have the full capabilities of SmartStack yet, in particular Synapse ( https:
47.
by jolynch 11y ago
The most significant issue with HAProxy and large SOAs is the lack of really good dynamic re-configuration. You don't want to have to manually change your configuration every time someone wants to add or change load balancing to a serv
48.
by jolynch 11y ago
Check out the deep dive I wrote into how you can help prevent HAProxy reloads from dropping traffic without a large 99% hit over at http://engineeringblog.yelp.com/2015/04/true-zero-downtime-h... Looks like bamboo
49.
by jolynch 11y ago
I'm excited for all the buzz going on around service discovery, it's often one of the most crucial design decisions for a SOA architecture. I'm really bummed, however, that so many of them rely on clients doing DNS lookups. I
50.
by jolynch 11y ago
To be honest my favorite part of SmartStack is that you are not tied to a single discovery backend or mechanism. Both Synapse and Nerve support custom backends using whatever system you want (zookeeper, etcd, DNS, etc). At the end of the da
51.
by jolynch 11y ago
I wish they did this and had a better search interface. Their current "all in one bar + suggest" functionality works somewhat poorly for me. Also I feel like they could reduce the needs for super great API docs if they had somethi
52.
by jolynch 11y ago
Elasticsearch is built on top of Apache Lucene and is very similar to technologies like Solr but with a nicer interface (really more like SolrCloud these days). It is different in that it provides higher level APIs and distributed functiona
53.
by jolynch 11y ago
No, nginx should not have this problem as it uses fd passing to gracefully handoff connections. HAProxy only has this issue on Linux because Linux's SO_REUSEPORT implementation unfortunately introduces a race condition between accept a
54.
by jolynch 11y ago
I've been a little confused by the claims of how complex the SYN delay solution is. Is it actually all that more complex? The original blog post wraps a restart command in two iptables invocations (and relies on a hacky sleep interval
55.
by jolynch 11y ago
As other posters have mentioned the solution in this link is actually suboptimal. I showed how the iptables solution leads to 1-3s latency for connections that establish during the restart but you can avoid it by getting a little more hardc
56.
by jolynch 12y ago
Patching HAProxy to reload config is really hard. There have been ideas, patches, and discussions on the HAProxy mailing list for a few years now trying to get zero downtime reloads natively supported in HAProxy, but the reality is that it
57.
by jolynch 12y ago
Heh, the number of times DNS has bitten me in production... My main objections to DNS are: 1) You have no control over clients, there is absolutely no rule that says clients have to respect TTLs (I'm looking at you Java) 2) We are talk
58.
by jolynch 12y ago
Totally agree, I mention this as the largest drawback of this approach in the post.
59.
by jolynch 12y ago
Josh gets all the credit for finding the plug qdisc. I was originally planning on using netem and having a fixed delay of 50ms or something, but Josh decided that wasn't elegant enough and went looking for a better solution. As for the
60.
by jolynch 12y ago
I am not familiar with IPVS, I'll look into it. From your description though I imagine it might have the same drawbacks that exist with the multiple HAProxy + iptables swap solution. With a solution where we run multiple HAProxies and

More ›