3 ms·
The timing of this article couldn't have been more painful. I'm currently working on solving a problem with an insanely over complicated setup (for the task at
by LeonM 7y ago
The timing of this article couldn't have been more painful.
I'm currently working on solving a problem with an insanely over complicated setup (for the task at hand) that is build by another engineer who has since left the company.
It's a cluster of 3 virtualised machines running docker swarm where a RabbitMQ instance ties 40+ worker pods together. Once every 5 days or so the connection between RabbitMQ and (some of) the worker container stops working, causing the worker to crash and the queue message is lost.
We are talking like 5 layers of virtualization and/or abstraction. It's impossible to debug. I honestly don't know how to explain this to my customer.
- deleted 7y ago[deleted]
- rmetzler 7y agoThe bug description sounds a little bit like a memory leak or connections aren’t closed properly.
- noncoml 7y agoThis. You are leaking connections probably.
- iddqd 7y agoYou might be better off focusing on improving fault tolerance instead of trying to hunt down the root cause.
- eternalban 7y agoRabbit supports the choice of durable exchanges, queues, and messages. There are (tunable, negotiable) performance penalties in terms of latency and throughput. We are partly in agreement regarding f/t. Logging comes into play when digging into recurring failure (bugs). Fault tolerance will not save you from bugs and bug hunting. [p.s. to add, performance tuning/trouble-shooting also require instrumentation. Instrumentation would appear to be a fundamental (and useful) capability.]
- itronitron 7y agoit sounds like this setup has proven very successful in helping your predecessor get hired (away from the company), I wonder if that was an implicit or an explicit requirement when they were designing the system