3 ms·
This sounds like the kind of thing that would be very powerful if built in from the start, but an absolute nightmare to try and retrofit to an existing system.
by taneq 9y ago
This sounds like the kind of thing that would be very powerful if built in from the start, but an absolute nightmare to try and retrofit to an existing system.
It reminds me of the Amazon "everything must be a network service" architecture. Painful to institute but with great payoffs later in terms of robustness and scalability.
- ealexhudson 9y agoI know what you mean here, but I have to disagree. I see constant over-engineering in teams, and one of the main reasons for this is copying patterns like this way too early in the life of the business. I see this as a version of Monte Carlo testing - I'm not totally sold on doing it in production, but certainly if you have a load that's difficult to replicate in scale, then this is the type of thing you should be doing. But I don't think anyone should be doing this from the start. This is something that you evolve into. Sure, it's easier to bring it in if you planned it from the start - but as an engineer, you cannot proceed on that basis. You know and understand stuff has to change as you grow, and the art of it is making (more often than not) good calls on architecture to allow that.
- empath75 9y agoWe started our current project running chaos monkey from almost day one, and a few weeks ago, someone fucked up a tag enforcement script on our dev aws account and started shutting down every single instance, while all of our asgs kept restarting them. So basically every single service from consul to kafka to our microservices was down to a few or sometimes even no instances. Because we had already been dealing with chaos monkey, everything recovered by itself within a few minutes of the script stopping, and we barely had any alerts from any services going down. Also, when aws was pushing our meltdown patches, we just ignored all the notifications because we didn’t care about instances being shut down, because they go down randomly all the time. It’s a pain in the ass at first to deal with, but it definitely forces you to fix a lot of potential problems unless you want to spend all your time manually fixing stuff every day. Even just aside from chaos monkey, we build new base amis and push out rolling updates daily. That’ll also quickly suss out problems with not having version numbers locked, etc.
- drdrey 9y agoWhat you can do from the start that will help you down the line is to use a common RPC framework/library across the board, especially if it supports failure injection. It's a shame when you can't easily do chaos testing on a system because it chose to use a different RPC system.