3 ms·

I'd say this is a good introduction into reasons why self hosting clustered software is not something to do lightly. He's basically running rabbit mq on Windows

I'd say this is a good introduction into reasons why self hosting clustered software is not something to do lightly. He's basically running rabbit mq on Windows. Yikes. And also, why? I'm sure it can be done responsibly. But allowing all nodes to self update and reboot randomly sounds like amateur hour to me (that actually happened apparently). So, don't do that. I don't use rabbit mq currently but we do use Elasticsearch, which has similar clustering capability and used to be more susceptible to split brain situations (been there, dealt with that) These days, I recommend using Elastic Cloud and avoid self hosting it. It's only cheaper until the first time you have to deal with a split brain cluster because you botched an update, mis-configured it, etc. One of the nice features in Elastic cloud is that you can click an update button and it will orchestrate a rolling restart. If you don't know what that is, you should not be operating a cluster of any kind. I'm sure there are similar cloud based services for rabbitmq. Probably well worth the money unless your in house ops team is super experienced with operating it (which clearly wasn't the case here). Such a team would cost you many hundreds of thousands of dollars per year. One person does not cut it. You need at least a 3 or 4 so you can afford them taking vacations, sick leave, leaving, or dying in some tragic accident. Half a million pays for some pretty nice cloud based clustering capacity. A good team will cost you more. For reference, we pay about 70/month for a tiny Elastic cloud search cluster that is actually good enough. I can double the price and capacity with a simple slider and it would still be cheap. One hour of my time is more than that with my normal freelance rate. My monthly rate would pay for an enormous cluster that far exceeds anything we need and it would still be cheaper than making sure we have four people with my skill set in the team (we don't) able and willing to look after it at all hours. The largest cluster I've ever dealt with was a self hosted monster that could index billions of documents per hour (millions per second). You so much as looked wrong at it, all hell would break loose. That cluster was one of several managed by a very experienced ops team that probably cost millions per years. That's the price of doing business at scale when self hosting. Most companies running into trouble with ES cut corners on doing it right and then pay the price with preventable outages, scaling issues, technical debt, etc. Cloud based services don't completely prevent this but if you know what you are doing, they provide a nice level of safety and risk mitigation.

vladvasiliu 4y ago
I don't have much experience with RabbitMQ, but I don't fully agree with your point about ES. We run a cluster for ingesting logs, and it's much cheaper. The clustering part is basically painless. I had to script a bunch of things to easily change instance sizes (adjust the memory according to available RAM), and that's it. Sure, if you don't want to have to deal with this, going the managed route is OK. The most issues we've had, however, is with the actual documents sent for indexing, querying them, etc. So basically something with which ES Cloud, or other managed solution, wouldn't have helped. I'd expect RMQ to be roughly the same. I've seen a bunch of people do weird things shoveling messages from left to right and not consuming them before the RAM exploded. All this on managed RMQ.
- mst 4y ago
  I think "use a managed service until that starts to suck" is often not a bad heuristic - with a bit of luck by the time (if ever) it does start to suck you'll be familiar enough with the system that spinning up your own cluster will involve significantly less "learning the hard way" than it would've done had you done it first off. And I say this as somebody who really really dislikes managed services from a "but I want to have total control" point of view (this is not entirely rational on my part but true of me nonetheless.)
codedokode 4y ago
If you have a "tiny" cluster, then do you really need a cluster? Wouldn't a single node be enough for you?
- jillesvangurp 4y ago
  I like having zero downtime upgrades and failover.
xorcist 4y ago
While I've been in similar situations and mostly agree with your reasoning, one thing bothers me. Your calculations of engineering hours, which is how most cloudy stuff gets motivated, hinges on two points. One is that operating the software and using it are so disparate that those two or three trained experts could not also be your developers. But in a small organization, that's likely to be the case. Or put another way, if your developers refuses to gain experitse in the technology, what business do you have relying on it? The other point is the implicated idea that, somewhat rightfully, operating the software without expert knowledge is "amateur hour" but apparently developing for the same software without knowing what you are doing is somehow ok? To me, that doesn't make sense. The same logic applies there, where you need a handful of people to counter sick leave, accidents and other unplanned human outages. This is especially true with Elastic. Sure, running a production cluster on some developer laptop which suddenly reboots for updates isn't great, but neither are broken schemas, uncontrolled bucket growth, broken stemming, or any other of a million things that can go haywire logically without operations being an issue. There's simply no getting away from requiring expertise. If it's your part of your day job, you need to really know the stuff. Yet not knowing what you're doing is so taken for granted that "throw it in the cloud so at least the operations people know what they're doing" is completely accepted logic. That doesn't mean cloud services are bad per se, but the expertise-not-needed argument is. This in turn changes the economics considerably. Sorry for going on a tangent. I know I'm the odd one here but I can't bring myself to accept it.

vladvasiliu 4y ago

I don't have much experience with RabbitMQ, but I don't fully agree with your point about ES. We run a cluster for ingesting logs, and it's much cheaper. The clustering part is basically painless. I had to script a bunch of things to easily change instance sizes (adjust the memory according to available RAM), and that's it. Sure, if you don't want to have to deal with this, going the managed route is OK. The most issues we've had, however, is with the actual documents sent for indexing, querying them, etc. So basically something with which ES Cloud, or other managed solution, wouldn't have helped. I'd expect RMQ to be roughly the same. I've seen a bunch of people do weird things shoveling messages from left to right and not consuming them before the RAM exploded. All this on managed RMQ.

mst 4y ago
I think "use a managed service until that starts to suck" is often not a bad heuristic - with a bit of luck by the time (if ever) it does start to suck you'll be familiar enough with the system that spinning up your own cluster will involve significantly less "learning the hard way" than it would've done had you done it first off. And I say this as somebody who really really dislikes managed services from a "but I want to have total control" point of view (this is not entirely rational on my part but true of me nonetheless.)

codedokode 4y ago

If you have a "tiny" cluster, then do you really need a cluster? Wouldn't a single node be enough for you?

jillesvangurp 4y ago
I like having zero downtime upgrades and failover.

xorcist 4y ago

While I've been in similar situations and mostly agree with your reasoning, one thing bothers me. Your calculations of engineering hours, which is how most cloudy stuff gets motivated, hinges on two points. One is that operating the software and using it are so disparate that those two or three trained experts could not also be your developers. But in a small organization, that's likely to be the case. Or put another way, if your developers refuses to gain experitse in the technology, what business do you have relying on it? The other point is the implicated idea that, somewhat rightfully, operating the software without expert knowledge is "amateur hour" but apparently developing for the same software without knowing what you are doing is somehow ok? To me, that doesn't make sense. The same logic applies there, where you need a handful of people to counter sick leave, accidents and other unplanned human outages. This is especially true with Elastic. Sure, running a production cluster on some developer laptop which suddenly reboots for updates isn't great, but neither are broken schemas, uncontrolled bucket growth, broken stemming, or any other of a million things that can go haywire logically without operations being an issue. There's simply no getting away from requiring expertise. If it's your part of your day job, you need to really know the stuff. Yet not knowing what you're doing is so taken for granted that "throw it in the cloud so at least the operations people know what they're doing" is completely accepted logic. That doesn't mean cloud services are bad per se, but the expertise-not-needed argument is. This in turn changes the economics considerably. Sorry for going on a tangent. I know I'm the odd one here but I can't bring myself to accept it.