10 ms·
I think these problems occurred because an ESB is too easy to over-use. Once you have it, it's very tempting to use it for everything. But of course the more y
by Galanwe 6y ago
I think these problems occurred because an ESB is too easy to over-use.
Once you have it, it's very tempting to use it for everything. But of course the more you use it, the more features it needs to have.
Let me illustrate that with a example compiling some of the patterns I have come to see at companies that overly used their ESBs:
Say Bob is working at a trading company and sets up an ESB where he publishes the daily trades of the company.
Soon, the trading desk learns about that, and decide that it's actually cool to use this for trading, they just have to listen on the ESB, and send the trades accordingly. That's the error pattern 1: making the ESB business critical.
The next day, the risk team learns about that ESB thing, and decide its very handy to perform post-trading checks by just listening to the trades flowing on the ESB. So they setup a system that listens for trades on the ESB, check that the trade is compliant with some limits, and send an other message on the ESB to let everyone know that this trade is validated. This is error pattern 2: cascading messages triggered from other messages.
The week after that, the security team learns about the ESB, and decides its very insecure to let anyone see the trades of the company, so they start implementing access control on the ESB. This is error pattern 3: now you have an overly complex layer on top of the ESB to decide who can see what and who can publish what.
Rince and repeat patterns 1, 2,and 3 for 5 years and here is the situation you end up with:
- The ESB is not the easy and handy system it was in the beginning. Since it has become the de-facto standard for publishing information in the whole company, it has to support features for _all_ the company use cases. There is access control to publish per topic, access control to listen per topic, multiple bindings of varying quality for each technological stack/language that each team in the company is using. The company of course is not capable / prepared to maintain a software of this scope, so the ESB is crippled with bugs that nobody can fix, because, you know, the infrastructure team cannot fix their groovy scripts using the ESB cause the guy that wrote them left. And the marketing team has some interns using the excel plug-in but they don't have time to rewrite them this year. The ESB is now partly un maintained, because the company relied totally on it without having the capacity / willingness / foresight of understanding how intricate it can be to update something that everyone use.
- The ESB is now very slow, because it was so tempting to publish anything of various interest on it that everyone did it. The problem is that the ESB is also critical for the company, so the whole flow of message is now slowly moving and overflowing, requiring endless tuning and tentatives at scaling it better. Of course 80% of the messages on the ESB are actually not listened to by anyone, but since nobody really knows who listens to the published messages, it's very tempting to just _not_ stop programs from publishing, ever, because god knows if some random team at the other end of the company might have a program reading these messages.
- You most likely have now an IT team dedicated to maintain the ESB. They are squeezed and pressured by the business teams to keep the ESB fast and easy to stable without requiring them to recode all the crap they plugged on it. On the other end, the other IT teams are pressuring them to update the ESB to support <place your language/stack here>. Of course the ESB team has no incentive to make any improvement whatsoever to the ESB, because that would definitely crash most of the crap the less technical teams of the company plugged to it. But the ESB team is the de facto guardian of the temple of the ESB, so everyone ends up frustrated by the situation.
---
I'm not sure I did a good job at explaining the various problems here, but basically, the one size fits all that ESBs are promoting is often not a future proof choice.
The reality is that you don't want your whole company coupled to a single system like that. Otherwise your system will be as good as the worst user of it.
- mumblemumble 6y agoFWIW, I worked at a trading firm that had a very tidy and long-lived ESB that I found to be a joy to work with. -but- My experience there 100% confirms what you say about it not being something that scales well. It was a relatively modestly sized firm by headcount, as trading firms go, and there was a corporate culture of absolutely intense inter-team and inter-departmental communication. One absolutely would not dream of subscribing to an event stream without first talking to the team that maintained the program that published it. Any changes to the event stream - both the grotty little details about what was being published and how, and the grotty little details about what was being consumed and how - would be preceded by a discussion among all the people who were working with it, to make sure that everyone involved continued to have a complete picture of the interactions involving it. That level of communication, which I do believe was essential to the bus's long term success, just wouldn't scale to a large company. Nor could it have been maintained at a company that had a more relaxed attitude in general. Nor could it have been maintained at a company where programmers are allowed to believe that most of their time at the office can be spent with hands on a keyboard.
- the_af 6y ago> That level of communication, which I do believe was essential to the bus's long term success, just wouldn't scale to a large company What would though? An ESB could be a way to enforce a standard and slow down gung-ho devs & teams using ad hoc solutions for every problem. But say an ESB is not the solution -- what is? As the company grows, overhead and friction become more important. I'm not sure "every piece connecting to every other piece in whatever way" would help with scale, rather than compounding the problem...
- mumblemumble 6y agoIMO, there is no software solution, because it's not a software problem, it's a social problem. As far as specific things to try go, domain-driven design is my personal favorite off-the-shelf mental framework for dealing with these sorts of things. Especially the concept of bounded contexts. Embrace Conway's Law; recognize it's not a criticism, per se, it's also a scaling strategy.