4 ms·
I've seen the guts of a few major financial organizations, and there are some common themes regarding their infrastructures. The one that really sticks out to
by bob1029 6y ago
I've seen the guts of a few major financial organizations, and there are some common themes regarding their infrastructures.
The one that really sticks out to me as an engineer is the fact that the whole system in most cases seems to be tied together by a fragile arrangement of 100+ different vendors' systems & middleware that were each tailored to fit specific audit items that cropped up over the years.
Individually, all of these components have highly-available assurances up and down their contracts, but combine all these durable components together haphazardly and you get emergent properties that no one person or vendor can account for comprehensively.
When the article says a full reset entails killing the power and restarting, this is my actual experience. These complex leviathans have to be brought up in a special snowflake sequence or your infra state machine gets fucked up and you have to start all over. When dependency chains are 10+ systems long and part of a complex web of other dependency chains it starts to get hopeless pretty quickly.
- outworlder 6y ago> that were each tailored to fit specific audit items that cropped up over the years. Or worse, "compliance" line items, that some tool or some company identified in their cookie-cutter processes. As long as that line item goes away, noone really cares what the long term implications are.
- Aeolun 6y agoThat’s closer to the truth in my experience.
- x87678r 6y agoExternal Audit are the most important people in financial co's Technology. Everything else is a nice to have.
- lmilcin 6y agoExactly. If you are manager, a disaster is something that is only potential. Audit and its consequences are unavoidable.
- dan_quixote 6y agoYup. This is why you're seeing rapid adoption of event-based architectures. You can more confidently surmise the larger system state in a well-designed system.
- meesles 6y agoThere's so many trade-offs within this statement I felt like it deserved some color - * Spinning up an event-based architecture is prone to the same issues GP describes - For example what if you spin up a 'pub' side without the corresponding 'sub' side? * Event-based architecture does not inherently give you better observability of the whole system. I would argue it's actually worse to begin with because of the decoupled nature of the services. You have to build tooling to monitor traces across services or cross-system health * Any 'well-designed' system will solve these problems. A monolith with good tooling and monitoring will be more easily understood than a poorly designed event-based system. * I think metrics and monitoring are really the key to knowing about a system, regardless of the architecture. Basically, I think you can remove your first sentence and leave the last to make a truer statement. A great system should have design considerations and tools to make it understood at various scopes by various people of various skills. I think people have falsely conflated event-based architecture with systems health due to having to rewrite a system almost from scratch, which results in better tooling since it's something you're thinking about actively.
- kinow 6y ago>Basically, I think you can remove your first sentence and leave the last to make a truer statement. +1 not sure how event driven arch would have mitigated here. A well designed (with redundancy where necessary, better tooling, monitoring, scalable, etc) would.
- junon 6y agoThis. Event-based systems do not satisfy the "passive" part of active/passive dist-sys design - ergo, they are not fault tolerant by themselves, and most machinery designed around making them quasi-fault-tolerant tend to be bigger headaches in the long run (see Apache Kafka). I've had to explain this to a dozen teams so far in my career and most of them go ahead with the design anyway, regretting it not even 6 months later.
- systemvoltage 6y agoHow come they don't have integration testing? Full 100 services should have end-to-end integration testing and any change made to that chain of tooling should have to run through a massive integration test. If anything fails, the change is no longer acceptable.
- TheDong 6y agoYou can comment this on any outage that ever happens. "Why didn't they have a test for that?" The answer is that tests are never perfect. If you want to create an integration environment that mimics prod, you have to fork an entire parallel universe into your integ environment to run the test. Anything else will diverge from the reality of the future. Even if every vendor's service or hardware had integration tests, that doesn't mean that the integration tests covered every case. It doesn't mean there's not an emergent property of two systems behaving in a slightly unexpected way that turns into a catastrophic result. It's not necessarily even possible to have two copies of some of the systems; who knows how expensive a given vendor's hardware box is. It's definitely not possible to exactly mimic future traffic. Perhaps in the test environment it works, then the prod environment, requests are different so it fails. Hardware errors happen, and integ testing those is difficult to say the least.
- astrobe_ 6y agoLong story short, tests don't prove there's no bug, they can only show there is a bug.
- Aeolun 6y ago> It's not necessarily even possible to have two copies of some of the systems; who knows how expensive a given vendor's hardware box is. Given you are running a stock exchange, chances are you have enough money for at least one copy.
- toast0 6y agoCertainly there's money for one copy, that's what runs production.
- AmericanChopper 6y agoI’ve worked in a lot of banks, and other similar organisations, and the truth is that big enterprise just sucks at compliance. You can approach compliance frameworks in two ways, you can put a lot of effort into designing your infrastructure and compliance testing methodology (I would suggest Amazon as a canonical example of how to do this well), so that it performs well and meets all the requirements, or you can take out the sledgehammer, implement a new control for every occasion, and create a cumbersome bureaucracy. The good design approach is obviously superior in many ways. But the downside of it is that you have to trust the competency of a lot of different business units to maintain it. A cumbersome bureaucracy on the other hand ensures that incompetent/lazy/low-initiative internal actors can’t impact your compliance. If you fail, at least you fail in a compliant and auditor-approved way. That said, a lot of the failures I’ve seen in organisation like this stem from silo’d expertise. People don’t know that much about the systems that are outside their remit, so they will make changes that impact connected systems in ways they failed to imagine. As an example I have seen 3 seperate banks have non-trivial service disruptions stem from the same independently made mistake. A person enabling debug logging on the SIP phones. The traffic DOSes their networks, and all of a sudden, core network infrastructure starts to die. Afterwards they send the right reports off to the right people, make the correct adjustments to the bureaucracy, and proceed with their compliance intact. There is also a 3rd way to approach compliance, via negligence. But the more you are in the regulatory spotlight, the less of an option that is.