8 ms·

Not that I don't appreciate the subject matter, I wave the flag for K8S all the time and I've shipped bank product presentation stuff on it... But... The funda

Not that I don't appreciate the subject matter, I wave the flag for K8S all the time and I've shipped bank product presentation stuff on it... But... The fundamental challenges of building a bank are almost entirely orthogonal to things like distributed system uptime and resiliency (unless, I suppose, you could lose consistency during the types service loss Kubernetes makes easy to ameliorate). Evidence for this abounds: nearly every major bank out there is at least 10 years behind the tech we're discussing here. Sure, banks are getting savvy to modern techniques for their presentation and API layers (my employer, Capital One, for example). But the actual challenges are data consistency and liveliness and audit-ability are preserved. I'm really curious if you're using a novel technique to achieve this that Docker and self-managed micro-app swarms can deliver on better. Because what really limits most financial institutions from embracing a lot more modern tech is their core systems of record AND the acceptance of said systems by their governing agencies. So when we talk about basic datacenter ops, that's all great. But I don't think they're the things that make K8S great for a bank. I think most people in a position to evaluate a financial institution roadmap would be unmoved by this deck. Now, if you talk about other things we both know K8S is great for... Things like discovering the genesis of a database action by preserving it throughout the chain of responders, or being able to rapidly respond to site exploits with rolling restarts and a built in mechanism for feature flagging, or having a really great way to offer data scientists the environments they need without risk for data theft, or being able to use traditional CI/CD methodologies but end up with a single deployable unit that is amenable to both automated an manual review and mechanical deployment in spite of the tooling used within. Not that I think selling K8S is your job. But I thought I'd mention the perspective of someone doing inf modernization and product work at a major bank. And of course, as always: the opinions above are my own and not those of my employer or co-workers.

nstj 10y ago
> what really limits most financial institutions from embracing a lot more modern tech is their core systems of record AND the acceptance of said systems by their governing agencies I tend to think (from experience) that the reason banks don't embrace modern tech is inertia - they've been able to "milk the cow" of regulatory body sanctioned profits for so long that they're in a mindset of not wishing to introduce volatility into what has been a highly predictable revenue stream. They've been using garbage technology for so long and making money at the same time that there has been no institutional impetus to innovate and embrace new tech. This is now coming to bite them in the ass as regulatory bodies are allowing new players to market and therefore eroding some of the guaranteed profits banks enjoyed up until now. I doubt most large capital markets institutions (on both the retail and IB sides) will be able to weather the current (and impending) storm of disruption and come out unscathed.
- raesene6 10y ago
  I'd say inertia is a large factor but also risk aversion. Bank systems have to be very available (or they attract large fines) and so there's a tendency to stick with things that are proven to be robust even when they have other problems. The other major problem I've seen is that most banks don't think of themselves as technology companies, and treat IT as an overhead to be minimized, which is absolutely the wrong approach. This tends to lead to things that save money in the short term (e.g. outsourcing deals) but could well cost money in the longer term, as they make replacing legacy systems harder.
  - riskable 10y ago
    This is nonsense. Not every system in banks needs to be highly available. That's just silly. Like the print server on the 3rd floor needs an active standby! Haha. No, just like any organization banks have "critical" systems and "everything else." Docker is mostly being "sold" as a means to replace and improve the non-critical stuff. Like that internal web app everyone uses to look up <whatever>. Or the system that generates daily reports on <whatever>. Just like most organizations, banks have a few critical systems and everything else is less so (to varying degrees).
    - raesene6 10y ago
      yes and the topic of discussion in this thread is ..... core banking systems.... which do have to be highly available. If the topic had been bank print servers and I had made my comment yours may have made more sense My comment didn't say "banks print servers need to be highly available" anywhere.. at all..
- user5994461 10y ago
  > I tend to think (from experience) that the reason banks don't embrace modern tech is inertia A lot of inertia. A lot of new-hype-tech is unreliable undocumented shit not ready for production. A lot of new-tech-is-old-tech that's been done for many years but with a new name. Whenever someone asks "why don't you use Docker for xxx?" Reply with "When is the last time you had an issue with Docker? Tell me about it." You'll LOVE the horror stories <3
obeattie 10y ago
>what really limits most financial institutions from embracing a lot more modern tech is their core systems of record AND the acceptance of said systems by their governing agencies. I just want to say that I think there is a huge amount of FUD about how you can and cannot build your technology as a regulated entity – and in particular as a bank. In reality, close to 100% of requirements from a regulator will tell you _what_ you must build, not _how_ you must build it. Even then, especially in terms of resilience and security, they are almost always a subset of our own requirements. What can be more of a challenge is convincing an auditor that what you have done is acceptable, since it can be so different from what they may have seen before. Again, I don't think this is a reason to compromise. We see technology as a major competitive advantage, so it is worth the effort to find open-minded auditors, and spend time to explain and demonstrate how (and why) our software meets the requirements. I don't think there's any way we could build a secure, resilient bank with the kind of product experience we want, AND do it on the budget of a startup if we approach technology through the same lens as existing banks.
- tluyben2 10y ago
  > I just want to say that I think there is a huge amount of FUD about how you can and cannot build your technology as a regulated entity – and in particular as a bank. In reality, close to 100% of requirements from a regulator will tell you _what_ you must build, not _how_ you must build it. Even then, especially in terms of resilience and security, they are almost always a subset of our own requirements. Having worked with banks and insurers solely in Java and connecting to legacy (mostly Cobol) before, I was surprised, in my current position, to see some companies doing their complete banking back end in PHP & MySQL. I knew the how is not part of the regulations, but I did expect the CTOs to pick the 'no one ever got fired for choosing' choice.
  - webscaleizfun 10y ago
    It seems common knowledge these days among the slightly but not too technically inclined that any new major project should use a LAMP stack as its base. People see Facebook, Amazon, and many others running PHP & MySQL on Linux at scale and they know it works reliably, so while it may not have the support of Cisco or Oracle, it is pretty close on the 'no one ever got fired for choosing' X scale, since you can point to every other major company using these building blocks reliably if your investors, CEO, board or auditors asked why you chose to use PHP & MySQL. In summary, PHP & MySQL have become the modern equivalent of a "safe" choice for your stack to be built on. Its not necessarily a bad choice either, you get access to a large community of skilled people who can write PHP & write SQL statements, and while everyone likes to hate on PHP, it isn't about to up and disappear any time in the next decade either (unlike COBOL).
kchoudhu 10y ago
>what really limits most financial institutions from embracing a lot more modern tech is their core systems of record AND the acceptance of said systems by their governing agencies. Let me put this even more bluntly: if your bank's technology plan isn't 99.5% about dealing with government regulation and maintaining core record integrity and auditability, you are not a serious player in the space.
- riskable 10y ago
  Man, it's like people think banks are special when it comes to IT. They're not! You have lots of extra regulations for sure but most of them are about retention of financial records. If a system isn't processing/storing financial records or "privileged" information nobody gives a damn. The "technology plan" is 99.5% about making or saving money. That remaining .5%? Yeah, that's compliance. Because that's all it costs. Unless you think central logging systems are going to take up some large percentage of a multi-billion dollar quarterly budget? People love to complain about "the costs of regulation" but you know what? In finance it really doesn't amount much in terms of "how much we spend." How much "it holds back the market" is a different debate entirely. Aside: Without those regulations we'd just repeat all the same financial disasters throughout history.
  - kchoudhu 10y ago
    If it's a computer in a bank and it touches risk, trading or treasury, it's fair game for the Fed auditor. So you tell me: what computer system of any import in a bank doesn't touch one of these three things?
    - riskable 10y ago
      Actually all systems are fair game to the auditors. If an auditor wants to see something they get to see it 99% of the time. End of story. They really don't care about systems that don't process financial information! They don't care about your dev or qa environments. They don't care about your DNS servers or your switches or much else for that matter. Regulators are 100% laser-focused on financial information and transactions. They want to see ledgers and logs and they want to see evidence that your systems prevent tampering. That's it. There's no financial regulators that actually audit IT stuff. We probably should have them but we don't. The closest is the FFIEC but they only publish non-binding guidelines. If you think the PCI-DSS matters to banks you're mistaken. Every year we audit ourselves and put the results in a filing cabinet somewhere. We have no obligation to show it to anyone and no one would hold us accountable for failing to be PCI compliant anyway.
aidos 10y ago
As a Monzo customer I have to say that the product Monzo offers and the features that traditional banks want me to use are also somewhat orthogonal :-) So long as they can satisfy the regulatory requirements, I'd rather they were using whatever else it took to build a great product.
deleted 10y ago
[deleted]

nstj 10y ago

> what really limits most financial institutions from embracing a lot more modern tech is their core systems of record AND the acceptance of said systems by their governing agencies I tend to think (from experience) that the reason banks don't embrace modern tech is inertia - they've been able to "milk the cow" of regulatory body sanctioned profits for so long that they're in a mindset of not wishing to introduce volatility into what has been a highly predictable revenue stream. They've been using garbage technology for so long and making money at the same time that there has been no institutional impetus to innovate and embrace new tech. This is now coming to bite them in the ass as regulatory bodies are allowing new players to market and therefore eroding some of the guaranteed profits banks enjoyed up until now. I doubt most large capital markets institutions (on both the retail and IB sides) will be able to weather the current (and impending) storm of disruption and come out unscathed.

raesene6 10y ago
I'd say inertia is a large factor but also risk aversion. Bank systems have to be very available (or they attract large fines) and so there's a tendency to stick with things that are proven to be robust even when they have other problems. The other major problem I've seen is that most banks don't think of themselves as technology companies, and treat IT as an overhead to be minimized, which is absolutely the wrong approach. This tends to lead to things that save money in the short term (e.g. outsourcing deals) but could well cost money in the longer term, as they make replacing legacy systems harder.
- riskable 10y ago
  This is nonsense. Not every system in banks needs to be highly available. That's just silly. Like the print server on the 3rd floor needs an active standby! Haha. No, just like any organization banks have "critical" systems and "everything else." Docker is mostly being "sold" as a means to replace and improve the non-critical stuff. Like that internal web app everyone uses to look up <whatever>. Or the system that generates daily reports on <whatever>. Just like most organizations, banks have a few critical systems and everything else is less so (to varying degrees).
  - raesene6 10y ago
    yes and the topic of discussion in this thread is ..... core banking systems.... which do have to be highly available. If the topic had been bank print servers and I had made my comment yours may have made more sense My comment didn't say "banks print servers need to be highly available" anywhere.. at all..
user5994461 10y ago
> I tend to think (from experience) that the reason banks don't embrace modern tech is inertia A lot of inertia. A lot of new-hype-tech is unreliable undocumented shit not ready for production. A lot of new-tech-is-old-tech that's been done for many years but with a new name. Whenever someone asks "why don't you use Docker for xxx?" Reply with "When is the last time you had an issue with Docker? Tell me about it." You'll LOVE the horror stories <3

obeattie 10y ago

>what really limits most financial institutions from embracing a lot more modern tech is their core systems of record AND the acceptance of said systems by their governing agencies. I just want to say that I think there is a huge amount of FUD about how you can and cannot build your technology as a regulated entity – and in particular as a bank. In reality, close to 100% of requirements from a regulator will tell you _what_ you must build, not _how_ you must build it. Even then, especially in terms of resilience and security, they are almost always a subset of our own requirements. What can be more of a challenge is convincing an auditor that what you have done is acceptable, since it can be so different from what they may have seen before. Again, I don't think this is a reason to compromise. We see technology as a major competitive advantage, so it is worth the effort to find open-minded auditors, and spend time to explain and demonstrate how (and why) our software meets the requirements. I don't think there's any way we could build a secure, resilient bank with the kind of product experience we want, AND do it on the budget of a startup if we approach technology through the same lens as existing banks.

tluyben2 10y ago
> I just want to say that I think there is a huge amount of FUD about how you can and cannot build your technology as a regulated entity – and in particular as a bank. In reality, close to 100% of requirements from a regulator will tell you _what_ you must build, not _how_ you must build it. Even then, especially in terms of resilience and security, they are almost always a subset of our own requirements. Having worked with banks and insurers solely in Java and connecting to legacy (mostly Cobol) before, I was surprised, in my current position, to see some companies doing their complete banking back end in PHP & MySQL. I knew the how is not part of the regulations, but I did expect the CTOs to pick the 'no one ever got fired for choosing' choice.
- webscaleizfun 10y ago
  It seems common knowledge these days among the slightly but not too technically inclined that any new major project should use a LAMP stack as its base. People see Facebook, Amazon, and many others running PHP & MySQL on Linux at scale and they know it works reliably, so while it may not have the support of Cisco or Oracle, it is pretty close on the 'no one ever got fired for choosing' X scale, since you can point to every other major company using these building blocks reliably if your investors, CEO, board or auditors asked why you chose to use PHP & MySQL. In summary, PHP & MySQL have become the modern equivalent of a "safe" choice for your stack to be built on. Its not necessarily a bad choice either, you get access to a large community of skilled people who can write PHP & write SQL statements, and while everyone likes to hate on PHP, it isn't about to up and disappear any time in the next decade either (unlike COBOL).

kchoudhu 10y ago

>what really limits most financial institutions from embracing a lot more modern tech is their core systems of record AND the acceptance of said systems by their governing agencies. Let me put this even more bluntly: if your bank's technology plan isn't 99.5% about dealing with government regulation and maintaining core record integrity and auditability, you are not a serious player in the space.

riskable 10y ago
Man, it's like people think banks are special when it comes to IT. They're not! You have lots of extra regulations for sure but most of them are about retention of financial records. If a system isn't processing/storing financial records or "privileged" information nobody gives a damn. The "technology plan" is 99.5% about making or saving money. That remaining .5%? Yeah, that's compliance. Because that's all it costs. Unless you think central logging systems are going to take up some large percentage of a multi-billion dollar quarterly budget? People love to complain about "the costs of regulation" but you know what? In finance it really doesn't amount much in terms of "how much we spend." How much "it holds back the market" is a different debate entirely. Aside: Without those regulations we'd just repeat all the same financial disasters throughout history.
- kchoudhu 10y ago
  If it's a computer in a bank and it touches risk, trading or treasury, it's fair game for the Fed auditor. So you tell me: what computer system of any import in a bank doesn't touch one of these three things?
  - riskable 10y ago
    Actually all systems are fair game to the auditors. If an auditor wants to see something they get to see it 99% of the time. End of story. They really don't care about systems that don't process financial information! They don't care about your dev or qa environments. They don't care about your DNS servers or your switches or much else for that matter. Regulators are 100% laser-focused on financial information and transactions. They want to see ledgers and logs and they want to see evidence that your systems prevent tampering. That's it. There's no financial regulators that actually audit IT stuff. We probably should have them but we don't. The closest is the FFIEC but they only publish non-binding guidelines. If you think the PCI-DSS matters to banks you're mistaken. Every year we audit ourselves and put the results in a filing cabinet somewhere. We have no obligation to show it to anyone and no one would hold us accountable for failing to be PCI compliant anyway.

aidos 10y ago

As a Monzo customer I have to say that the product Monzo offers and the features that traditional banks want me to use are also somewhat orthogonal :-) So long as they can satisfy the regulatory requirements, I'd rather they were using whatever else it took to build a great product.

deleted 10y ago

[deleted]