5 ms·

>what really limits most financial institutions from embracing a lot more modern tech is their core systems of record AND the acceptance of said systems by thei

>what really limits most financial institutions from embracing a lot more modern tech is their core systems of record AND the acceptance of said systems by their governing agencies. I just want to say that I think there is a huge amount of FUD about how you can and cannot build your technology as a regulated entity – and in particular as a bank. In reality, close to 100% of requirements from a regulator will tell you _what_ you must build, not _how_ you must build it. Even then, especially in terms of resilience and security, they are almost always a subset of our own requirements. What can be more of a challenge is convincing an auditor that what you have done is acceptable, since it can be so different from what they may have seen before. Again, I don't think this is a reason to compromise. We see technology as a major competitive advantage, so it is worth the effort to find open-minded auditors, and spend time to explain and demonstrate how (and why) our software meets the requirements. I don't think there's any way we could build a secure, resilient bank with the kind of product experience we want, AND do it on the budget of a startup if we approach technology through the same lens as existing banks.

tluyben2 10y ago
> I just want to say that I think there is a huge amount of FUD about how you can and cannot build your technology as a regulated entity – and in particular as a bank. In reality, close to 100% of requirements from a regulator will tell you _what_ you must build, not _how_ you must build it. Even then, especially in terms of resilience and security, they are almost always a subset of our own requirements. Having worked with banks and insurers solely in Java and connecting to legacy (mostly Cobol) before, I was surprised, in my current position, to see some companies doing their complete banking back end in PHP & MySQL. I knew the how is not part of the regulations, but I did expect the CTOs to pick the 'no one ever got fired for choosing' choice.
- webscaleizfun 10y ago
  It seems common knowledge these days among the slightly but not too technically inclined that any new major project should use a LAMP stack as its base. People see Facebook, Amazon, and many others running PHP & MySQL on Linux at scale and they know it works reliably, so while it may not have the support of Cisco or Oracle, it is pretty close on the 'no one ever got fired for choosing' X scale, since you can point to every other major company using these building blocks reliably if your investors, CEO, board or auditors asked why you chose to use PHP & MySQL. In summary, PHP & MySQL have become the modern equivalent of a "safe" choice for your stack to be built on. Its not necessarily a bad choice either, you get access to a large community of skilled people who can write PHP & write SQL statements, and while everyone likes to hate on PHP, it isn't about to up and disappear any time in the next decade either (unlike COBOL).
  - lgieron 10y ago
    I'd say the reliability the bank is looking for is much higher than what's acceptable for web companies (i.e. web companies are fine with eventual consistency, which would is obviously unacceptable in a banking system outside of trivial non-core features).
    - UK-AL 10y ago
      I would assume its pretty hard to make something ACID style when doing stuff across Microservices, which i'm assuming mondo is doing.
      - KirinDave 10y ago
        That part is surprisingly easy if you architect it right. The core abstraction most banks use is your "available balance" and the fact that they can reconcile on a longer time period than seconds.
    - jerf 10y ago
      I'd also suggest that it's not just scale; the kind of reliability Facebook needs is fundamentally different than what a bank needs. Broadly speaking, Facebook needs the site to keep working as well as possible even if some subservice fails, and a bank needs a subservice not to fail. I'm summarizing here and I know it; clearly neither of them is actually on the absolute extreme end, as Facebook needs authentication to work and a bank may not care if the interest rate display widget on their customer banking app fails to load a couple of times. But I'd still suggest there's enough difference between the requirements to be a fundamentally different domain. Even in "the cloud" things differ between services. A social media app has very different reliability requirements than a backup cloud.
      - hueving 10y ago
        Well actually there are many sub services in a bank that can go down without major impacts. The two major banks I use have weekly planned outages of features like old statement retrieval, person to person payments, ACH transfers, etc. Basically everything in the web interface could experience outages without any major crisis. As long as ATM requests always work, nobody really seems to care.
- tormeh 10y ago
  PHP in banking? That's scary. Pretending to be Facebook and being able to handle PHP development looks like a bad judgment call from where I'm standing. PHP seems like it is made for everything banks are not. I have the impression PHP is hard to do securely, invites bugs but let you get going quickly. This sounds like Lockheed deciding to use C++ for the F35.
  - obeattie 10y ago
    To be clear, we (Monzo) are not writing banking software in PHP, or using MySQL.
- KirinDave 10y ago
  > , I was surprised, in my current position, to see some companies doing their complete banking back end in PHP & MySQL. Glad to know! Now I need to find these folks so I can hide in a dumpster from them!
- deleted 10y ago
  [deleted]
KirinDave 10y ago
> I just want to say that I think there is a huge amount of FUD about how you can and cannot build your technology as a regulated entity – and in particular as a bank. In reality, close to 100% of requirements from a regulator will tell you _what_ you must build, not _how_ you must build it. Even then, especially in terms of resilience and security, they are almost always a subset of our own requirements. I'm not sure what your involvement here is, and mine is limited (thankfully) to a substantial distance. However, I think there is a rather big difference between different FI's experience; because it's a true game of politics. When investigating if Level Money should be a bank (and learning that almost no one wants to be a bank, it's very hard to make money just being a direct-to-consumer deposit bank unless you're quite scaled), I basically had a surprisingly credulous audience because the CFPB was basically willing to doorbuster anything that they thought would spur the big banks to action. It was... very surprising to ultimately decide that it was impossible for financial reasons to actually succeed at being a bank. But they will insist on things that every bank should have, like a credible way for analytics to run in an environment where the raw data is not subject to exfiltration by a compromised data scientist's machines without an audit trail a mile wide. Good luck to you.
- mjs 10y ago
  > I'm not sure what your involvement here is … Parent poster is Head of Engineering at the bank (not CTO?), and author of the slides.
  - KirinDave 10y ago
    Yes but that may or may not mean direct involvement with regulators. For small companies it's really hard to tell where they are in the process.
    - obeattie 10y ago
      We're _very_ directly involved with regulation, and with regulators :-) It's a very important part of our business, so we put a lot of effort into ensuring that we please regulators _and_ that we run our business in the way we want.
riskable 10y ago
I work for a huge bank and the system of record thing is spot-on. I don't think there's any law that says we must operate systems of record in certain ways or keep records of every little thing but remember that this is banking. Keeping records of every little thing is practically the religion of banking! So for a long time now the big banks have been trying (and often failing) to keep track--centrally--of every little server or device that pops up on their networks. So there was a big push recently (few years ago) at the big banks to improve their systems of record. The goal being mostly related to better financial (asset) tracking. So they can figure out which internal teams were using the most/least resources as well as figure out who's not upgrading their stuff on a regular basis (technical debt builders). So they spent all this money improving their systems of record and in walks Docker. It practically turns the entire concept of having a central place to track "systems" on its head! To give you an example of the difficulties: We have loads of policies that say things like, "all systems/applications must be registered in <system of record>." Sounds simple enough: Just make sure that wherever a Docker container comes up we create a new record in the system of record and remove it when it comes down. Except it's not that simple for many reasons the most obviously problematic of which is that the "system of record" works in batch. As in, you submit your request to add a new record and then maybe 8 hours later it'll show up. Did I mention that there's also policies that say you can't put any system into production until it shows up in the system of record? =) That's just scratching the surface though. Because the system of record at most financial institutions doesn't just allow you to delete records. Once you create one it is there forever. It merely gets marked as "retired" (or similar) and most banks require phases as well. For example, before a production system can be marked as retired it must first go through a mandatory, "waning" period (what it's called depends on the bank) that can often be weeks. I can go on and on about all the zillions of ways in which systems of record (and the policies that go with them) are anathema to usage of Docker but I think everyone reading should "get the picture" at this point. If not, just imagine the havoc that would entail when you have thousands of Docker containers coming up and down every second. Or the entire concept of a container only being up for a few seconds to perform a single batch operation (banks love batch, remember!). If you think the system of record requirements make adoption of Docker difficult you should know that the security policies are worse! Imagine a policy that states that all systems must undergo a (excruciatingly slow) security scan before being put into production. That's just one of the headaches, sigh.

tluyben2 10y ago

webscaleizfun 10y ago
It seems common knowledge these days among the slightly but not too technically inclined that any new major project should use a LAMP stack as its base. People see Facebook, Amazon, and many others running PHP & MySQL on Linux at scale and they know it works reliably, so while it may not have the support of Cisco or Oracle, it is pretty close on the 'no one ever got fired for choosing' X scale, since you can point to every other major company using these building blocks reliably if your investors, CEO, board or auditors asked why you chose to use PHP & MySQL. In summary, PHP & MySQL have become the modern equivalent of a "safe" choice for your stack to be built on. Its not necessarily a bad choice either, you get access to a large community of skilled people who can write PHP & write SQL statements, and while everyone likes to hate on PHP, it isn't about to up and disappear any time in the next decade either (unlike COBOL).
- lgieron 10y ago
  I'd say the reliability the bank is looking for is much higher than what's acceptable for web companies (i.e. web companies are fine with eventual consistency, which would is obviously unacceptable in a banking system outside of trivial non-core features).
  - UK-AL 10y ago
    I would assume its pretty hard to make something ACID style when doing stuff across Microservices, which i'm assuming mondo is doing.
    - KirinDave 10y ago
      That part is surprisingly easy if you architect it right. The core abstraction most banks use is your "available balance" and the fact that they can reconcile on a longer time period than seconds.
  - jerf 10y ago
    I'd also suggest that it's not just scale; the kind of reliability Facebook needs is fundamentally different than what a bank needs. Broadly speaking, Facebook needs the site to keep working as well as possible even if some subservice fails, and a bank needs a subservice not to fail. I'm summarizing here and I know it; clearly neither of them is actually on the absolute extreme end, as Facebook needs authentication to work and a bank may not care if the interest rate display widget on their customer banking app fails to load a couple of times. But I'd still suggest there's enough difference between the requirements to be a fundamentally different domain. Even in "the cloud" things differ between services. A social media app has very different reliability requirements than a backup cloud.
    - hueving 10y ago
      Well actually there are many sub services in a bank that can go down without major impacts. The two major banks I use have weekly planned outages of features like old statement retrieval, person to person payments, ACH transfers, etc. Basically everything in the web interface could experience outages without any major crisis. As long as ATM requests always work, nobody really seems to care.
tormeh 10y ago
PHP in banking? That's scary. Pretending to be Facebook and being able to handle PHP development looks like a bad judgment call from where I'm standing. PHP seems like it is made for everything banks are not. I have the impression PHP is hard to do securely, invites bugs but let you get going quickly. This sounds like Lockheed deciding to use C++ for the F35.
- obeattie 10y ago
  To be clear, we (Monzo) are not writing banking software in PHP, or using MySQL.
KirinDave 10y ago
> , I was surprised, in my current position, to see some companies doing their complete banking back end in PHP & MySQL. Glad to know! Now I need to find these folks so I can hide in a dumpster from them!
deleted 10y ago
[deleted]

KirinDave 10y ago

> I just want to say that I think there is a huge amount of FUD about how you can and cannot build your technology as a regulated entity – and in particular as a bank. In reality, close to 100% of requirements from a regulator will tell you _what_ you must build, not _how_ you must build it. Even then, especially in terms of resilience and security, they are almost always a subset of our own requirements. I'm not sure what your involvement here is, and mine is limited (thankfully) to a substantial distance. However, I think there is a rather big difference between different FI's experience; because it's a true game of politics. When investigating if Level Money should be a bank (and learning that almost no one wants to be a bank, it's very hard to make money just being a direct-to-consumer deposit bank unless you're quite scaled), I basically had a surprisingly credulous audience because the CFPB was basically willing to doorbuster anything that they thought would spur the big banks to action. It was... very surprising to ultimately decide that it was impossible for financial reasons to actually succeed at being a bank. But they will insist on things that every bank should have, like a credible way for analytics to run in an environment where the raw data is not subject to exfiltration by a compromised data scientist's machines without an audit trail a mile wide. Good luck to you.

mjs 10y ago
> I'm not sure what your involvement here is … Parent poster is Head of Engineering at the bank (not CTO?), and author of the slides.
- KirinDave 10y ago
  Yes but that may or may not mean direct involvement with regulators. For small companies it's really hard to tell where they are in the process.
  - obeattie 10y ago
    We're _very_ directly involved with regulation, and with regulators :-) It's a very important part of our business, so we put a lot of effort into ensuring that we please regulators _and_ that we run our business in the way we want.

riskable 10y ago

I work for a huge bank and the system of record thing is spot-on. I don't think there's any law that says we must operate systems of record in certain ways or keep records of every little thing but remember that this is banking. Keeping records of every little thing is practically the religion of banking! So for a long time now the big banks have been trying (and often failing) to keep track--centrally--of every little server or device that pops up on their networks. So there was a big push recently (few years ago) at the big banks to improve their systems of record. The goal being mostly related to better financial (asset) tracking. So they can figure out which internal teams were using the most/least resources as well as figure out who's not upgrading their stuff on a regular basis (technical debt builders). So they spent all this money improving their systems of record and in walks Docker. It practically turns the entire concept of having a central place to track "systems" on its head! To give you an example of the difficulties: We have loads of policies that say things like, "all systems/applications must be registered in <system of record>." Sounds simple enough: Just make sure that wherever a Docker container comes up we create a new record in the system of record and remove it when it comes down. Except it's not that simple for many reasons the most obviously problematic of which is that the "system of record" works in batch. As in, you submit your request to add a new record and then maybe 8 hours later it'll show up. Did I mention that there's also policies that say you can't put any system into production until it shows up in the system of record? =) That's just scratching the surface though. Because the system of record at most financial institutions doesn't just allow you to delete records. Once you create one it is there forever. It merely gets marked as "retired" (or similar) and most banks require phases as well. For example, before a production system can be marked as retired it must first go through a mandatory, "waning" period (what it's called depends on the bank) that can often be weeks. I can go on and on about all the zillions of ways in which systems of record (and the policies that go with them) are anathema to usage of Docker but I think everyone reading should "get the picture" at this point. If not, just imagine the havoc that would entail when you have thousands of Docker containers coming up and down every second. Or the entire concept of a container only being up for a few seconds to perform a single batch operation (banks love batch, remember!). If you think the system of record requirements make adoption of Docker difficult you should know that the security policies are worse! Imagine a policy that states that all systems must undergo a (excruciatingly slow) security scan before being put into production. That's just one of the headaches, sigh.