30 ms·
How we secure Monzo's banking platform
- raesene9 4y agoGood to see some practices like default deny networking (ingress and egress) and very limited interactive production access being laid out here. A couple of other areas that aren't mentioned, although perhaps they're still doing them are around container breakout risks. There's no mention of what (if any)hardening is being done on the container runtime, either restrictive seccomp, Apparmor/SELinux policies or using something like gVisor/Firecracker. With this year's number of container breakout CVEs, seems like an important area. A related one is whether container aware runtime security is being used to detect where an attacker might have got access to a single container and be trying to breakout to either the underlying platform or to other containers in the cluster.
- TruthWillHurt 4y agoExactly. containers are not secure sandboxes by default and if one is breached all those K8s networking ACLs are worthless.
- csmpltn 4y ago> "Exactly. containers are not secure sandboxes by default and if one is breached all those K8s networking ACLs are worthless." Your suggestion being? Putting a sandbox inside a sandbox? How many layers deep should this be, before being considered "secure"?
- chrisseaton 4y agoI don't think this is the bottomless pit that you think it is. A virtualised instance is a lot more secure than a container, and it's probably fine to stop at virtualised instances.
- csmpltn 4y agoA lot more secure? In what ways?
- chrisseaton 4y agoContainers are really a kind of process-isolation - you still share a kernel. You can find a lot of people saying that containers aren’t enough for running untrusted user code. If you run a fully virtualised instance you get your own kernel and aren’t relying on process isolation. Would you be happy if your cloud provider was running your containers on the same virtual I stance as someone else’s? Most people wouldn’t be.
- csmpltn 4y agoThe only meaningful difference between breaking out of a process-isolated "container" and a full-blown VM is what's waiting for you outside once you've broken out. Whether it's kernel/OS or a bare metal hypervisor isn't really all that meaningful: exploits and vulnerabilities exist for either. There should be proper hardware-level isolation here, depending on the scenario. Most cloud companies can't afford that though, because they're not rolling out their own hardware.
- chrisseaton 4y agoGenuinely, would you be happy with just container isolation between you and other customers of your cloud provider? Most people absolutely would not.
- csmpltn 4y ago> "Genuinely, would you be happy with just container isolation between you and other customers of your cloud provider? Most people absolutely would not." But that's exactly how VPS hosting works today - you don't get your own private blade unless you're ready to pay premium prices and have the competence needed to run them yourself. The technicalities of how private resources in a VPS are isolated from each other will differ, but the concept remains the same nonetheless. People bite the bullet, only to be subject to things like rowhammer [1], or other container escape scenarios [2]. The top comment in this thread reflects the proper way of dealing with this: containers or sandboxes are may not be treated as a secure boundary. [1] https://www.usenix.org/conference/usenixsecurity16/technical-sessions/presentation/xiao https://www.usenix.org/conference/usenixsecurity16/technical... [2] https://www.intezer.com/blog/research/how-we-escaped-docker-in-azure-functions/ https://www.intezer.com/blog/research/how-we-escaped-docker-...
- kasey_junk 4y agoMost serious security teams do not consider containers a security boundary. So it’s not a sandbox inside a sandbox, it’s just a sandbox. Gvisor and firecracker are the most popular sandboxes for containerized workloads.
- staticassertion 4y agoI think this is outdated. Docker is a security boundary. There is no built-in way to get out of a Docker container just by asking by default (if you mount the socket into the container, it's trivial). How good of a boundary it is may be another story. There's some seccomp filters going on and namespacing is pretty sweet too. But an attacker can escape by exploiting the kernel, which I think most security people would consider to be not particularly high effort. So, suitable for internal services that you generally trust, not suitable for hostile code or highly exposed services. In an ideal world maybe we'd all use Firecracker but it's not nearly as easy to do that vs just putting something in a container.
- TrueDuality 4y agoThe reason that containers are not generally considered a security boundary is that many of the namespace primitives were _not designed_ as a security layer, they aren't designed to actively reduce the privileges from the current user's context. Since most containers are started as the root user, the namespace transition inherits root's permissions even if they're later dropped. Without SELinux or seccomp restrictions, root can still pretty much do anything to the host even inside the containers. For the most part this is troublesome when parts of the kernel or host userspace code are not fully aware of the different forms of namespacing (there are still portions that just check for an effective UID of 0, without checking whether they're in a namespace for example). These are the components where a lot of container breakouts happen and is largely mitigated by having internal processes in the container not running as root in the namespace. Dropping privileges to a different user still trace's it origin back to the root user on the host, so in some cases being partially aware of namespaces in a section of the kernel or host user code actively hurts the security by tracing the user back to root and using those privileges again. SELinux really tightens the potential to pull these shenanigans, but most production k8s clusters at least that I've seen are built on Ubuntu where those protections aren't available. In this case the security layer is once again SELinux not the namespacing. As long as the container runtime is performing the various namespace isolation primitives starting from the root user these container bypasses are going to be a risk. There are 'rootless' versions of containers which can only use the privileges available to lower (presumably heavily restricted) user but those aren't widely used. Once again this is relying on the security protections of the host user authorization, not on the namespaces. The networking analogy is NAT. People treat it like a security layer as it kind-of-sort-of looks like an ingress firewall since you can't directly address devices inside a NAT, but its not and can be pierced pretty easily. NAT is not a firewall. Namespaces are not a security layer.
- TruthWillHurt 4y agoOpposite - don't mess with sandboxing. Use PaaS services like its > 2008, and let AWS / Google security teams harden their platform.
- staticassertion 4y ago> With this year's number of container breakout CVEs, seems like an important area. Worth noting that even basic hardening in docker will prevent a lot of them. I say "in docker" because K8s disables seccomp, which matters a lot since `unshare` is denied by docker's seccomp and is very useful for attackers in a container. If you use Docker the main thing to do is just not run as root. If you do that much, and it's not hard at all, you are in a much better place than a default k8s pod.
- lbriner 4y agoI think it's great that more companies are open about what they are doing for security. It makes it sound like they are confident in their abilities unlike other people who are nervous to mention things like "We use Octopus" or "We use AKS" because we are less confident that the information is not an invitation to a hacker! Now all we need is to somehow capture some of this "best practice" and make it normal practice, enabled by default and documented well so that organisations don't set stuff up and then disable all the controls because it is too hard to understand.
- 878654Tom 4y agoOne reason why companies would not do this is to give a little bit of protection against zero-days. When a zero-day is released all providers notice a huge scan for the vulnerability. Scanning huge blocks of the internet takes time but if a hacker has a list of companies using which tools and where it can be narrowed down a lot. AWS/Azure/GCP/... for example have published IP-ranges of services. If a zero-day for any of those services is released a hacker can already narrow down the attack-range and gain a lot of time.
- NicolaiS 4y agoThat seems like a bad reason. With a good enough connection and `masscan` you can "scan the whole internet" (single port) in 5 min. Security through obscurity on IPv4 make no sense.
- andrewstuart 4y agoShould banking really be on a cloud platform? I do believe AWS is likely far more secure than any DIY computing environment but even so, should banking be on cloud infrastructure? I'm not saying I think this is a bad idea but it came to mind when I read this. Also, is it really a good idea for a bank to be talking openly about its security strategy? Isn't an important part of security not to let on anything that might be used against you? For example if determined hackers know your systems then they can keep an eye out long term for vulnerabilities in those technologies and be ready to strike. Does this sort of thing matter or not?
- carnitine 4y agoI don’t see why banking shouldn’t be on a cloud platform, you’re not really giving any reason why we should question it either. As to your second point, security through obscurity is generally believed to not be worthwhile.
- andrewstuart 4y ago>> I don’t see why banking shouldn’t be on a cloud platform What if AWS gets cracked/hacked/compromised? I know it's not happened yet, but it's not impossible.
- tome 4y agoI guess there are two important questions: * For individual banks and their customers, is it more likely that an AWS-wide exploit will compromise an AWS-hosted bank, or is it more likely that a self-hosting-specific exploit will compromise a self-hosted bank? * For society, is it better that security efforts are concentrated in on centralised providers like AWS, or is it better that security efforts are distributed, on individual hosting entities?
- dmitriid 4y agoThat's more or less the same question as "what if the data center/servers operated by the bank gets compromised". In reality it's always about tradeoffs: who to delegate to and who to trust.
- Narkov 4y ago> more than 20,000 containerised workloads across more than 2000 microservices to date. This is insane. What am I missing here that an organization is bragging about having 2000 moving parts?
- ivan_gammel 4y agoThose moving parts are defined by the complexity of the business. Banking software with 20000 classes deployed in J2EE application server on a mainframe would not be much different.
- nly 4y agoThat's not really true, since microservices involve what boils down to RPC over a network. There are so many more failure modes involved when you have 2000 asynchronous processes talking to one.
- jacquesm 4y agoThat is true, but it also allows for much more orderly start-up and shut-down as well as automatic recovery. A service is a pretty well defined entity that can be exhaustively tested far easier than the corresponding monolith with 2000 classes and tons of non-local effects. To use processes for that purpose has definitive advantages. See "Erlang/OTP" for an example of how this can give you incredibly solid distributed architectures.
- chrisseaton 4y agoNot sure they're bragging about it - they're just stating it as context for their blog post. And is 2000 moving parts too many? How many moving parts do you need to run a bank? I can imagine they're having to comply with ~2000 legislation clauses, for example. Isn't that just the complexity of their domain?
- andrewingram 4y agoFrom my understanding the approach to microservices was more or less a day 1 thing due to some of the early engineering hires being very experienced with them; so the number of them was comparatively high pretty early on.
- andrewstuart 4y agoThe biggest security hole for every organisation is its remote work from home workers. I'd be interested to hear how this Monzo bank addresses the problem of someone walking in the home of one of their programmers and lifting access keys to AWS whilst that person is at the supermarket, and leaving with no-one the wiser. Or installing a keylogger USB device onto their keyboard cable.
- gbrindisi 4y agoIt's not like office networks are a security paradise either
- JMS2021 4y agoYou wouldn't run or develop code locally. AWS keys would be secrets managed by Vault or something. If you have AWS keys on staff laptops at home, you've already failed. We don't allow any code at all on local machines.
- vladvasiliu 4y agoI'd say the keylogger can be an issue if they're able to be alone with the computer for a while. I'm not sure that all laptops can detect that they've been opened (my HP Elitebook and previous Probooks don't), but I'd assume it unlikely that the attacker wouldn't leave other traces in the house. But other than that, enforcing session auto-locking should work fairly well. Of course, if this is combined with some kind of agent that checks whether you're doing something that the employer defeated with a mouse jigger, all bets are off... They can also enforce using MFA for AWS (and probably for GCP and Azure, too, but I don't use those) and not use plain access keys.
- mocko 4y agoDisgruntled former Monzo customer here. Do they still have a haywire fraud detection system that randomly freezes innocent people's accounts? It's happened to countless users and the customer experience when they do it ("we refuse to tell you why" and in some cases holding onto their money for months) is a kafkaesque nightmare. https://www.vice.com/en/article/bvg7n3/monzo-freezing-closing-accounts-complaints https://www.vice.com/en/article/bvg7n3/monzo-freezing-closin... https://www.reddit.com/r/UKPersonalFinance/comments/db9j5z/monzo_locked_my_account_and_held_my_funds_for_3/ https://www.reddit.com/r/UKPersonalFinance/comments/db9j5z/m... https://www.theguardian.com/money/2020/jun/06/monzo-customers-shut-out-of-accounts-in-lockdown https://www.theguardian.com/money/2020/jun/06/monzo-customer... I'd be far more interested in a blog post explaining that than how great their infrastructure is.
- lnxg33k1 4y agothe fear of that happening to me was the reason I terminated my account too, reading people having their main account frozen out of nowhere for months was very concerning
- dwb 4y agohttps://monzo.com/blog/2019/04/04/why-we-block-freeze-close-monzo-accounts/ https://monzo.com/blog/2019/04/04/why-we-block-freeze-close-...
- mocko 4y agoSure, but that doesn't explain the sheer scale on which it happens (more than other UK banks and with seemingly worse consequences) or how the system actually works. It's also notable that this post was written a year before some of the news articles, indicating that Monzo really hadn't done anything to improve it. One gets the feeling they're hiding behind the "we can't tell you why" excuse to escape accountability for a broken system that's literally ruining lives.
- Milner08 4y agoThey are 100% not behind the 'We cant tell you why'. It only takes the smallest amount of reading to figure that out. They have to report it to the FCA and you can read the reports from the FCA that suggest its not actually any worse than other banks. Its just more talked about. (No I am not going to dig them out for you).
- m00dy 4y agowell, DeFi is secured by default.
- deleted 4y ago[deleted]
- stavros 4y agoThe "S" in DeFi stands for Security.
- wdb 4y agoOne day I would love to see what having 2.000 microservices entails. Which features each service covers. I can't think of 2.000 micro services an online bank would have.
- staticassertion 4y agoMaybe they mean instances? That doesn't seem too crazy.