9 ms·

Author of the talk here. Happy to answer any questions anyone has. This post also contains more info on how we build our systems: https://monzo.com/blog/2016/09

Author of the talk here. Happy to answer any questions anyone has. This post also contains more info on how we build our systems: https://monzo.com/blog/2016/09/19/building-a-modern-bank-backend/ https://monzo.com/blog/2016/09/19/building-a-modern-bank-bac... And as an aside, I'll also be giving a longer talk at Kubecon going into more detail on some of this stuff :-)

olalonde 10y ago
I hadn't heard of linkerd before. How does it compare to etcd or consul and why did you chose it?
- lobster_johnson 10y ago
  Linkerd is not very similar to Etcd or Consul. It's a proxy that sits between your microservices/apps to act as a middleman for RPC calls. Being a proxy, it can handle things like load balancing, throttling and failure handling, meaning you don't have to build that into each app. When used together with Kubernetes, Linkerd will use Kubernetes for discovery, so you don't need Etcd or Consul directly (K8s itself relies on Etcd, though).
KirinDave 10y ago
Not that I don't appreciate the subject matter, I wave the flag for K8S all the time and I've shipped bank product presentation stuff on it... But... The fundamental challenges of building a bank are almost entirely orthogonal to things like distributed system uptime and resiliency (unless, I suppose, you could lose consistency during the types service loss Kubernetes makes easy to ameliorate). Evidence for this abounds: nearly every major bank out there is at least 10 years behind the tech we're discussing here. Sure, banks are getting savvy to modern techniques for their presentation and API layers (my employer, Capital One, for example). But the actual challenges are data consistency and liveliness and audit-ability are preserved. I'm really curious if you're using a novel technique to achieve this that Docker and self-managed micro-app swarms can deliver on better. Because what really limits most financial institutions from embracing a lot more modern tech is their core systems of record AND the acceptance of said systems by their governing agencies. So when we talk about basic datacenter ops, that's all great. But I don't think they're the things that make K8S great for a bank. I think most people in a position to evaluate a financial institution roadmap would be unmoved by this deck. Now, if you talk about other things we both know K8S is great for... Things like discovering the genesis of a database action by preserving it throughout the chain of responders, or being able to rapidly respond to site exploits with rolling restarts and a built in mechanism for feature flagging, or having a really great way to offer data scientists the environments they need without risk for data theft, or being able to use traditional CI/CD methodologies but end up with a single deployable unit that is amenable to both automated an manual review and mechanical deployment in spite of the tooling used within. Not that I think selling K8S is your job. But I thought I'd mention the perspective of someone doing inf modernization and product work at a major bank. And of course, as always: the opinions above are my own and not those of my employer or co-workers.
- nstj 10y ago
  > what really limits most financial institutions from embracing a lot more modern tech is their core systems of record AND the acceptance of said systems by their governing agencies I tend to think (from experience) that the reason banks don't embrace modern tech is inertia - they've been able to "milk the cow" of regulatory body sanctioned profits for so long that they're in a mindset of not wishing to introduce volatility into what has been a highly predictable revenue stream. They've been using garbage technology for so long and making money at the same time that there has been no institutional impetus to innovate and embrace new tech. This is now coming to bite them in the ass as regulatory bodies are allowing new players to market and therefore eroding some of the guaranteed profits banks enjoyed up until now. I doubt most large capital markets institutions (on both the retail and IB sides) will be able to weather the current (and impending) storm of disruption and come out unscathed.
  - raesene6 10y ago
    I'd say inertia is a large factor but also risk aversion. Bank systems have to be very available (or they attract large fines) and so there's a tendency to stick with things that are proven to be robust even when they have other problems. The other major problem I've seen is that most banks don't think of themselves as technology companies, and treat IT as an overhead to be minimized, which is absolutely the wrong approach. This tends to lead to things that save money in the short term (e.g. outsourcing deals) but could well cost money in the longer term, as they make replacing legacy systems harder.
    - riskable 10y ago
      This is nonsense. Not every system in banks needs to be highly available. That's just silly. Like the print server on the 3rd floor needs an active standby! Haha. No, just like any organization banks have "critical" systems and "everything else." Docker is mostly being "sold" as a means to replace and improve the non-critical stuff. Like that internal web app everyone uses to look up <whatever>. Or the system that generates daily reports on <whatever>. Just like most organizations, banks have a few critical systems and everything else is less so (to varying degrees).
Annatar 10y ago
The most valuable things in the talk were linkerd and BGP (especially configuration of Zebra and linkerd), unfortunately both were only glossed over briefly. Has linkerd been ported over to SmartOS? (So far I haven't found anything saying one way or the other.) For your next talks, please describe as many failure scenarios as you can think of, and what happens at the OS / application / network level. The more concrete failure examples you can provide, the better. The why's are clear. The mechanisms are not.
- obeattie 10y ago
  This is great feedback – thank you. A lot of the feedback from my blog post a month or so ago was that the "why" was not clear enough, so I guess I focussed more on this. Now I've (hopefully!) covered that better, I agree that more depth on these topics would be helpful.
- moderation 10y ago
  > Has linkerd been ported over to SmartOS? (So far I haven't found anything saying one way or the other.) linkerd is a JVM application. It should run fine under SmartOS with a JVM. I run it successfully on a Raspberry Pi, OS X, Linux etc.
ktamura 10y ago
What are you using for container/pod logging?
- obeattie 10y ago
  Good question. We have a couple of approaches to this: * Every request that comes into our system is assigned a unique ID, which is propagated on every downstream call and returned in a response header. When logs are emitted during request processing, they are tagged with this ID. A system we've built in-house indexes these log events against their trace ID in Cassandra (on a separate cluster). This lets us take a failing, slow, or otherwise interesting request and look up all the things that happened to it during processing. Events in this system are TTL'd according to their severity – so an event at critical or error severity is kept longer than one at debug severity. * stdout/stderr from all our containers is forwarded to journald on each host. Logstash then pushes all these logs to Elastic (and also to permanent cold storage). This is useful to look at the "big picture" and means we can analyse all the logs in aggregate and makes it very obvious when something is _very_ wrong and causing a lot of requests to fail, but is less useful than slog for pinpointing a specific issue. It's worth also noting (since I find often a whole load of things can be mixed into logging) that we do not drive our monitoring off these logs, at least at the moment. We have separate systems for that.
  - sandGorgon 10y ago
    this is awesome! how do you do this? a new header with a unique id... generated by something like lua+nginx. but then how do you pass this request from one service to another?
    - amatix 10y ago
      For us, a `X-Request-ID` header is generated by any app if it doesn't receive it from upstream -- but normally nginx or the CDN will generate it. There's a few nginx modules to do it, we use https://github.com/newobj/nginx-x-rid-header https://github.com/newobj/nginx-x-rid-header Most languages/logging frameworks have some sort of per-thread context (eg. Filters in Python, MDC in log4j, etc) to be able to tag log messages with. If you're using postgresql, you can call `SET application_name='{requestID}';` and that can be output as part of logs too.
raesene6 10y ago
Hi, I'd be very interested to hear if you've come across any good documentation on hardening Kubernetes? From what I've seen so far there's very limited documentation on that when compared to other components like Docker, and the defaults can sometimes not be suitable for a high security environment (e.g. https://raesene.github.io/blog/2016/10/08/Kubernetes-From-Container-To-Cluster/ https://raesene.github.io/blog/2016/10/08/Kubernetes-From-Co... )
- ben_hall 10y ago
  There is a solution to this coming in v1.5 - https://github.com/kubernetes/kubernetes/pull/32518 https://github.com/kubernetes/kubernetes/pull/32518
  - raesene6 10y ago
    yeah I've got that noted in the blog post :) but it was really just an example. One of the things thats struck me about docker/Kubernetes etc is that they tend to be tuned for a general use case in terms of security and configuration. Choices that might improve security and restrict usefulness of services are not usually defaults. As such there needs to be a level of hardening done from an out of the box perspective where they're being used in a high-security environment (e.g. banking). For Docker we have resources like docker_bench and the CIS guide which provide a list of possible hardening steps, but I've not managed to find anything like that for Kubernetes, which is why I'm interested in how Monzo are addressing that issue.
    - dimastopel 10y ago
      Not sure if you are looking for a commercial solution, but we (Twistlock [1]) develop a security suite for enterprises working with Docker and / or Kubernetes. In fact, we are officially recommended by Google for working with GKE which is very much based on Kubernetes [2]. I'd be glad to elaborate if relevant. [1] https://www.twistlock.com/ https://www.twistlock.com/ [2] https://cloudplatform.googleblog.com/2015/11/enhancements-to-Container-Engine-and-Container-Registry.html https://cloudplatform.googleblog.com/2015/11/enhancements-to...
nvarsj 10y ago
What kind of resource usage do you see with linkerd? It looks good, but I'm concerned about the resource overhead of a JVM based proxy on every node / pod. edit: Ok answered myself - found https://blog.buoyant.io/2016/06/17/small-memory-jvm-techniques-for-microservice-sidecars/ https://blog.buoyant.io/2016/06/17/small-memory-jvm-techniqu.... So with some work it can be reduced from 500Mi->100Mi per instance. It'd be interesting to see the kind of CPU time it uses under load, though.
tonyhb 10y ago
If K8S secrets management is storing them in plain-text in etcd how do you go about making it actually secure for a bank?
- obeattie 10y ago
  We don't store things that are actually secret, as k8s secrets ;-) Hashicorp Vault is quite good at this.
  - tonyhb 10y ago
    Wahoo! Good to know. Definitely don't trust the security by default :)
justifier 10y ago
this is an excellent talk a link to the video: https://skillsmatter.com/skillscasts/9146-building-a-bank-with-kubernetes https://skillsmatter.com/skillscasts/9146-building-a-bank-wi... thanks for giving and posting it i especially appreciate the effort because a month ago i posted this(o) comment: > can anyone suggest reading to understand how contemporary banks function, where can i get an understanding of a bank or credit union from a software engineer's perspective: dependencies, steps to start, challenges of running, protections from common problems, interesting emerging disruptions; best of luck with monzo, https://monzo.com/ https://monzo.com/ (o) https://news.ycombinator.com/item?id=12567536 https://news.ycombinator.com/item?id=12567536

olalonde 10y ago

I hadn't heard of linkerd before. How does it compare to etcd or consul and why did you chose it?

lobster_johnson 10y ago
Linkerd is not very similar to Etcd or Consul. It's a proxy that sits between your microservices/apps to act as a middleman for RPC calls. Being a proxy, it can handle things like load balancing, throttling and failure handling, meaning you don't have to build that into each app. When used together with Kubernetes, Linkerd will use Kubernetes for discovery, so you don't need Etcd or Consul directly (K8s itself relies on Etcd, though).

KirinDave 10y ago

Not that I don't appreciate the subject matter, I wave the flag for K8S all the time and I've shipped bank product presentation stuff on it... But... The fundamental challenges of building a bank are almost entirely orthogonal to things like distributed system uptime and resiliency (unless, I suppose, you could lose consistency during the types service loss Kubernetes makes easy to ameliorate). Evidence for this abounds: nearly every major bank out there is at least 10 years behind the tech we're discussing here. Sure, banks are getting savvy to modern techniques for their presentation and API layers (my employer, Capital One, for example). But the actual challenges are data consistency and liveliness and audit-ability are preserved. I'm really curious if you're using a novel technique to achieve this that Docker and self-managed micro-app swarms can deliver on better. Because what really limits most financial institutions from embracing a lot more modern tech is their core systems of record AND the acceptance of said systems by their governing agencies. So when we talk about basic datacenter ops, that's all great. But I don't think they're the things that make K8S great for a bank. I think most people in a position to evaluate a financial institution roadmap would be unmoved by this deck. Now, if you talk about other things we both know K8S is great for... Things like discovering the genesis of a database action by preserving it throughout the chain of responders, or being able to rapidly respond to site exploits with rolling restarts and a built in mechanism for feature flagging, or having a really great way to offer data scientists the environments they need without risk for data theft, or being able to use traditional CI/CD methodologies but end up with a single deployable unit that is amenable to both automated an manual review and mechanical deployment in spite of the tooling used within. Not that I think selling K8S is your job. But I thought I'd mention the perspective of someone doing inf modernization and product work at a major bank. And of course, as always: the opinions above are my own and not those of my employer or co-workers.

nstj 10y ago
> what really limits most financial institutions from embracing a lot more modern tech is their core systems of record AND the acceptance of said systems by their governing agencies I tend to think (from experience) that the reason banks don't embrace modern tech is inertia - they've been able to "milk the cow" of regulatory body sanctioned profits for so long that they're in a mindset of not wishing to introduce volatility into what has been a highly predictable revenue stream. They've been using garbage technology for so long and making money at the same time that there has been no institutional impetus to innovate and embrace new tech. This is now coming to bite them in the ass as regulatory bodies are allowing new players to market and therefore eroding some of the guaranteed profits banks enjoyed up until now. I doubt most large capital markets institutions (on both the retail and IB sides) will be able to weather the current (and impending) storm of disruption and come out unscathed.
- raesene6 10y ago
  I'd say inertia is a large factor but also risk aversion. Bank systems have to be very available (or they attract large fines) and so there's a tendency to stick with things that are proven to be robust even when they have other problems. The other major problem I've seen is that most banks don't think of themselves as technology companies, and treat IT as an overhead to be minimized, which is absolutely the wrong approach. This tends to lead to things that save money in the short term (e.g. outsourcing deals) but could well cost money in the longer term, as they make replacing legacy systems harder.
  - riskable 10y ago
    This is nonsense. Not every system in banks needs to be highly available. That's just silly. Like the print server on the 3rd floor needs an active standby! Haha. No, just like any organization banks have "critical" systems and "everything else." Docker is mostly being "sold" as a means to replace and improve the non-critical stuff. Like that internal web app everyone uses to look up <whatever>. Or the system that generates daily reports on <whatever>. Just like most organizations, banks have a few critical systems and everything else is less so (to varying degrees).

Annatar 10y ago

The most valuable things in the talk were linkerd and BGP (especially configuration of Zebra and linkerd), unfortunately both were only glossed over briefly. Has linkerd been ported over to SmartOS? (So far I haven't found anything saying one way or the other.) For your next talks, please describe as many failure scenarios as you can think of, and what happens at the OS / application / network level. The more concrete failure examples you can provide, the better. The why's are clear. The mechanisms are not.

obeattie 10y ago
This is great feedback – thank you. A lot of the feedback from my blog post a month or so ago was that the "why" was not clear enough, so I guess I focussed more on this. Now I've (hopefully!) covered that better, I agree that more depth on these topics would be helpful.
moderation 10y ago
> Has linkerd been ported over to SmartOS? (So far I haven't found anything saying one way or the other.) linkerd is a JVM application. It should run fine under SmartOS with a JVM. I run it successfully on a Raspberry Pi, OS X, Linux etc.

ktamura 10y ago

What are you using for container/pod logging?

obeattie 10y ago
Good question. We have a couple of approaches to this: * Every request that comes into our system is assigned a unique ID, which is propagated on every downstream call and returned in a response header. When logs are emitted during request processing, they are tagged with this ID. A system we've built in-house indexes these log events against their trace ID in Cassandra (on a separate cluster). This lets us take a failing, slow, or otherwise interesting request and look up all the things that happened to it during processing. Events in this system are TTL'd according to their severity – so an event at critical or error severity is kept longer than one at debug severity. * stdout/stderr from all our containers is forwarded to journald on each host. Logstash then pushes all these logs to Elastic (and also to permanent cold storage). This is useful to look at the "big picture" and means we can analyse all the logs in aggregate and makes it very obvious when something is _very_ wrong and causing a lot of requests to fail, but is less useful than slog for pinpointing a specific issue. It's worth also noting (since I find often a whole load of things can be mixed into logging) that we do not drive our monitoring off these logs, at least at the moment. We have separate systems for that.
- sandGorgon 10y ago
  this is awesome! how do you do this? a new header with a unique id... generated by something like lua+nginx. but then how do you pass this request from one service to another?
  - amatix 10y ago
    For us, a `X-Request-ID` header is generated by any app if it doesn't receive it from upstream -- but normally nginx or the CDN will generate it. There's a few nginx modules to do it, we use https://github.com/newobj/nginx-x-rid-header https://github.com/newobj/nginx-x-rid-header Most languages/logging frameworks have some sort of per-thread context (eg. Filters in Python, MDC in log4j, etc) to be able to tag log messages with. If you're using postgresql, you can call `SET application_name='{requestID}';` and that can be output as part of logs too.

raesene6 10y ago

Hi, I'd be very interested to hear if you've come across any good documentation on hardening Kubernetes? From what I've seen so far there's very limited documentation on that when compared to other components like Docker, and the defaults can sometimes not be suitable for a high security environment (e.g. https://raesene.github.io/blog/2016/10/08/Kubernetes-From-Container-To-Cluster/ https://raesene.github.io/blog/2016/10/08/Kubernetes-From-Co... )

ben_hall 10y ago
There is a solution to this coming in v1.5 - https://github.com/kubernetes/kubernetes/pull/32518 https://github.com/kubernetes/kubernetes/pull/32518
- raesene6 10y ago
  yeah I've got that noted in the blog post :) but it was really just an example. One of the things thats struck me about docker/Kubernetes etc is that they tend to be tuned for a general use case in terms of security and configuration. Choices that might improve security and restrict usefulness of services are not usually defaults. As such there needs to be a level of hardening done from an out of the box perspective where they're being used in a high-security environment (e.g. banking). For Docker we have resources like docker_bench and the CIS guide which provide a list of possible hardening steps, but I've not managed to find anything like that for Kubernetes, which is why I'm interested in how Monzo are addressing that issue.
  - dimastopel 10y ago
    Not sure if you are looking for a commercial solution, but we (Twistlock [1]) develop a security suite for enterprises working with Docker and / or Kubernetes. In fact, we are officially recommended by Google for working with GKE which is very much based on Kubernetes [2]. I'd be glad to elaborate if relevant. [1] https://www.twistlock.com/ https://www.twistlock.com/ [2] https://cloudplatform.googleblog.com/2015/11/enhancements-to-Container-Engine-and-Container-Registry.html https://cloudplatform.googleblog.com/2015/11/enhancements-to...

nvarsj 10y ago

What kind of resource usage do you see with linkerd? It looks good, but I'm concerned about the resource overhead of a JVM based proxy on every node / pod. edit: Ok answered myself - found https://blog.buoyant.io/2016/06/17/small-memory-jvm-techniques-for-microservice-sidecars/ https://blog.buoyant.io/2016/06/17/small-memory-jvm-techniqu.... So with some work it can be reduced from 500Mi->100Mi per instance. It'd be interesting to see the kind of CPU time it uses under load, though.

tonyhb 10y ago

If K8S secrets management is storing them in plain-text in etcd how do you go about making it actually secure for a bank?

obeattie 10y ago
We don't store things that are actually secret, as k8s secrets ;-) Hashicorp Vault is quite good at this.
- tonyhb 10y ago
  Wahoo! Good to know. Definitely don't trust the security by default :)

justifier 10y ago

this is an excellent talk a link to the video: https://skillsmatter.com/skillscasts/9146-building-a-bank-with-kubernetes https://skillsmatter.com/skillscasts/9146-building-a-bank-wi... thanks for giving and posting it i especially appreciate the effort because a month ago i posted this(o) comment: > can anyone suggest reading to understand how contemporary banks function, where can i get an understanding of a bank or credit union from a software engineer's perspective: dependencies, steps to start, challenges of running, protections from common problems, interesting emerging disruptions; best of luck with monzo, https://monzo.com/ https://monzo.com/ (o) https://news.ycombinator.com/item?id=12567536 https://news.ycombinator.com/item?id=12567536