8 ms·

I hate to rain on the parade, but on the face of it, this sounds utterly ridiculous. I simply cannot believe that their product really calls for an architectur

I hate to rain on the parade, but on the face of it, this sounds utterly ridiculous. I simply cannot believe that their product really calls for an architecture consisting of 1600 individual microservices. Don't get me wrong, I don't think one huge monolith is ideal either, but how is this better? Who at Monzo could possibly understand how a significant portion of their platform works, in detail, and be confident that knowledge isn't going to be outdated in ten minutes from now? Who understands how changes to each of these tiny pieces will impact the others? What are they actually solving by doing things this way? Would this abomination exist if they didn't feel the need to hire people whose primary objective is to get the words "microservice" and "Kubernetes" stamped onto their resumes? Am I just out of touch? Is this the new normal?

conjectures 7y ago
Suppose you design a bank. How many different RPCs would be involved? 1600 sounds modest.
- GrumpyNl 7y ago
  Cant find how many developers they have to maintain this monster.
  - lofties 7y ago
    An article[1] from 2018 mentions 70 engineers. Let's assume 50% growth yoy which puts them at 157 engineers now. Making another assumption that only 20% of these engineers are productive which gives us 31 engineers. That's 1600 microservices for 31 engineers. That sounds large but if you have 1600 microservices you are _most likely_ writing a single service where you'd normally write a class. Does 50 classes per engineer sound like a lot? Sure does but they didn't create this infrastructure out of thin air - this is an accumulation of many years. For sake of the arguments let's say 2 years. Across 24 months that's only 2 microservices per developer per month. Not unthinkable, and this is assuming only 20% of their engineering force is actually productive. [1]https://monzo.com/blog/2018/06/27/engineering-management-at-monzo https://monzo.com/blog/2018/06/27/engineering-management-at-...
    - okal 7y ago
      I think a lot of people are extrapolating from their experience with monoliths and imagining that Monzo devs are maintaining 1600 tiny monoliths. That's an anxiety-inducing thought.
      - Nextgrid 7y ago
        I would prefer maintaining a "tiny monolith" over a micro service. At least with the monolith you're writing value-delivering business-logic code. From my experience with micro services you spend more time on boilerplate and the whole HTTP/RPC part and the communication with other services than the actual task the service is supposed to be doing.
        conjectures 7y ago
        Or, maybe you are spending a bunch of time making sure you don't break a rats' nest of dependencies and waiting for things to build. After the n^th iteration of building an rpc server, such things may be fairly standardised, no?
Fr0styMatt88 7y ago
The orchestration overhead must be huge, but it sounds like they have focused on automating the pain of that away as much as possible. To me it looks like an Alan Kay style of OOP design. From that standpoint, replace 'microservice' with 'class' (or 'actor') and 1600 of them isn't that ridiculous. It'd be an interesting system to browse. I'm actually curious if they've built any tooling around making the codebase easier to navigate and edit. Smalltalk handles a huge amount of tiny classes by having good browsing tools. What would you use for a Go codebase of this size?
- x3al 7y ago
  You can replace 'microservice' with 'class, but completely asynchronous and any interaction with it can fail without caller knowing if it succeed or failed at all'.
  - imtringued 7y ago
    Yes, that type of class is known as "actor".
    - cfontes 7y ago
      As someone with a little experience troubleshooting actor based systems, having 1600 of them sounds (actors not instances) like a nightmare.
      - vbezhenar 7y ago
        Some people love erlang.
      - blackoil 7y ago
        I have some experience with Akka. The amount of praying you have to do during production issues, no surprise it feels like a religion. We tried it about 4 years back. Maybe there are some better debugging tools now, but for us both learning curve and prod issue pain did not make it worth it.
okal 7y ago
Why should someone want/need to understand how the whole platform/significant portion of it works in detail? You just need to understand what happens at the boundaries of the systems you build or interact with. This works beautifully when you need to scale a team. Trying to get everyone to know every system in intimate detail seems the wrong way to go about it (to me). It's a bank. I imagine they automate a lot of things that would be considered minutiae in other organisations. Every Lambda, every little bot is a service. I get your cynicism, and I'd have shared it a few years ago, but 1600 now sounds like a perfectly reasonable number to me.
- vbezhenar 7y ago
  To understand the problem when everyone blames one another, every component seems to work fine but the entire system does not.
pjc50 7y ago
Not to mention that it's a financial service, and doing transactions across microservices is a total nightmare. I wonder how many of those services are allowed to touch actual account data directly.
- okal 7y ago
  There's a lot of back office tech you need to build/buy if you're working in consumer, or financial tech. Monzo is both. Most people in working at a bank aren't finance workers. There are entire domains that will never be touched in the course of a transaction.
- thu2111 7y ago
  They're using Apache Cassandra which only got transactions in v2.0 and they're "lightweight transactions". It's not really designed for very strongly consistent data. I wonder how well that really works for financial OLTP.
ajsnigrutin 7y ago
Also, how many of those microservices do exactly the same thing, but are implemented by different people who don't know about the other microservices existing.
jmchuster 7y ago
I think for traditional webservices it sounds a bit strange, but look at things like actors in erlang, or how ROS (robot operating system) uses nodes. It's all individual processes that are fully asynchronous, and the number can run into the hundreds or thousands.
kitd 7y ago
Who at Monzo could possibly understand how a significant portion of their platform works, in detail, and be confident that knowledge isn't going to be outdated in ten minutes from now? Who understands how changes to each of these tiny pieces will impact the others? Tbf, this can apply to any sufficiently complex system, whether or not it uses microservices. It's why interfaces are so damned important, especially communicating them with stakeholders. Done properly, you only need to focus on the bits of a system that matter to you. At least microservices get to use OpenAPI.
- discreteevent 7y ago
  There's a major difference between 600 components that follow the fallacies of network computing and 600 that don't.
  - kitd 7y ago
    Agree completely, but that wasn't the point I was replying to, which was about totally understanding your system. Don't worry. I've suffered (and still do) as much as anyone working with microservices, trying to reconcile and decipher logs. But total knowledge is rarely possible in a large system. At least microservices acknowledges this. If anything, they can help by following Conway's Law.
zorbash 7y ago
Sometimes I get paranoid and assume that startups of this kind get subsidised by the state to lower the unemployment. It looks like they had money to burn on engineers and the approach was to give them tasks to keep them busy so that they don't appear idle and give them an easy language to work with so that they don't complain.
chrischen 7y ago
If you look at what Apollo Federation (graphql) is proposing, then you can see the architecture calls for essentially turning each database table (users, orders, etc) into a micro service running with its own port.
Terretta 7y ago
Maybe a little out of touch. Not about the new normal, but about being a consumer bank. We are not micro services, we are a 'monolith', but with over 1600 components our developers call 'applications' and over 7,500 touch-points between them our developers call 'interfaces'. A single 'customer information file' touches dozens and dozens of these 'applications'. For us, at least, what's going on is regulation. Every aspect of banking is heavily regulated, and each component of the monolith has to be compliant. A notorious example is check settlement order. When checks come in for payment, the bank is supposed to process those checks in an order favorable to the bank customer not to the bank. For example, $1000 in an account, and you write one $995 check and a hundred $10 checks, if the bank pays the first one, the rest would bounce, causing $3500 in bounced check fees. If the bank pays all the $10 ones, just the $995 one would bounce, causing $35 in check fees. This is now regulated. There are countless such rules, and no dev team can know them all. So you assemble third party component software that has the rules baked in, or if trying to greenfield, you have compliance teams that study all the rules for their areas and bake them into the software requirements -- and good luck, because in the USA, these rules can be different state by state or even zip code by zip code. Easier to just decompose your monolith into components maintained by a group that contractually guarantees regulatory compliance for their specialization, and then pray (or rather, spend months regression testing after any change for quality assurance) the whole thing works when you glom it together. In that world, suddenly something like K8s makes a lot of sense.

conjectures 7y ago

Suppose you design a bank. How many different RPCs would be involved? 1600 sounds modest.

GrumpyNl 7y ago
Cant find how many developers they have to maintain this monster.
- lofties 7y ago
  An article[1] from 2018 mentions 70 engineers. Let's assume 50% growth yoy which puts them at 157 engineers now. Making another assumption that only 20% of these engineers are productive which gives us 31 engineers. That's 1600 microservices for 31 engineers. That sounds large but if you have 1600 microservices you are _most likely_ writing a single service where you'd normally write a class. Does 50 classes per engineer sound like a lot? Sure does but they didn't create this infrastructure out of thin air - this is an accumulation of many years. For sake of the arguments let's say 2 years. Across 24 months that's only 2 microservices per developer per month. Not unthinkable, and this is assuming only 20% of their engineering force is actually productive. [1]https://monzo.com/blog/2018/06/27/engineering-management-at-monzo https://monzo.com/blog/2018/06/27/engineering-management-at-...
  - okal 7y ago
    I think a lot of people are extrapolating from their experience with monoliths and imagining that Monzo devs are maintaining 1600 tiny monoliths. That's an anxiety-inducing thought.
    - Nextgrid 7y ago
      I would prefer maintaining a "tiny monolith" over a micro service. At least with the monolith you're writing value-delivering business-logic code. From my experience with micro services you spend more time on boilerplate and the whole HTTP/RPC part and the communication with other services than the actual task the service is supposed to be doing.
      - conjectures 7y ago
        Or, maybe you are spending a bunch of time making sure you don't break a rats' nest of dependencies and waiting for things to build. After the n^th iteration of building an rpc server, such things may be fairly standardised, no?

Fr0styMatt88 7y ago

The orchestration overhead must be huge, but it sounds like they have focused on automating the pain of that away as much as possible. To me it looks like an Alan Kay style of OOP design. From that standpoint, replace 'microservice' with 'class' (or 'actor') and 1600 of them isn't that ridiculous. It'd be an interesting system to browse. I'm actually curious if they've built any tooling around making the codebase easier to navigate and edit. Smalltalk handles a huge amount of tiny classes by having good browsing tools. What would you use for a Go codebase of this size?

x3al 7y ago
You can replace 'microservice' with 'class, but completely asynchronous and any interaction with it can fail without caller knowing if it succeed or failed at all'.
- imtringued 7y ago
  Yes, that type of class is known as "actor".
  - cfontes 7y ago
    As someone with a little experience troubleshooting actor based systems, having 1600 of them sounds (actors not instances) like a nightmare.
    - vbezhenar 7y ago
      Some people love erlang.
    - blackoil 7y ago
      I have some experience with Akka. The amount of praying you have to do during production issues, no surprise it feels like a religion. We tried it about 4 years back. Maybe there are some better debugging tools now, but for us both learning curve and prod issue pain did not make it worth it.

okal 7y ago

Why should someone want/need to understand how the whole platform/significant portion of it works in detail? You just need to understand what happens at the boundaries of the systems you build or interact with. This works beautifully when you need to scale a team. Trying to get everyone to know every system in intimate detail seems the wrong way to go about it (to me). It's a bank. I imagine they automate a lot of things that would be considered minutiae in other organisations. Every Lambda, every little bot is a service. I get your cynicism, and I'd have shared it a few years ago, but 1600 now sounds like a perfectly reasonable number to me.

vbezhenar 7y ago
To understand the problem when everyone blames one another, every component seems to work fine but the entire system does not.

pjc50 7y ago

Not to mention that it's a financial service, and doing transactions across microservices is a total nightmare. I wonder how many of those services are allowed to touch actual account data directly.

okal 7y ago
There's a lot of back office tech you need to build/buy if you're working in consumer, or financial tech. Monzo is both. Most people in working at a bank aren't finance workers. There are entire domains that will never be touched in the course of a transaction.
thu2111 7y ago
They're using Apache Cassandra which only got transactions in v2.0 and they're "lightweight transactions". It's not really designed for very strongly consistent data. I wonder how well that really works for financial OLTP.

ajsnigrutin 7y ago

Also, how many of those microservices do exactly the same thing, but are implemented by different people who don't know about the other microservices existing.

jmchuster 7y ago

I think for traditional webservices it sounds a bit strange, but look at things like actors in erlang, or how ROS (robot operating system) uses nodes. It's all individual processes that are fully asynchronous, and the number can run into the hundreds or thousands.

kitd 7y ago

Who at Monzo could possibly understand how a significant portion of their platform works, in detail, and be confident that knowledge isn't going to be outdated in ten minutes from now? Who understands how changes to each of these tiny pieces will impact the others? Tbf, this can apply to any sufficiently complex system, whether or not it uses microservices. It's why interfaces are so damned important, especially communicating them with stakeholders. Done properly, you only need to focus on the bits of a system that matter to you. At least microservices get to use OpenAPI.

discreteevent 7y ago
There's a major difference between 600 components that follow the fallacies of network computing and 600 that don't.
- kitd 7y ago
  Agree completely, but that wasn't the point I was replying to, which was about totally understanding your system. Don't worry. I've suffered (and still do) as much as anyone working with microservices, trying to reconcile and decipher logs. But total knowledge is rarely possible in a large system. At least microservices acknowledges this. If anything, they can help by following Conway's Law.

zorbash 7y ago

Sometimes I get paranoid and assume that startups of this kind get subsidised by the state to lower the unemployment. It looks like they had money to burn on engineers and the approach was to give them tasks to keep them busy so that they don't appear idle and give them an easy language to work with so that they don't complain.

chrischen 7y ago

If you look at what Apollo Federation (graphql) is proposing, then you can see the architecture calls for essentially turning each database table (users, orders, etc) into a micro service running with its own port.

Terretta 7y ago

Maybe a little out of touch. Not about the new normal, but about being a consumer bank. We are not micro services, we are a 'monolith', but with over 1600 components our developers call 'applications' and over 7,500 touch-points between them our developers call 'interfaces'. A single 'customer information file' touches dozens and dozens of these 'applications'. For us, at least, what's going on is regulation. Every aspect of banking is heavily regulated, and each component of the monolith has to be compliant. A notorious example is check settlement order. When checks come in for payment, the bank is supposed to process those checks in an order favorable to the bank customer not to the bank. For example, $1000 in an account, and you write one $995 check and a hundred $10 checks, if the bank pays the first one, the rest would bounce, causing $3500 in bounced check fees. If the bank pays all the $10 ones, just the $995 one would bounce, causing $35 in check fees. This is now regulated. There are countless such rules, and no dev team can know them all. So you assemble third party component software that has the rules baked in, or if trying to greenfield, you have compliance teams that study all the rules for their areas and bake them into the software requirements -- and good luck, because in the USA, these rules can be different state by state or even zip code by zip code. Easier to just decompose your monolith into components maintained by a group that contractually guarantees regulatory compliance for their specialization, and then pray (or rather, spend months regression testing after any change for quality assurance) the whole thing works when you glom it together. In that world, suddenly something like K8s makes a lot of sense.