12 ms·
We built network isolation for 1500 services
- brentis 7y agoNice work. If you define your policies based on a tagging taxonomy you could centrally manage these inbound/outbound service relationships. Every new instance or container would assume same network policies based on tag.
- d4nt 7y agoLocking down this network of services is a massive security improvement and they've used some very neat ways of achieving it. Overall, I really appreciate them writing this up. However, 1500 services? That really feels like they're separating things at too granular a level. Does every one of those things really need to sit behind a network call? Couldn't some of that re-use be via code libraries? I wonder what the service to developer ratio is?
- segmondy 7y agoCode libraries in microservices can lead to a distributed monolith. each service needs to be be able to evolve independently. if you end up with a library where an update will require more than one service to be updated at the same time, then you are losing the advantage of microservices. nevertheless 1500 is a lot, but if you think it's a lot, wait till someone builds out their entire business with lambda/serveless. :-)
- p10jkle 7y agoYeah, our (mostly) ban on shared code is a major reason we end up with a lot of services; there's even service.business-day which tells you if a day is a working day
- d4nt 7y agoI’m interested in the term “distributed monolith” and why shared libraries are bad, can you elaborate? (I’m an experienced dev who’s never worked on micro services). Say a service that consumed 6 other services became a service that consumed 3 other services and 3 libraries. The interfaces were the same, the libraries are consumed via package management and have good test coverage. Why is that worse? I’m actually not surprised you have a service for handling working days. The list of UK bank holidays should ideally be stored in one place, with everything else consuming that. But not all of the 1500 services have their own data stores, do they?
- p10jkle 7y agoThe common idea is that if you have to regularly roll out all your services, then you still basically have a monolith. So if you have a lot of shared code in libaries, there are no benefits to microservices. In our case, maybe we do have core libaries like to read from Cassandra, but you don't necessarily have to roll out everything if you improve them. Some pods could be even a year old All the services have their own keyspace in Cassandra, there is near-zero cross service data access
- virgilp 7y agoThat's not what makes a distributed monolith, IMO. Or to put it differently, it's not what makes a "monolith" be equal to "bad" in people minds. You want to avoid the badness, don't want to avoid the label. So, what is bad about a monolith? The complexity. Coupling of unrelated things; because it's simple to do so, because it's right there next to you and you know how it's implemented and what it performs, so you do thing A in component X knowing/relying on the fact that component Y will do the thing B, because that's how it's implemented (not because it's inherent unavoidable business logic). How do you end up with a distributed monolith? E.g. assume that microservice B will call microservice C only after you (i.e. microservice A) had a chance to call C first. Yes your code is distributed, but you don't get the benefit of logic segregation between the modules - you just get all the downsides of distributed systems. Do note that my scenario requires no shared code or library. Not sharing code doesn't save you from the distributed monolith - careful thought does.
- viraptor 7y agoHow do you deal with passing common data between services? For example, does every service checking the holiday endpoint need to write its own access/deserialisation/mapping starting from http or some other RPC layer? If yes, how do you keep that in sync/compatible? Do you have an explosion of versions on each endpoint? This is likely trivial for the holidays endpoint, but I'm interested in the more complex ones.
- lwb 7y agoThis is what I want to know too -- seems like you have to have at least one common library, that is, the library that lets you write a service call in one line instead of twenty lines. IME Go can be quite verbose, especially with >3 lines of error checking after every function call.
- lossolo 7y agoThey are probably using grpc and automatically generate clients libraries.
- ealexhudson 7y agoWhy (mostly) ban shared code? If you have that many microservices the deployment has to be well automated; so rolling out updated services on library update can't be difficult, and the interface to a compiled library is about as controllable as the interface to a network service. Is there a separate reason to those? Do you worry about the additional latency of additional service calls (for some trivial functions that doesn't feel terribly worthwhile)?
- cameronbrown 7y agoI can imagine these shared libraries being in an inconsistent state at one time or the risks of updating thousands of services at once, would be two factors.
- marcinzm 7y agoI'm curious why the investment in so many services instead of investing in improved CI/CD/build tools. For example, automatically deploying services when the libraries they depend on are updated. Monitoring to automatically check if a deployment is having issues. I mean, due to security concerns you still will need to re-deploy services every so often so having great CI/CD seems a more worthwhile investment. edit: Also, for the same reason chaos engineering, short lived certificates and periodic backup tests make sense; constantly testing out the deployment of all services makes sense to me. If you do something like deploying a service rarely then the chance of something going wrong (forgot a step, radical external changes, etc.) is decently higher.
- random3 7y ago> wait till someone builds out their entire business with lambda/serveless. Perhaps you could built in a "regular" fashion and just compile it like that? You could write normal code and let the compiler do the massive work to handle all that and compile everything into somethign that distributes over the network.
- dsirola 7y agoIn my opinion, this is the point at which FaaS becomes viable
- mattacular 7y agoInteresting idea. Did you mean to imply that such a compiler exists or just that it's theoretically possible?
- random3 7y agoIt's 1) theoretically possible in generalized fashion (although the language could make things easier) 2) it does exist in loose sense if you consider actor models (like someone else was mentioning)
- lkschubert8 7y agoIf I'm not mistaken that's how most of the actor model frameworks are supposed to work. An example is Akka. The developer writes code communicating between actors using an interface that abstracts how they are actually communicating.
- akie 7y agoThe author wrote (somewhere) on this Twitter thread [1] that they have 150 developers, so that's 10 services for every employee. [1] https://twitter.com/JackKleeman/status/1190354757308862468 https://twitter.com/JackKleeman/status/1190354757308862468
- tonyedgecombe 7y agoI bet 10% of their developers wrote 90% of the services.
- p10jkle 7y agoNot really
- all_usernames 7y agoGreat post. I really appreciate engineering blogs written in this storytime format. I don't have time to dive into the implementation of Calico or <insert one of the 1,261 kubernetes projects here>, but I learn a lot from reading the process a team goes through in figuring out and iterating on a solution. https://landscape.cncf.io/ https://landscape.cncf.io/
- angry_octet 7y agoImpressive achievement. It still sounds like callee's have more knowledge of callers than is justified. Is it a security property or a component functionality property? How do those interact? A centralised graph representation of the security/functionality properties would be a better way to represent this information, so it can catch adding interfaces which should be forbidden. Also able to be configuration managed as sets of microservices. If you have a connectivity graph it would be good to do taint analysis to see how far bad information can propagate.
- p10jkle 7y agoI think we do effectively have a graph, but its not stored in a centralised way. If it was, then we'd have to somehow gatekeep that state, which when you consider that we do hundreds of deployments a day, becomes tricky
- angry_octet 7y agoThen the state at present is changing hundreds of times a day, without being centrally visible, via runtime query or configuration service. Seems like it would be hard to retrospectively analyze what could connect to what. Maybe consider a permissive system, that allows a service code change to change the connectivity graph without a review step, but centrally logs the change; another service can supervise what's happening, and alert if strange things happen. It would help to have a system for categorising service types; the graph properties of the system might be useful for understanding that. Looking at service changes in terms of graph properties, like edit distance, can also be useful.
- p10jkle 7y agoI feel like we could do all of that by consuming the GitHub API. We essentially have directed graph stored as files
- kasey_junk 7y agoUsing YAML for critical infrastructure specification is one of the stupidest things we’ve ever done as an industry.
- pokoleo 7y agoIt’s better than XML!
- angry_octet 7y agoWhen you see how many parsing ambiguity problems there are with YAML you begin to wish for XML. Other modern alternatives include: https://jsonnet.org/ https://jsonnet.org/ https://json5.org/ https://json5.org/ See https://arp242.net/yaml-config.html https://arp242.net/yaml-config.html
- mschuster91 7y agoXML is verbose as hell but: - it's unambiguous - with a proper XSD schema you can leverage validation already during parsing, which means you don't even have to boot your application at all - simply load your configuration in any XML parser and you'll automatically catch errors.
- elsonrodriguez 7y agoWhat are the alternatives? The minute you get into using code for these things, cleverness begins to turn configuration repos into something you have to debug before editing.
- james_s_tayler 7y agoInstead you have YAML you have to debug because all of a sudden you have decided to support Norway as a country and something about Norway's country code and the way YAML handles Boolean values bites you in the ass.
- unwind 7y ago
- hu3 7y ago> This would read all the Go code in our platform, and attempt to find code that looked like it was making a request to another service. Is there a link about how much Go does Monzo they use?
- Spinosaurus 7y agoAlmost all of the micro services referenced are written in Go.
- rawoke083600 7y ago"But we already have over 1,500" wow... I would start there...
- sansnomme 7y agoAnother potential solution is to use a constraint solver like MSFT Z3, or if you want a nicer syntax and more flexibility, Prolog. E.g. https://medium.com/@ahelwer/checking-firewall-equivalence-with-z3-c2efe5051c8f https://medium.com/@ahelwer/checking-firewall-equivalence-wi... This is much more scalable in the long run.
- gravypod 7y agoIf the authors are reading this I was wondering two things: 1. Why was static analysis of the code chosen over observing the system during runtime and integration testing? 2. What was the reason rhe CNI layer was chosen for the implementation of this over the service mesh layer? Something that really interests me about bazel/buck/pants/please is it automates #1 entirely with dep queries.
- p10jkle 7y ago1. We have a lot of code paths, there isn't an integration test for everything. And for runtime, just because something is rarely called doesn't mean it never called; a bank can have yearly processes 2. We'll do service mesh too but this was probably an easier first step. We have rolled our own mesh with envoy sidecars and moving it towards istio style behaviour isn't trivial
- tjungblut 7y ago> We have a lot of code paths, there isn't an integration test for everything. but you will have the service names that you're contacting configured somewhere (or in the worst case hardcoded). I would think this is much easier to analyze than the traffic flows.
- YawningAngel 7y agoIt's done by importing a protobuf, so I imagine rpcmap just checks all the protos imported by a service
- p10jkle 7y agoIt's slightly more complicated than that because you can import protos to use a constant or to consume an event from another service. But that is the gist
- p10jkle 7y agoThat's what we're doing, basically
- grandinj 7y agoStrikes me that some services ideally need to expose multiple interfaces, and that isolation should be on a per-service-interface basis. E.g. the monitoring service should only be able to access the metrics part of each service.
- p10jkle 7y agoMonitoring can only access metrics, it's scoped to port. We'd like endpoint level controls and that's the next step
- mschuster91 7y ago1.500 services? What the... the run times for calls must be atrocious with all the network communication and latency that is happening.
- jacquesm 7y agoIt does not have to be. But that is the least of your worries, even so there might be valid reasons for setting things up this way but it would be a relatively rare case. They do have 150 devs and that alone is a difference that makes this approach untenable for most companies.
- p10jkle 7y agoNetworks are fast https://twitter.com/obeattie/status/1190676549395025921 https://twitter.com/obeattie/status/1190676549395025921
- majky538 7y agoPico-service architecture as the trend for 2020. And in the evening, your wife will say, oh, dear, you are damn good architecture with many.. many services but my ex could deploy me with one several times a day much better.
- z3t4 7y agoApplying network filtering, while being a nice extra layer, it should not be the only layer. Services should need authorization like if it was an open api.
- p10jkle 7y agoCompletely agree
- matdehaast 7y agoCurious if you looked at using oAuth with client credentials grant for each service? Also didn't see any mention of prior art like https://cloud.google.com/beyondcorp/ https://cloud.google.com/beyondcorp/. Thanks for the great writeup!
- p10jkle 7y agoThis is probably our next project, although we might solve this problem with mutual tls instead of JWTs
- matdehaast 7y agooAuth generally won't give you JWT's. Its recommended it gives you an opaque token which you introspect. Whats nice is you delegate the Auth part to your AS. And then your resource servers (micro-services) can make the determinations based on the client_id/claims from introspection.
- voltarolin 7y agoCan a service mesh such as Istio provide the capability that Monzo have implemented themselves here?
- aSplash0fDerp 7y agoNice write-up! Thats the beauty of scale, explain a part in detail, then go with the 30,000 foot view. IMM, the security orchestration may actually become the "app" as speeds continue to increase, compute costs go even lower and losses incurred from compromised data/networks increase. A true zero trust platform that keeps all of the doors closed or "instances/vm" offline until (the milliseconds) they're needed is the security symphony we might see on the horizon. Data silos and walled gardens may never go out of style, they'll just take on new acronyms.
- purple_ducks 7y ago> attempt to find code that looked like it was making a request to another service. > We generally fixed those cases by adding a special comment in the code that told rpcmap about the link Why not enforce all endpoints/urls be defined in a config file and sidestep this? - scanning code for URLs/constructed URL is overkill and brittle.