4 ms·
I have spent a lot of time thinking about this problem. I would gladly pay for a product that could make a real, honest to god preview environment that you coul
by echelon 4y ago
I have spent a lot of time thinking about this problem. I would gladly pay for a product that could make a real, honest to god preview environment that you could spin up on-demand and populate it with all of the data you need.
This isn't it, and I think it's pretty much an impossible problem to solve without an entirely new engineering culture around testing.
How do you duplicate the behavior of the 10 direct data dependencies of your app and the constellation of transitive microservice dependencies? Wingman, Galactus, etc. At a company with 1000+ microservices?
And how do you populate it with actual data so you can log in and perform actions? Every single microservice probably needs test data, and it would take a lot of engineering forethought to populate the entire graph.
It gets gnarly. I've been on teams at the center of it all - user accounts and login, representation of the core business entities that the rest of the company is built upon. We couldn't even solve the problem for ourselves. There were so many permutations of account states, creating a dedicated API to make test data deviates from a first class API. So how do you coordinate with downstream services to skip IDV (matching SSN), compromised password checks, etc.
Perhaps you only bring online a subset of services and fake out the rest. At what level can you fake stuff out? And how do you ensure that the test data works with whatever subset you don't fake? If a single service responds in an unexpected way, it could break or corrupt the state you're relying upon to test.
Maybe "preview" works for frontend. Backend, not so much. This is such a massive problem, and I'm only scratching the surface with my description of the kinds of issues you have to solve.
- phendrenad2 4y agoMicroservices became this mess because people only think about production. Nobody builds microservices as standalone units that should be testable and runnable in isolation, with ZERO external data. Instead, everything relies on everything, so it's just a monolith split up into microrepos. Worse, people have started putting app configuration in S3 and other dumb things (I've seen it...) do you cannot run things locally, or even match production unless you have AWS creds. Madness.
- TimTheTinker 4y agoMicroservices should indeed be able to run in isolation, but that doesn't remove the need for end-to-end and integration testing. At a minimum, the product owner should be able to try out the new feature in the UI without having to deal with a staging environment.
- gedy 4y ago> Nobody builds microservices as standalone units that should be testable and runnable in isolation, with ZERO external data. Instead, everything relies on everything, so it's just a monolith split up into microrepos. Yeah this is the surprising thing to me - I've worked with otherwise bright people but whenever we've introduced services/microservices, it seems the default reaction is to just make a "distributed monolith" with circular dependencies and tight coupling. I'm only half joking that 90% of my "architect" role is just point out we shouldn't have circular dependencies.
- codethief 4y ago> Instead, everything relies on everything, so it's just a monolith split up into microrepos …and with added network calls! > Microservices. grug wonder why big brain take hardest problem, factoring system correctly, and introduce network call too. seem very confusing to grug (Source: https://grugbrain.dev/ https://grugbrain.dev/)
- robertlagrant 4y agoThe way I originally did it at my previous company was to have each service have a "dev" Flask blueprint that got registered on non-production environments, that had an endpoint that could populate the database with sample data, and a coordinating service to populate them in the right order (so IDs matched up). Worked pretty well.
- jeremy_k 4y agoSounds like a large dependency graph and multiple variants of spinning up the environments. I'll try to simplify what I mean by that but something akin to having Services A, B, and C. Each of A, B, and C need their own definitions to define how they need to be spun up as well as a definition on how to populate their data. That data population, say through RDS, needs to be spin up RDS instances and make them ready and available whenever someone creates a corresponding ephemeral version of a service. Toss the database afterwards. In the above case, assuming you needed a full graph, you would spin up the three services as well as attach each service to its corresponding database. This approach works well when the number of services remains small. Once that number starts growing, it may not be cost effective to spin everything up. Say service N is needed, but you really only need to read from it. In this case, you might have a dedicated staging environment for N and create a dependency graph that says "in my ephemeral environment, create new instances of A, B, and C with their databases, but also allow network requests to make their way to N." That might be over simplifying, but I've been working in this space for the past few years and have been thinking about this exact issue recently
- howinteresting 4y agoWhile the problem is still hard, it is definitely easier to solve with fewer, larger services. If your backend consists of 3-5 services, the problem is far more tractable than if it consists of 1000+ services.
- hangonhn 4y agoYeah that's a really good point. Our microservices are like 30? 1000+ is intractable but up to 50 or even a 100 it's doable. We rely pretty heavily on AWS Cloudformation to deploy. All our services have Cloudformation templates. I'm sure Terraform would work too or be even better. Since every service knows how to deploy itself, building a preview environment wasn't too difficult. It's just a matter of stitching them all together, which was made a lot easier with Cloudformation nesting of stacks.
- hangonhn 4y agoThat's a really good point. I think what made it quite easy for us is that we maintain a staging environment. Staging is constantly tested by QA simulating our workflows so non-production data is being constantly generated. When we spin up our preview environments, we use snapshots from the staging systems to seed the preview environments. Your point on data though really gets to the crux of the problem. Deploying the services is trivial these days with k8s, Cloud Formation, Terraform, Docker, etc. but the data part is hard.
- jcraft 4y ago> but the data part is hard. I'd love to learn more about the challenges you have with data. My company essentially provides data-anonymization-as-a-service and we integrate with ephemeral environments, but we could probably improve the integrations.
- hangonhn 4y agoGot a link to your company's service? Would love to learn more. We work in healthcare so data-anonymization is essential. If we can take prod data and erase all the PII, it would be AMAZING.
- codethief 4y ago> How do you duplicate the behavior of the 10 direct data dependencies of your app and the constellation of transitive microservice dependencies? Wingman, Galactus, etc. For those, like me, who had to think for a moment where they had heard that reference: https://www.youtube.com/watch?v=y8OnoxKotPQ https://www.youtube.com/watch?v=y8OnoxKotPQ