5 ms·

I wonder what setting up a local dev instance is like for anything involving more than one or two of those.

by davidw 3y ago

threeseed 3y ago
I've worked on a couple of extremely large micro services projects. And the thing is that nobody ever needs to run the entire stack other than end to end tests which get run in the cloud. You just checkout the services you need and because they are designed to be isolated the dependencies will usually be automatically stubbed out. So it's just a matter of running them or chaining them together if you have a particular scenario to test.
- plonk 3y ago
  Is there typically a “cold restart” plan to rebuild the whole infra from scratch? I’m thinking of things like circular dependencies when services boot.
  - nurple 3y ago
    I've found that the eventual consistency provided by orchestrators like k8s solve this problem rather well. If the services are written using the right paradigms to handle such a situation, they will also be much more resilient to platform disruptions. I like to view k8s a lot like erlang's OTP, if something isn't right with the state of a service, I advocate calling 'exit()' and letting the restart with exponential backoff handle the transient.
- cma 3y ago
  Nobody needs to if they can't possibly.
- x86x87 3y ago
  Question for you: how does performance measurement and optimization work in that environment? Is the key some sort of meta tooling that understands relationships between microservices? How would you express such relationships in the first place? Q2: How do you ensure the stubbed deps behave like the real thing? Q3: how do you handle logging and metrics in an unified way across the stack? And related to this: how do you ever get to upgrade services crosscutting concerns that ideally are not invented in every service?
  - joshmanders 3y ago
    Not OP, but I worked on a large microservices based system at a leading financial institute and when we needed to work on a single service, we had docker compose files that pulled the images needed for the dependency services of that service for us to develop what we needed. They all just ran in our local docker. If we wanted we could have a massive compose file with all services in it, but typically we only needed the iam service and a few other small ones depending on what services we were working on.
  - eddtries 3y ago
    Q2 - techniques like contract testing help here, beyond simple stubs. Also, mocked services maintained by the services original devs you can work against help
  - oooyay 3y ago
    > how does performance measurement and optimization work in that environment? SRE here. Generally speaking, each API or each service will have a contract that it must adhere to depending on upstream and downstream relationships and their fail safes. Each service (or API) will then load test in isolation. After that, if you want to be really sure about regressions (which would include fail safes) you load test the whole thing put together. > Is the key some sort of meta tooling that understands relationships between microservices? This is quite hard to do when you have a lot of transactions. I don't think there's commodity software that does this because you'd need to configure that software to map on keys, then map those keys to services. Generally, the easiest way is to get engineering teams to declare upstreams and downstreams. > Q2: How do you ensure the stubbed deps behave like the real thing? Generally, generation. Something like protobuf or Open API generation will do. > Q3: how do you handle logging and metrics in an unified way across the stack? You issue high level standards like, "We'll use JSON logging with UTC time formatting". At the end of the day logging is very contextual and in a service ownership model the service owners are usually the ones reading and alerting on their logs. > And related to this: how do you ever get to upgrade services crosscutting concerns that ideally are not invented in every service? Shared dependencies. I'm not actually sure what's a cross cutting concern; generally services that are this small should be designed to operate mostly independently. They're small, but "microservices" tend to have a lot of fail safes built in. If you're referring to how do we not write 4000 config loaders then there's usually a team that builds a very generic config loader and everyone or a majority use it.
jameshart 3y ago
In general in a micro service environment, you try to build things so that 1) you don't need to run other things locally, and 2) if you did need to, the services are just containers so it's pretty easy to run one. But you tend to try to write your service so that it treats everything else it depends on like a vendor-provided API. Like, if you were building a Slack bot, you wouldn't ask Slack to let you pull down and run a local copy of Slack's API to test against. You'd maybe set up a test account in Slack's production system, and run your local bot against that to test it before you deploy it with credentials to run against your real slack account. In a microservice architecture, you integrate with other internal systems in the same way.
- magicalhippo 3y ago
  We're not big on microservices, but we do integrate with a lot of other systems. I find the opaqueness of other services to reduce development speed quite drastically. With local code I can view both sides of the fence and easily see if I'm using it wrong or if it's a bug in my colleagues code. Seems that if you're constantly developing against opaque services you'd end up in the same quagmire quite quickly?
  - scarface_74 3y ago
    You shouldn’t care about anything outside of published contracts for dependencies. You’re slwags dependent on underlying APIs
    - magicalhippo 3y ago
      Of course I shouldn't. But when things don't work as I expect, it's far more efficient to be able to view the code on both sides, rather than only on my own side and try to guess what the other side is doing. Besides the usual suspect of wrong understanding on my end leading to misuse, this can also be due to lacking or wrong documentation of the other system, or bugs in the other system due to unexpected inputs or similar. Like just a few days ago we spent an unreasonable amount of time with an API of one of our customers, where we would get empty list back for some of our queries. Turned out something in their service crashed when handed national characters, despite accepting JSON and hence UTF-8 input and nothing in the documentation about English letters only. Rather than returning 400 or 500, the service returned 200 with an empty list, leading us to assume we did something wrong.
fizx 3y ago
Something like https://tilt.dev/ https://tilt.dev/ where you spin up a subset of the service graph in a cloud environment that hot-reloads based on local edits.
aeyes 3y ago
Looks like they are shifting away from local development: https://www.uber.com/en-CL/blog/devpod-improving-developer-productivity-at-uber/ https://www.uber.com/en-CL/blog/devpod-improving-developer-p...
maronato 3y ago
When you want to run a local version of a service, you start it locally as normal and configure either local stubs of dependencies and dependents, or you can hook up the actual “production” services to your local service process. If doing the latter, you can create special user accounts and events that change how the networking routing happens. Events from those fake users pass through regular production apps up until the service you’re testing, then they are instead routed to your local version, and you can continue the calls to other prod services. Uber employees’ apps are special and allow us to log in as these fake users and create fake rides or deliveries, and then we can look at the traces and logs to debug and stuff.

I wonder what setting up a local dev instance is like for anything involving more than one or two of those.

by davidw 3y ago

I wonder what setting up a local dev instance is like for anything involving more than one or two of those.

I've worked on a couple of extremely large micro services projects. And the thing is that nobody ever needs to run the entire stack other than end to end tests which get run in the cloud. You just checkout the services you need and because they are designed to be isolated the dependencies will usually be automatically stubbed out. So it's just a matter of running them or chaining them together if you have a particular scenario to test.

plonk 3y ago
Is there typically a “cold restart” plan to rebuild the whole infra from scratch? I’m thinking of things like circular dependencies when services boot.
- nurple 3y ago
  I've found that the eventual consistency provided by orchestrators like k8s solve this problem rather well. If the services are written using the right paradigms to handle such a situation, they will also be much more resilient to platform disruptions. I like to view k8s a lot like erlang's OTP, if something isn't right with the state of a service, I advocate calling 'exit()' and letting the restart with exponential backoff handle the transient.
cma 3y ago
Nobody needs to if they can't possibly.
x86x87 3y ago
Question for you: how does performance measurement and optimization work in that environment? Is the key some sort of meta tooling that understands relationships between microservices? How would you express such relationships in the first place? Q2: How do you ensure the stubbed deps behave like the real thing? Q3: how do you handle logging and metrics in an unified way across the stack? And related to this: how do you ever get to upgrade services crosscutting concerns that ideally are not invented in every service?
- joshmanders 3y ago
  Not OP, but I worked on a large microservices based system at a leading financial institute and when we needed to work on a single service, we had docker compose files that pulled the images needed for the dependency services of that service for us to develop what we needed. They all just ran in our local docker. If we wanted we could have a massive compose file with all services in it, but typically we only needed the iam service and a few other small ones depending on what services we were working on.
- eddtries 3y ago
  Q2 - techniques like contract testing help here, beyond simple stubs. Also, mocked services maintained by the services original devs you can work against help
- oooyay 3y ago
  > how does performance measurement and optimization work in that environment? SRE here. Generally speaking, each API or each service will have a contract that it must adhere to depending on upstream and downstream relationships and their fail safes. Each service (or API) will then load test in isolation. After that, if you want to be really sure about regressions (which would include fail safes) you load test the whole thing put together. > Is the key some sort of meta tooling that understands relationships between microservices? This is quite hard to do when you have a lot of transactions. I don't think there's commodity software that does this because you'd need to configure that software to map on keys, then map those keys to services. Generally, the easiest way is to get engineering teams to declare upstreams and downstreams. > Q2: How do you ensure the stubbed deps behave like the real thing? Generally, generation. Something like protobuf or Open API generation will do. > Q3: how do you handle logging and metrics in an unified way across the stack? You issue high level standards like, "We'll use JSON logging with UTC time formatting". At the end of the day logging is very contextual and in a service ownership model the service owners are usually the ones reading and alerting on their logs. > And related to this: how do you ever get to upgrade services crosscutting concerns that ideally are not invented in every service? Shared dependencies. I'm not actually sure what's a cross cutting concern; generally services that are this small should be designed to operate mostly independently. They're small, but "microservices" tend to have a lot of fail safes built in. If you're referring to how do we not write 4000 config loaders then there's usually a team that builds a very generic config loader and everyone or a majority use it.

jameshart 3y ago

In general in a micro service environment, you try to build things so that 1) you don't need to run other things locally, and 2) if you did need to, the services are just containers so it's pretty easy to run one. But you tend to try to write your service so that it treats everything else it depends on like a vendor-provided API. Like, if you were building a Slack bot, you wouldn't ask Slack to let you pull down and run a local copy of Slack's API to test against. You'd maybe set up a test account in Slack's production system, and run your local bot against that to test it before you deploy it with credentials to run against your real slack account. In a microservice architecture, you integrate with other internal systems in the same way.

magicalhippo 3y ago
We're not big on microservices, but we do integrate with a lot of other systems. I find the opaqueness of other services to reduce development speed quite drastically. With local code I can view both sides of the fence and easily see if I'm using it wrong or if it's a bug in my colleagues code. Seems that if you're constantly developing against opaque services you'd end up in the same quagmire quite quickly?
- scarface_74 3y ago
  You shouldn’t care about anything outside of published contracts for dependencies. You’re slwags dependent on underlying APIs
  - magicalhippo 3y ago
    Of course I shouldn't. But when things don't work as I expect, it's far more efficient to be able to view the code on both sides, rather than only on my own side and try to guess what the other side is doing. Besides the usual suspect of wrong understanding on my end leading to misuse, this can also be due to lacking or wrong documentation of the other system, or bugs in the other system due to unexpected inputs or similar. Like just a few days ago we spent an unreasonable amount of time with an API of one of our customers, where we would get empty list back for some of our queries. Turned out something in their service crashed when handed national characters, despite accepting JSON and hence UTF-8 input and nothing in the documentation about English letters only. Rather than returning 400 or 500, the service returned 200 with an empty list, leading us to assume we did something wrong.

fizx 3y ago

Something like https://tilt.dev/ https://tilt.dev/ where you spin up a subset of the service graph in a cloud environment that hot-reloads based on local edits.

aeyes 3y ago

Looks like they are shifting away from local development: https://www.uber.com/en-CL/blog/devpod-improving-developer-productivity-at-uber/ https://www.uber.com/en-CL/blog/devpod-improving-developer-p...

maronato 3y ago

When you want to run a local version of a service, you start it locally as normal and configure either local stubs of dependencies and dependents, or you can hook up the actual “production” services to your local service process. If doing the latter, you can create special user accounts and events that change how the networking routing happens. Events from those fake users pass through regular production apps up until the service you’re testing, then they are instead routed to your local version, and you can continue the calls to other prod services. Uber employees’ apps are special and allow us to log in as these fake users and create fake rides or deliveries, and then we can look at the traces and logs to debug and stuff.