4 ms·

At my company our system is composed of 2 dozen different services and all of them can run locally in minikube and easily be debugged in jetbrains.

phito 1y ago
Yes, it's perfectly doable even if you're doing microservices. Not being able to debug your application is an engineering failure.
- thunspa 1y ago
  can you say more? how do you do it?
  - phito 1y ago
    We mostly have dotnet services in k8s, using Rider (IDE) and Telepresence for remote debugging. Having observability (OpenTelemetry) is also really useful.
  - idontwantthis 1y ago
    See ninkendo’s comment. They are doing it with the same tools we are.
- 9rx 1y ago
  A multi-process application isn't the same as microservices. Microservices is a team organization technique, seeing individual teams operate in isolation, as if they were individual businesses. You can't debug other team's services any more than you can debug what happens when you make a call to OpenAI. That is the level of separation you are up against in a microservices context. If you can, you're on the same team, and thus don't have microservices, just a single service.
  - phito 1y ago
    What? We have dozens of microservices owned by multiple teams, but nothing stops you from cloning the git repository of another team's microservice and debug it the same way you would debug your own.
    - deleted 1y ago
      [deleted]
    - 9rx 1y ago
      Service is provided by people. You, for example, discover a problem with OpenAI's system that you integrate with and the only way you can address it is to employ the services of the people who work for OpenAI. While that is an example of a macroservice (or what we usually just call a service), it playing out in the macro economy, microservice is the same concept except applied in the micro scale. But you checking out the code and debugging it means that you are providing the service. Where, exactly, do you find the service (micro or otherwise) boundary in this case? Or are you just struggling to say that your service utilizes multiple applications?
      - phito 1y ago
        Sorry but I fail to see how your comments are relevant in this discussion.
roncesvalles 1y ago
Where do configs and secrets come from? Also, big company = hundreds of microservices.
- ninkendo 1y ago
  In my experience you just slap minikube or k3s on your dev machine, and treat it as any other environment. Argo, helm, kustomize, whatever can all work against a local single-node cluster just fine. It takes some effort to make sure your configs are overridable per-environment, but it’s worth doing. (And something you’re hopefully doing anyway if you’re doing any kind of integration/test environment.) It also requires that each of your services can scale down as well as they can scale up… none of them should be so huge that your whole cluster can’t fit on a single machine, if you’re just simulating one “request” at a time. (Single instances of everything, don’t reserve memory, etc.) There’s use cases where this isn’t practical, but in most cases it’s very doable.
  - idontwantthis 1y ago
    Yes all of this.
ninkendo 1y ago
I certainly know how to debug each of the services in my environment, but how do you step-through debug a single request across services? Like, if service A make a gRPC call to service B, are you saying you can “step into” the call from A and your debugger is able to break on the corresponding call in B? And frames from the call in A are there in a logical “stack” from the breakpoint in B? (Honest question… if such a workflow is possible I’d love to hear about it. Debugging just a single service at a time in isolation is boring and obvious, but if there’s something I’m missing I’d be really curious.)
- idontwantthis 1y ago
  No I can’t debug multiple services at once, unfortunately. But I will switch between them as I track the request over multiple runs. Also extensive logging in available in grafana helps me know which service is having the issue before I start debugging.
  - steve_adams_86 1y ago
    This is usually enough for me, too. Use tracing, figure out where things fell apart in the traces, isolate those service(s), and debug from there. It's definitely more work. When we start new projects, I encourage people not to use services until proven necessary because this added layer of friction is a real drag. For a lot of us that isn't a choice, though.

phito 1y ago

Yes, it's perfectly doable even if you're doing microservices. Not being able to debug your application is an engineering failure.

thunspa 1y ago
can you say more? how do you do it?
- phito 1y ago
  We mostly have dotnet services in k8s, using Rider (IDE) and Telepresence for remote debugging. Having observability (OpenTelemetry) is also really useful.
- idontwantthis 1y ago
  See ninkendo’s comment. They are doing it with the same tools we are.
9rx 1y ago
A multi-process application isn't the same as microservices. Microservices is a team organization technique, seeing individual teams operate in isolation, as if they were individual businesses. You can't debug other team's services any more than you can debug what happens when you make a call to OpenAI. That is the level of separation you are up against in a microservices context. If you can, you're on the same team, and thus don't have microservices, just a single service.
- phito 1y ago
  What? We have dozens of microservices owned by multiple teams, but nothing stops you from cloning the git repository of another team's microservice and debug it the same way you would debug your own.
  - deleted 1y ago
    [deleted]
  - 9rx 1y ago
    Service is provided by people. You, for example, discover a problem with OpenAI's system that you integrate with and the only way you can address it is to employ the services of the people who work for OpenAI. While that is an example of a macroservice (or what we usually just call a service), it playing out in the macro economy, microservice is the same concept except applied in the micro scale. But you checking out the code and debugging it means that you are providing the service. Where, exactly, do you find the service (micro or otherwise) boundary in this case? Or are you just struggling to say that your service utilizes multiple applications?
    - phito 1y ago
      Sorry but I fail to see how your comments are relevant in this discussion.

roncesvalles 1y ago

Where do configs and secrets come from? Also, big company = hundreds of microservices.

ninkendo 1y ago
In my experience you just slap minikube or k3s on your dev machine, and treat it as any other environment. Argo, helm, kustomize, whatever can all work against a local single-node cluster just fine. It takes some effort to make sure your configs are overridable per-environment, but it’s worth doing. (And something you’re hopefully doing anyway if you’re doing any kind of integration/test environment.) It also requires that each of your services can scale down as well as they can scale up… none of them should be so huge that your whole cluster can’t fit on a single machine, if you’re just simulating one “request” at a time. (Single instances of everything, don’t reserve memory, etc.) There’s use cases where this isn’t practical, but in most cases it’s very doable.
- idontwantthis 1y ago
  Yes all of this.

ninkendo 1y ago

I certainly know how to debug each of the services in my environment, but how do you step-through debug a single request across services? Like, if service A make a gRPC call to service B, are you saying you can “step into” the call from A and your debugger is able to break on the corresponding call in B? And frames from the call in A are there in a logical “stack” from the breakpoint in B? (Honest question… if such a workflow is possible I’d love to hear about it. Debugging just a single service at a time in isolation is boring and obvious, but if there’s something I’m missing I’d be really curious.)

idontwantthis 1y ago
No I can’t debug multiple services at once, unfortunately. But I will switch between them as I track the request over multiple runs. Also extensive logging in available in grafana helps me know which service is having the issue before I start debugging.
- steve_adams_86 1y ago
  This is usually enough for me, too. Use tracing, figure out where things fell apart in the traces, isolate those service(s), and debug from there. It's definitely more work. When we start new projects, I encourage people not to use services until proven necessary because this added layer of friction is a real drag. For a lot of us that isn't a choice, though.