5 ms·

I'd love to use a real debugger but as someone who has only ever worked at large companies, this was just never an option. In a microservices mesh architecture,

I'd love to use a real debugger but as someone who has only ever worked at large companies, this was just never an option. In a microservices mesh architecture, you can't really run anything locally at all, and the test environment is often not configured to allow hooking up a stepping debugger. Print debugging is all you have. If there's a problem with the logging system itself or something that crashes the program before the logs can flush, then not even that.

frollogaston 1y ago
Same, this isn't my choice, debuggers don't work here. And we don't even have microservices.
- monkeyelite 1y ago
  > debuggers don't work here It’s impossible? Or would take engineering work to enable
  - frollogaston 1y ago
    It's not a realtime system kind of thing where the debugger would change the behavior too much... It's possible with enough engineering work, but nobody has put that work in, in fact they had a debugger for some staging envs that they deleted. Lately they keep adding more red tape making it hard to even run something locally, let alone attach a debugger. I guess you can attach a debugger for unit tests, but that's not very useful.
    - Kwantuum 1y ago
      > I guess you can attach a debugger for unit tests, but that's not very useful. That is in fact incredibly useful
      - frollogaston 1y ago
        Eh, it's there for those who want it, but nobody uses it
        monkeyelite 1y ago
        See this just sounds like you do not have an engineering culture or learning, enabling, and using debuggers.
        deleted 1y ago
        [deleted]
        frollogaston 1y ago
        Not sure what an engineering culture is, but I don't want it. I just want to attach a debugger to our stuff (not the unit tests).
        monkeyelite 1y ago
        *engineering culture of It’s a similar muscle to exercise as using a profiler.
idontwantthis 1y ago
At my company our system is composed of 2 dozen different services and all of them can run locally in minikube and easily be debugged in jetbrains.
- phito 1y ago
  Yes, it's perfectly doable even if you're doing microservices. Not being able to debug your application is an engineering failure.
  - thunspa 1y ago
    can you say more? how do you do it?
    - phito 1y ago
      We mostly have dotnet services in k8s, using Rider (IDE) and Telepresence for remote debugging. Having observability (OpenTelemetry) is also really useful.
    - idontwantthis 1y ago
      See ninkendo’s comment. They are doing it with the same tools we are.
  - 9rx 1y ago
    A multi-process application isn't the same as microservices. Microservices is a team organization technique, seeing individual teams operate in isolation, as if they were individual businesses. You can't debug other team's services any more than you can debug what happens when you make a call to OpenAI. That is the level of separation you are up against in a microservices context. If you can, you're on the same team, and thus don't have microservices, just a single service.
    - phito 1y ago
      What? We have dozens of microservices owned by multiple teams, but nothing stops you from cloning the git repository of another team's microservice and debug it the same way you would debug your own.
alisonatwork 1y ago
This is basically it. When I started programming in C, I used a debugger all the time. Even a bit later doing Java monoliths I could spin up the whole app on my local and debug in the IDE. But nowadays running a dozen processes and containers and whatnot, it's just hopeless. The individual developer experience has gone very much backwards in the microservice era so the best thing to do is embrace observability, feature toggles etc and test in prod or a staging environment somewhere outside of your local machine.
mananaysiempre 1y ago
> In a microservices mesh architecture, you can't really run anything locally at all, and the test environment is often not configured to allow hooking up a stepping debugger. I don't often use a debugger, and I still feel the need to point out Visual Studio could step into DCOM RPCs across machines in the first release of DCOM, ca. 1995. (The COM specification has a description of how that is accomplished.)
soseng 1y ago
Curious to learn more about why it is difficult to debug. I'm not familiar with service mesh. I also work at a large corp, but we use gateways and most things are event driven with kafka across domain boundaries. I spend most of my time debugging each service locally by injecting mock messages or objects. I do this one at a time if the problem is upstream. Usually, our logging helps us pinpoint the exact service at to target for debugging. It's really easy. Our devops infrastructure has built out patterns and libraries when teams need to mint a new service. Everything is standardized with terraform. Everything has the same standard swagger pages, everything is using okta, etc.. Seems a bit boring (which is good)
- theshrike79 1y ago
  It's easy if Someone(tm) has already set up a system where you can whip up the relevant bits with something like localstack. But if there is no support from anyone and you'll be starting from scratch, you've print-debugged and fixed the issue before you get the debugger attached to anything relevant.
  - soseng 1y ago
    I suppose in the description of "large corp" and service mesh, Someone would already exist and this would already have been worked out. It would be a nightmare dealing with hundreds of microservices without this kind of game plan.

frollogaston 1y ago

Same, this isn't my choice, debuggers don't work here. And we don't even have microservices.

monkeyelite 1y ago
> debuggers don't work here It’s impossible? Or would take engineering work to enable
- frollogaston 1y ago
  It's not a realtime system kind of thing where the debugger would change the behavior too much... It's possible with enough engineering work, but nobody has put that work in, in fact they had a debugger for some staging envs that they deleted. Lately they keep adding more red tape making it hard to even run something locally, let alone attach a debugger. I guess you can attach a debugger for unit tests, but that's not very useful.
  - Kwantuum 1y ago
    > I guess you can attach a debugger for unit tests, but that's not very useful. That is in fact incredibly useful
    - frollogaston 1y ago
      Eh, it's there for those who want it, but nobody uses it
      - monkeyelite 1y ago
        See this just sounds like you do not have an engineering culture or learning, enabling, and using debuggers.
        deleted 1y ago
        [deleted]
        frollogaston 1y ago
        Not sure what an engineering culture is, but I don't want it. I just want to attach a debugger to our stuff (not the unit tests).
        monkeyelite 1y ago
        *engineering culture of It’s a similar muscle to exercise as using a profiler.

idontwantthis 1y ago

At my company our system is composed of 2 dozen different services and all of them can run locally in minikube and easily be debugged in jetbrains.

phito 1y ago
Yes, it's perfectly doable even if you're doing microservices. Not being able to debug your application is an engineering failure.
- thunspa 1y ago
  can you say more? how do you do it?
  - phito 1y ago
    We mostly have dotnet services in k8s, using Rider (IDE) and Telepresence for remote debugging. Having observability (OpenTelemetry) is also really useful.
  - idontwantthis 1y ago
    See ninkendo’s comment. They are doing it with the same tools we are.
- 9rx 1y ago
  A multi-process application isn't the same as microservices. Microservices is a team organization technique, seeing individual teams operate in isolation, as if they were individual businesses. You can't debug other team's services any more than you can debug what happens when you make a call to OpenAI. That is the level of separation you are up against in a microservices context. If you can, you're on the same team, and thus don't have microservices, just a single service.
  - phito 1y ago
    What? We have dozens of microservices owned by multiple teams, but nothing stops you from cloning the git repository of another team's microservice and debug it the same way you would debug your own.

alisonatwork 1y ago

This is basically it. When I started programming in C, I used a debugger all the time. Even a bit later doing Java monoliths I could spin up the whole app on my local and debug in the IDE. But nowadays running a dozen processes and containers and whatnot, it's just hopeless. The individual developer experience has gone very much backwards in the microservice era so the best thing to do is embrace observability, feature toggles etc and test in prod or a staging environment somewhere outside of your local machine.

mananaysiempre 1y ago

> In a microservices mesh architecture, you can't really run anything locally at all, and the test environment is often not configured to allow hooking up a stepping debugger. I don't often use a debugger, and I still feel the need to point out Visual Studio could step into DCOM RPCs across machines in the first release of DCOM, ca. 1995. (The COM specification has a description of how that is accomplished.)

soseng 1y ago

Curious to learn more about why it is difficult to debug. I'm not familiar with service mesh. I also work at a large corp, but we use gateways and most things are event driven with kafka across domain boundaries. I spend most of my time debugging each service locally by injecting mock messages or objects. I do this one at a time if the problem is upstream. Usually, our logging helps us pinpoint the exact service at to target for debugging. It's really easy. Our devops infrastructure has built out patterns and libraries when teams need to mint a new service. Everything is standardized with terraform. Everything has the same standard swagger pages, everything is using okta, etc.. Seems a bit boring (which is good)

theshrike79 1y ago
It's easy if Someone(tm) has already set up a system where you can whip up the relevant bits with something like localstack. But if there is no support from anyone and you'll be starting from scratch, you've print-debugged and fixed the issue before you get the debugger attached to anything relevant.
- soseng 1y ago
  I suppose in the description of "large corp" and service mesh, Someone would already exist and this would already have been worked out. It would be a nightmare dealing with hundreds of microservices without this kind of game plan.