5 ms·

What I did to solve this: I serialized the stack frames and sent it with the messages and later when an exception happened it could be easily traced.

by edem 9y ago

hvidgaard 9y ago
I'm not sure I follow. What problems are you solving by doing this, it sounds like a rather leaky and brittle thing to do in a distributed system.
- stareatgoats 9y ago
  Why 'leaky' and 'brittle'? Honest question. Sounds like a good solution to me in order to trace exceptions back to the source
  - hvidgaard 9y ago
    Process A should not care how Process B implements anything, only how it responds to messages in a platform agnostic manner. There is zero guarantee that the stack keeps looking the same, hence brittle, and since you need to know something about how it's implemented to use the stack, it's also leaky. As soon as an exception happens, it should be logged. Not shipped off to a different process. From that log you know something about the state and the messages leading to the exception, and you can debug directly. If A send "Process this data X" to B, and B fails with an OutOfMemoryException, it should respond "Unable to process X", or perhaps "Insufficient ressources to process X".
    - stareatgoats 9y ago
      OK thanks, makes sense in genuinely decoupled microservice architecture (with messages crossing language and environment boundaries). Monolithic systems (distributed or not) seem not to have the same problems though, in which case it would still seems like a good idea to me, for debugging purposes.
      - hvidgaard 9y ago
        I'm not convinced. If you're trying to debug a process, you should only operate on the messages and state. The messages you can log, and the state should ideally come only from messages, so you truly only need the messages. One of the strong points about this way of implementing interprocess communication, is that it allows you to have a new and old version of the same process working concurrently, with the new only running and comparing it's output with the old one. Once you're convinced they work the same, you can retire the old process. To fully leverage this, every message should be platform and implementation agnostic. Even if it's a monolithic system on a single server.
        stareatgoats 9y ago
        Well, all I can say is that I personally often find the full stack interesting when debugging exceptions, and might still try and implement passing stack info in my monolithic system (in spite of this discussion which might have been above my pay-grade honestly). In case it breaches some important CS principle that I don't fully grasp yet then it will surely come back and bite me - which has happened before ... :) . Anyway, thanks for your time.
        hvidgaard 9y ago
        You're welcome. I juggle those concepts and requirements daily, and I value explaining it to the developers. The "why are we doing it this way" is really important for their job satisfaction, and it happens that they have an aspect I didn't consider, or a better solution.
        edem 9y ago
        Your intuition is right. As a programmer your primary job is in most cases not mental masturbation and/or adhering to CS principles but shipping working software. We are all bound by the 80/20 rule and our job is do decide when to adhere to the principles and when to cut corners. In practice no one will care that you implemented a 100% pure microservice if you miss an important deadline, your company loses a $300M deal and your team ends up laid off. I'm not a business person, but a programmer. I also like writing beautiful code but we have to know the tradeoffs. If the tradeoff of having a beautiful service vs a nice service is 3 weeks of additional keyboard bashing it is not worth it. I once had a colleague who insisted on implementing all REST endpoints between microservices using HATEOAS/HAL because it will enable the consumption of such endpoints by programs and generated code. He spent a month refactoring our services and we ended up never using that feature, but modifying endpoints still carry that overhead he added.
    - edem 9y ago
      This is a typical over-architecting issue you describe. - All the programs I work on are running on the JVM so I know how the trace looks like and how it is implemented so it is not leaky nor brittle -Tracing back what caused what is a pain if you don't see the full trace. - The indirection caused by an eventbus message passing a microservice boundary makes it practically impossible to debug - passing on the trace will let you know __immediately__ where the problem is I'm not a purist, I'm a pragmatist. I don't care if a solution is not pure if it works, spares my time, and most importantly: does not come with a burden. This tracing can be put behind a feature toggle, or an aspect/decorator/whatever. You might call this a hack, but it lets me ship, maintain and support software while others still stand at the whiteboard planning.

What I did to solve this: I serialized the stack frames and sent it with the messages and later when an exception happened it could be easily traced.

by edem 9y ago

What I did to solve this: I serialized the stack frames and sent it with the messages and later when an exception happened it could be easily traced.

hvidgaard 9y ago

I'm not sure I follow. What problems are you solving by doing this, it sounds like a rather leaky and brittle thing to do in a distributed system.

stareatgoats 9y ago
Why 'leaky' and 'brittle'? Honest question. Sounds like a good solution to me in order to trace exceptions back to the source
- hvidgaard 9y ago
  Process A should not care how Process B implements anything, only how it responds to messages in a platform agnostic manner. There is zero guarantee that the stack keeps looking the same, hence brittle, and since you need to know something about how it's implemented to use the stack, it's also leaky. As soon as an exception happens, it should be logged. Not shipped off to a different process. From that log you know something about the state and the messages leading to the exception, and you can debug directly. If A send "Process this data X" to B, and B fails with an OutOfMemoryException, it should respond "Unable to process X", or perhaps "Insufficient ressources to process X".
  - stareatgoats 9y ago
    OK thanks, makes sense in genuinely decoupled microservice architecture (with messages crossing language and environment boundaries). Monolithic systems (distributed or not) seem not to have the same problems though, in which case it would still seems like a good idea to me, for debugging purposes.
    - hvidgaard 9y ago
      I'm not convinced. If you're trying to debug a process, you should only operate on the messages and state. The messages you can log, and the state should ideally come only from messages, so you truly only need the messages. One of the strong points about this way of implementing interprocess communication, is that it allows you to have a new and old version of the same process working concurrently, with the new only running and comparing it's output with the old one. Once you're convinced they work the same, you can retire the old process. To fully leverage this, every message should be platform and implementation agnostic. Even if it's a monolithic system on a single server.
      - stareatgoats 9y ago
        Well, all I can say is that I personally often find the full stack interesting when debugging exceptions, and might still try and implement passing stack info in my monolithic system (in spite of this discussion which might have been above my pay-grade honestly). In case it breaches some important CS principle that I don't fully grasp yet then it will surely come back and bite me - which has happened before ... :) . Anyway, thanks for your time.
        hvidgaard 9y ago
        You're welcome. I juggle those concepts and requirements daily, and I value explaining it to the developers. The "why are we doing it this way" is really important for their job satisfaction, and it happens that they have an aspect I didn't consider, or a better solution.
        edem 9y ago
        Your intuition is right. As a programmer your primary job is in most cases not mental masturbation and/or adhering to CS principles but shipping working software. We are all bound by the 80/20 rule and our job is do decide when to adhere to the principles and when to cut corners. In practice no one will care that you implemented a 100% pure microservice if you miss an important deadline, your company loses a $300M deal and your team ends up laid off. I'm not a business person, but a programmer. I also like writing beautiful code but we have to know the tradeoffs. If the tradeoff of having a beautiful service vs a nice service is 3 weeks of additional keyboard bashing it is not worth it. I once had a colleague who insisted on implementing all REST endpoints between microservices using HATEOAS/HAL because it will enable the consumption of such endpoints by programs and generated code. He spent a month refactoring our services and we ended up never using that feature, but modifying endpoints still carry that overhead he added.
  - edem 9y ago
    This is a typical over-architecting issue you describe. - All the programs I work on are running on the JVM so I know how the trace looks like and how it is implemented so it is not leaky nor brittle -Tracing back what caused what is a pain if you don't see the full trace. - The indirection caused by an eventbus message passing a microservice boundary makes it practically impossible to debug - passing on the trace will let you know __immediately__ where the problem is I'm not a purist, I'm a pragmatist. I don't care if a solution is not pure if it works, spares my time, and most importantly: does not come with a burden. This tracing can be put behind a feature toggle, or an aspect/decorator/whatever. You might call this a hack, but it lets me ship, maintain and support software while others still stand at the whiteboard planning.