6 ms·

Debugging Distributed Systems

201 points by luu 7y ago

di4na 7y ago
Aaaaaaaand ofc there is nothing about the tools thst are in the main runtime built around distributed systems. Their explanation on tracing show they have never played with an erlang system. As someone that build distributed systems everyday, i would really like all of our industry to play more with production erlang systems. That would answer a lot of these explorations instead of badly reinventing the wheel over and over again.
- tempotemporary 7y ago
  Any good stuff to read on topic you can recommend?
  - di4na 7y ago
    Hmm I would say try to read Erlang in Anger, test it in a toy app, that is not hard to get. https://www.erlang-in-anger.com/ https://www.erlang-in-anger.com/ This is also a good entry point https://medium.com/@mrjoelkemp/jvm-struggles-and-the-beam-4d9c58547410 https://medium.com/@mrjoelkemp/jvm-struggles-and-the-beam-4d...
- mcqueenjordan 7y ago
  Language/runtime is not the hard part of building distributed systems.
  - dnautics 7y ago
    Have you ever used erlang as a distributed system in production?
  - neutronicus 7y ago
    I always find myself holding this view ... Until someone bumps an MPI version and I can't fix my God damn build for a week
  - di4na 7y ago
    I meant about the debugging tools that OTP ship with. There is nothing in this about messages or function tracing dynamically. Even less about the kind of introspection you get with it. I totally agree with your points, i am just pointing that it seems they do not seem to know that this kind of tooling are even possible.
- vardump 7y ago
  The problem is legacy and how codebases have grown in the past. How would you deploy erlang given a system that has grown out of a legacy C++ codebase and needs to run on a mix of mobile devices, IoT embedded systems (some with less than 64 kB RAM), PCs and servers? Some distributed systems are heterogenous and will probably always be. Now, getting back to the main topic, any insights from erlang or otherwise how to troubleshoot said systems that are deployed to customers?
  - AzzieElbab 7y ago
    No idea about deployment but I would ask couchdb people. https://en.wikipedia.org/wiki/Apache_CouchDB https://en.wikipedia.org/wiki/Apache_CouchDB There is an embedded version too
  - di4na 7y ago
    I meant about the debugging tools that OTP ship with. There is nothing in this about messages or function tracing dynamically. Even less about the kind of introspection you get with it. I totally agree with your points, i am just pointing that it seems they do not seem to know that this kind of tooling are even possible. Give your operators the ability to explore the system free form with these kind of tracing, dynamically, on production systems. It helps a lot. And yes that will need a lot of work in current runtimes.
- h91wka 7y ago
  As a systems Erlang designer who worked for 2 of the biggest Erlang companies in the market, I can assure you that Erlang/OTP is not some magic pixie dust that suddenly makes all the distributed systems problems go away. Playing is one thing, making a reliable, fault-tolerant system is another. OTP doesn't give you _state of art_ distributed consensus algorithms out of the box.
  - ignoramous 7y ago
    AWS built simple-db with Erlang [0] but then seemingly haven't been using it much ever since... I wouldn't be surprised if simple-db has since been moved to aurora (like document-db [1], neptune, and timestream) or dynamo-db [2] behind the scenes? [0] https://en.wikipedia.org/wiki/Amazon_SimpleDB https://en.wikipedia.org/wiki/Amazon_SimpleDB [1] https://news.ycombinator.com/item?id=18869755 https://news.ycombinator.com/item?id=18869755 [2] https://news.ycombinator.com/item?id=18871854 https://news.ycombinator.com/item?id=18871854
    - rubiquity 7y ago
      Disclaimer: Former Erlanger, currently at AWS but these are disjoint properties. I’ve never worked on SimpleDB or even looked at it, but I think one lesson several database creators learned is that Erlang isn’t high throughout enough for certain classes of database systems. Especially with the last decade’s trend of making potential adoption of a given database as wide as possible and how important benchmarks are for marketing. I still think Erlang is wonderful for writing control plane type software, given the culture around using FSM-like structures.
      - ignoramous 7y ago
        > Erlang isn’t high throughout enough for certain classes of database systems. Interestingly, one of the early simple-db engineers, James Larson [0], later built Megastore [1] at Google, which seems a lot like a retake of simple-db, and a predecessor to Google F1 [2]. I wonder if Megastore uses Erlang. [0] https://www.linkedin.com/in/jilarson https://www.linkedin.com/in/jilarson [1] https://research.google/pubs/pub36971/ https://research.google/pubs/pub36971/ [2] https://research.google/pubs/pub41344/ https://research.google/pubs/pub41344/
- namelosw 7y ago
  From Erlang's perspective, debugging an average distributed application is just like being primitives hunting in the forest directly after graduating from college. I mean, there's even no parameter in the call stack, people can only guess instead of knowing things for sure. Of course, there are more important things like remote debugging in Erlang. To fix a production issue is like fixing a plane that is malfunctioned and crashing: For average language and runtime, the plan is to take a guess, then develop a theory, build a malfunctioned plane according to the theory. If it's not having same issue, repeat previous steps until people rebuild a plane having exact same issue as the crashing one. Then fix the plane having the same issue, and apply it to the crashing plane. For Erlang, the process would be, examine the malfunctioned plane to see where the problem is for sure, directly patch it in place to make it usable again. And then build a real fix, applying it without restart anything. Tools may not be the most useful things in the world but they definitely could be game-changers sometimes.
epdlxjmonad 7y ago
For debugging a distributed system, it may be just okay to use the traditional way consisting of testing, log analysis, and visualizing that everyone is familiar with. Yes, there are advanced techniques such as formal verification and model checking, but depending on the complexity of the targe distributed system, it may be practially out of the question or just not worth your time to try to apply these techniques (unless you are in a research lab or supported by FAANG). In other words, it may be that there is nothing inherently inefficient with sticking to the traditional way because distributed systems are hard to debug by definition and there is (and will be) no panacea. We have gone through the pain of testing and debugging a distributed system that is under development for the past 5 years. We investigated several fancy ways of debugging distributed systems such as model checking and formal verification. In the end, we decided to use (and are more or less happy with) the traditional way. The decision was made mostly because 1) the implementation is too complex (a lot of Java and Scala code) to allow formal techniques; 2) the traditional way can still be very effective when combined with careful design and thorough testing. Before building the distributed system, we worked on formal verification using program logic and were knowledgeable about a particular sub-field. From our experience, I guess it would require a PhD dissertation to successfully apply formal techniques to debugging our distributed system. The summary of our experience is at https://www.datamonad.com/post/2020-02-19-testing-mr3/ https://www.datamonad.com/post/2020-02-19-testing-mr3/.
- neutronicus 7y ago
  When I worked on supercomputer simulations, I always landed on just dumping intermediate results and looking at them over figuring out how to compile something TotalView could look at. When the latter was possible it saved time but it was always breaking under me and I eventually said fuck it
- collyw 7y ago
  I wonder is the promise of cheap hardware using distributed systems is offset by the increased complexity and developer time. Stack overflow scaled up rather than out and I have never seen a problem with their site.
  - 7y ago
orbifold 7y ago
This is perhaps one of the few places where software developers could learn something from hardware development: In hardware description languages everything happens in parallel by default. A large chip project has many components that communicate typically fully asynchronously over some network / bus. Tools for debugging these systems are - A GUI that allows one to trace all state changes down to the single bit level. - Assertions based on linear temporal logic to verify invariants. - Conventions around how to model transactions at various levels of abstraction, as well on how to monitor and extract these transaction from the "wire protocol".
- marcosdumay 7y ago
  The article talks about all three, but with different names. At the end of the day, we keep hardware as simple as possible, so it's easier to verify, and all the extra complexity goes into software, so it's easier to iterate. As a consequence, the hardware tooling is far from enough for software debugging. Software has more powerful tools, but those aren't enough either - distributed systems are a hairy problem.
- overgard 7y ago
  Personally I'd love to see an article that could relate those techniques to current cloud technologies (if it's possible?)
- user5994461 7y ago
  I've thought about it and I don't think hardware design can map to software. Hardware is all about individual blackbox components. Take some bits on the left, give some bits on the right. Chain an infinite amount of components. It's "easy" to layout and analyze as a linear flow. Software is not a linear flow with defined states. How to represent a syscall or an IO? The minimal state of an application is its entire memory.
neutronicus 7y ago
For systems using MPI there is the TotalView debugger, which is pretty good
formalsystem 7y ago
Does anyone know of a good distributed systems resource? It's one of the rare areas of CS I have a lot of trouble finding good tutorials and textbooks. Is it just something you pick up with pure practice?
- Denzel 7y ago
  https://pdos.csail.mit.edu/6.824/ https://pdos.csail.mit.edu/6.824/ is a good start.
  - enitihas 7y ago
    Does the link have any content? I don't find anything.
    - alexeldeib 7y ago
      Click the schedule on the left navbar for lecture links. Lab links are there too.
      - enitihas 7y ago
        Thanks, didn't notice that at first.
  - taude 7y ago
    It's pretty amazing how quick the industry has changed, and the course content has followed along with this. That's a lot of variety of content for 21 lectures. Would love to see a formalized online version of some of this stuff, for even an overview, high-level. Most of the stuff I find is pretty specific for a particular cloud-native environment.
  - taude 7y ago
    I also noticed that the lectures are on YouTube[1] [1] https://www.youtube.com/watch?v=cQP8WApzIQQ&list=PLrw6a1wE39_tb2fErI4-WkMbsvGQk9_UB https://www.youtube.com/watch?v=cQP8WApzIQQ&list=PLrw6a1wE39...
ram_rar 7y ago
similar to crypto libraries, one should seriously rethink about building a distributed systems from scratch. Unless thats the only goal. I have worked on this topic in academia and industry for many years and I can safely say, there is a huge disconnect between real world tools and academia. ROI for model checkers is too low, unless your building mission critical systems. For a CRUD like application, simple things like 1. Distributed Tracing 2. Log Analysis 3. Basic concepts like exponential backoff with retries etc seems to iron out most of the issues. Lately, we started using chaos testing methods, where we have dedicated build pipelines with fault injection and see, if the system can function as expected. That has helped us uncover / fix many distributed system bugs. In terms of ROI and learning curve, chaos testing seems a much better approach than using model checkers and other sophisticated methods.
e79 7y ago
This dismisses theorem proving as too difficult to use for existing systems. My experience with old, complex systems is that they’re often old, complex and not very well understood anymore. Theorem proving is all about producing a specification and asking software to help you certify its correctness. I get that it’s often very hard in practice, but wouldn’t this process be incredibly beneficial then for existing systems? The endeavor would likely motivate the creation of an up-to-date spec and prove or disprove its correctness all at the same time. Whether the existing implementation matches the spec is another challenge, but this could at least uncover serious design flaws in high assurance software.
winrid 7y ago
Logs and request ids are so important. One trick I found is to load a page with some ?devondebug query param, then search logs. Then look at those logs to get request ids to search with again to get the whole picture. Unless you have Istio. That's amazing.
cjlovett 7y ago
I agree with the paper, it has a great description of the problems with debugging distributed systems. But I think it is missing one type of solution which we call "Systematic Testing". See https://microsoft.github.io/coyote/ https://microsoft.github.io/coyote/. This is a testing tool that integrates into your development process and has been used successfully by several large very complex distributed systems inside of Azure.

201 points by luu 7y ago

Aaaaaaaand ofc there is nothing about the tools thst are in the main runtime built around distributed systems. Their explanation on tracing show they have never played with an erlang system. As someone that build distributed systems everyday, i would really like all of our industry to play more with production erlang systems. That would answer a lot of these explorations instead of badly reinventing the wheel over and over again.

tempotemporary 7y ago
Any good stuff to read on topic you can recommend?
- di4na 7y ago
  Hmm I would say try to read Erlang in Anger, test it in a toy app, that is not hard to get. https://www.erlang-in-anger.com/ https://www.erlang-in-anger.com/ This is also a good entry point https://medium.com/@mrjoelkemp/jvm-struggles-and-the-beam-4d9c58547410 https://medium.com/@mrjoelkemp/jvm-struggles-and-the-beam-4d...
mcqueenjordan 7y ago
Language/runtime is not the hard part of building distributed systems.
- dnautics 7y ago
  Have you ever used erlang as a distributed system in production?
- neutronicus 7y ago
  I always find myself holding this view ... Until someone bumps an MPI version and I can't fix my God damn build for a week
- di4na 7y ago
  I meant about the debugging tools that OTP ship with. There is nothing in this about messages or function tracing dynamically. Even less about the kind of introspection you get with it. I totally agree with your points, i am just pointing that it seems they do not seem to know that this kind of tooling are even possible.
vardump 7y ago
The problem is legacy and how codebases have grown in the past. How would you deploy erlang given a system that has grown out of a legacy C++ codebase and needs to run on a mix of mobile devices, IoT embedded systems (some with less than 64 kB RAM), PCs and servers? Some distributed systems are heterogenous and will probably always be. Now, getting back to the main topic, any insights from erlang or otherwise how to troubleshoot said systems that are deployed to customers?
- AzzieElbab 7y ago
  No idea about deployment but I would ask couchdb people. https://en.wikipedia.org/wiki/Apache_CouchDB https://en.wikipedia.org/wiki/Apache_CouchDB There is an embedded version too
- di4na 7y ago
  I meant about the debugging tools that OTP ship with. There is nothing in this about messages or function tracing dynamically. Even less about the kind of introspection you get with it. I totally agree with your points, i am just pointing that it seems they do not seem to know that this kind of tooling are even possible. Give your operators the ability to explore the system free form with these kind of tracing, dynamically, on production systems. It helps a lot. And yes that will need a lot of work in current runtimes.
h91wka 7y ago
As a systems Erlang designer who worked for 2 of the biggest Erlang companies in the market, I can assure you that Erlang/OTP is not some magic pixie dust that suddenly makes all the distributed systems problems go away. Playing is one thing, making a reliable, fault-tolerant system is another. OTP doesn't give you _state of art_ distributed consensus algorithms out of the box.
- ignoramous 7y ago
  AWS built simple-db with Erlang [0] but then seemingly haven't been using it much ever since... I wouldn't be surprised if simple-db has since been moved to aurora (like document-db [1], neptune, and timestream) or dynamo-db [2] behind the scenes? [0] https://en.wikipedia.org/wiki/Amazon_SimpleDB https://en.wikipedia.org/wiki/Amazon_SimpleDB [1] https://news.ycombinator.com/item?id=18869755 https://news.ycombinator.com/item?id=18869755 [2] https://news.ycombinator.com/item?id=18871854 https://news.ycombinator.com/item?id=18871854
  - rubiquity 7y ago
    Disclaimer: Former Erlanger, currently at AWS but these are disjoint properties. I’ve never worked on SimpleDB or even looked at it, but I think one lesson several database creators learned is that Erlang isn’t high throughout enough for certain classes of database systems. Especially with the last decade’s trend of making potential adoption of a given database as wide as possible and how important benchmarks are for marketing. I still think Erlang is wonderful for writing control plane type software, given the culture around using FSM-like structures.
    - ignoramous 7y ago
      > Erlang isn’t high throughout enough for certain classes of database systems. Interestingly, one of the early simple-db engineers, James Larson [0], later built Megastore [1] at Google, which seems a lot like a retake of simple-db, and a predecessor to Google F1 [2]. I wonder if Megastore uses Erlang. [0] https://www.linkedin.com/in/jilarson https://www.linkedin.com/in/jilarson [1] https://research.google/pubs/pub36971/ https://research.google/pubs/pub36971/ [2] https://research.google/pubs/pub41344/ https://research.google/pubs/pub41344/
namelosw 7y ago
From Erlang's perspective, debugging an average distributed application is just like being primitives hunting in the forest directly after graduating from college. I mean, there's even no parameter in the call stack, people can only guess instead of knowing things for sure. Of course, there are more important things like remote debugging in Erlang. To fix a production issue is like fixing a plane that is malfunctioned and crashing: For average language and runtime, the plan is to take a guess, then develop a theory, build a malfunctioned plane according to the theory. If it's not having same issue, repeat previous steps until people rebuild a plane having exact same issue as the crashing one. Then fix the plane having the same issue, and apply it to the crashing plane. For Erlang, the process would be, examine the malfunctioned plane to see where the problem is for sure, directly patch it in place to make it usable again. And then build a real fix, applying it without restart anything. Tools may not be the most useful things in the world but they definitely could be game-changers sometimes.

epdlxjmonad 7y ago

For debugging a distributed system, it may be just okay to use the traditional way consisting of testing, log analysis, and visualizing that everyone is familiar with. Yes, there are advanced techniques such as formal verification and model checking, but depending on the complexity of the targe distributed system, it may be practially out of the question or just not worth your time to try to apply these techniques (unless you are in a research lab or supported by FAANG). In other words, it may be that there is nothing inherently inefficient with sticking to the traditional way because distributed systems are hard to debug by definition and there is (and will be) no panacea. We have gone through the pain of testing and debugging a distributed system that is under development for the past 5 years. We investigated several fancy ways of debugging distributed systems such as model checking and formal verification. In the end, we decided to use (and are more or less happy with) the traditional way. The decision was made mostly because 1) the implementation is too complex (a lot of Java and Scala code) to allow formal techniques; 2) the traditional way can still be very effective when combined with careful design and thorough testing. Before building the distributed system, we worked on formal verification using program logic and were knowledgeable about a particular sub-field. From our experience, I guess it would require a PhD dissertation to successfully apply formal techniques to debugging our distributed system. The summary of our experience is at https://www.datamonad.com/post/2020-02-19-testing-mr3/ https://www.datamonad.com/post/2020-02-19-testing-mr3/.

neutronicus 7y ago
When I worked on supercomputer simulations, I always landed on just dumping intermediate results and looking at them over figuring out how to compile something TotalView could look at. When the latter was possible it saved time but it was always breaking under me and I eventually said fuck it
collyw 7y ago
I wonder is the promise of cheap hardware using distributed systems is offset by the increased complexity and developer time. Stack overflow scaled up rather than out and I have never seen a problem with their site.
- 7y ago

orbifold 7y ago

This is perhaps one of the few places where software developers could learn something from hardware development: In hardware description languages everything happens in parallel by default. A large chip project has many components that communicate typically fully asynchronously over some network / bus. Tools for debugging these systems are - A GUI that allows one to trace all state changes down to the single bit level. - Assertions based on linear temporal logic to verify invariants. - Conventions around how to model transactions at various levels of abstraction, as well on how to monitor and extract these transaction from the "wire protocol".

marcosdumay 7y ago
The article talks about all three, but with different names. At the end of the day, we keep hardware as simple as possible, so it's easier to verify, and all the extra complexity goes into software, so it's easier to iterate. As a consequence, the hardware tooling is far from enough for software debugging. Software has more powerful tools, but those aren't enough either - distributed systems are a hairy problem.
overgard 7y ago
Personally I'd love to see an article that could relate those techniques to current cloud technologies (if it's possible?)
user5994461 7y ago
I've thought about it and I don't think hardware design can map to software. Hardware is all about individual blackbox components. Take some bits on the left, give some bits on the right. Chain an infinite amount of components. It's "easy" to layout and analyze as a linear flow. Software is not a linear flow with defined states. How to represent a syscall or an IO? The minimal state of an application is its entire memory.

neutronicus 7y ago

For systems using MPI there is the TotalView debugger, which is pretty good

formalsystem 7y ago

Does anyone know of a good distributed systems resource? It's one of the rare areas of CS I have a lot of trouble finding good tutorials and textbooks. Is it just something you pick up with pure practice?

Denzel 7y ago
https://pdos.csail.mit.edu/6.824/ https://pdos.csail.mit.edu/6.824/ is a good start.
- enitihas 7y ago
  Does the link have any content? I don't find anything.
  - alexeldeib 7y ago
    Click the schedule on the left navbar for lecture links. Lab links are there too.
    - enitihas 7y ago
      Thanks, didn't notice that at first.
- taude 7y ago
  It's pretty amazing how quick the industry has changed, and the course content has followed along with this. That's a lot of variety of content for 21 lectures. Would love to see a formalized online version of some of this stuff, for even an overview, high-level. Most of the stuff I find is pretty specific for a particular cloud-native environment.
- taude 7y ago
  I also noticed that the lectures are on YouTube[1] [1] https://www.youtube.com/watch?v=cQP8WApzIQQ&list=PLrw6a1wE39_tb2fErI4-WkMbsvGQk9_UB https://www.youtube.com/watch?v=cQP8WApzIQQ&list=PLrw6a1wE39...

ram_rar 7y ago

similar to crypto libraries, one should seriously rethink about building a distributed systems from scratch. Unless thats the only goal. I have worked on this topic in academia and industry for many years and I can safely say, there is a huge disconnect between real world tools and academia. ROI for model checkers is too low, unless your building mission critical systems. For a CRUD like application, simple things like 1. Distributed Tracing 2. Log Analysis 3. Basic concepts like exponential backoff with retries etc seems to iron out most of the issues. Lately, we started using chaos testing methods, where we have dedicated build pipelines with fault injection and see, if the system can function as expected. That has helped us uncover / fix many distributed system bugs. In terms of ROI and learning curve, chaos testing seems a much better approach than using model checkers and other sophisticated methods.

e79 7y ago

This dismisses theorem proving as too difficult to use for existing systems. My experience with old, complex systems is that they’re often old, complex and not very well understood anymore. Theorem proving is all about producing a specification and asking software to help you certify its correctness. I get that it’s often very hard in practice, but wouldn’t this process be incredibly beneficial then for existing systems? The endeavor would likely motivate the creation of an up-to-date spec and prove or disprove its correctness all at the same time. Whether the existing implementation matches the spec is another challenge, but this could at least uncover serious design flaws in high assurance software.

winrid 7y ago

Logs and request ids are so important. One trick I found is to load a page with some ?devondebug query param, then search logs. Then look at those logs to get request ids to search with again to get the whole picture. Unless you have Istio. That's amazing.

cjlovett 7y ago

I agree with the paper, it has a great description of the problems with debugging distributed systems. But I think it is missing one type of solution which we call "Systematic Testing". See https://microsoft.github.io/coyote/ https://microsoft.github.io/coyote/. This is a testing tool that integrates into your development process and has been used successfully by several large very complex distributed systems inside of Azure.