4 ms·

Are there any visual "code flow" interpreters? Something that would separate the 1000s of interactions between functions and show flow lines between them?

gdoptimizer 6y ago
Sourcetrail?
- lstamour 6y ago
  Yep, Sourcetrail can do that, for the languages it supports. (It has an SDK so additional languages can be added, with effort.) Give it a method, another method, and it will draw a line from point A to B (with all the functions in between) using static analysis plus you can explore before and after to see what calls what. You can even see field usage though there it can be confused sometimes, understanding varies by language. But it’s still really useful. Doesn’t yet support cross-language integrations but it has a lot of potential now being open source. It works great for individual developer use, for team use I’d want to try my hand porting it to React or the web in order to more easily share views with others, and perhaps use a central database. For now you can make Sourcetrail projects as part of a CI system to share them with other team members. In addition to Sourcetrail, I also recommend adding OpenTelemetry for distributed projects or flame graphs for less distributed ones. Some of the videos Honeycomb.io put together really highlight the value of distributed tracing, such as this one: https://youtu.be/GuIWQ-EF7YE https://youtu.be/GuIWQ-EF7YE and the OpenTelemetry Collector makes it simple to filter telemetry, route it to services or drop a majority of traces which don’t have exceptions, for example. One day I hope OpenTelemetry tracing can be baked into any language the way flame graphs tend to enjoy first-class support in Java, and that tools like Sourcetrail can be baked into IDEs such that runtime metadata is available just by hovering your mouse over modules and functions. Kind of like CodeLens shown here, but for understanding the code: https://docs.microsoft.com/en-us/azure/azure-monitor/app/asp-net-exceptions#diagnosing-exceptions-using-visual-studio https://docs.microsoft.com/en-us/azure/azure-monitor/app/asp... Something like https://www.codestream.com/use-cases/code-documentation https://www.codestream.com/use-cases/code-documentation works as a social network and documentation hub but doesn’t necessarily bring in production telemetry or models/ontology from code (such as Lattix, but that’s specialized to code organization in a way…) Maybe Project Cortex but for source code? https://techcommunity.microsoft.com/t5/microsoft-365-blog/introducing-project-cortex/ba-p/966091 https://techcommunity.microsoft.com/t5/microsoft-365-blog/in... JetBrains Space or GitHub doesn’t yet analyze code beyond dependencies/security issues/CI but might in the future. Finally, there are tools like https://backstage.io/ https://backstage.io/ which hint at a future where developers build their own infra tools for the rest of the company to use… but that hasn’t extended much into the realm of modelling, documentation or telemetry yet. Folks might be lucky if they have a hosted copy of SourceGraph right now… the future, I think, builds on all of these ideas.
  - aste-risk 6y ago
    Can you use sourcetrail on properitery codebases as well? I see it's GPL and according to my understanding, it's okay to use it on properitery software as long as you don't make any modifications to the sourcetrail software itself. Are there any hidden commercial licenses before I try it out on my company's codebase?
    - lstamour 6y ago
      I’m not a lawyer but if you’re not embedding GPL code output into your code, you’re fine. Using GPL code to write or reason about code under a different license is not the same thing as having GPL software output a copy of its own GPL-licensed code, for example: https://softwareengineering.stackexchange.com/questions/52211/open-source-ides-released-under-gpl https://softwareengineering.stackexchange.com/questions/5221... The only other risk is letting your company’s proprietary code be visible by third-parties but Sourcetrail runs locally on your computer and can run completely offline. As to Sourcetrail’s licensing— it previously had a closed license and was supported by a startup with a number of employees. It recently went open source and can be supported financially through Patreon: https://www.sourcetrail.com/blog/open_source/ https://www.sourcetrail.com/blog/open_source/
nyellin 6y ago
Came here to ask the same question. I've tried to find something like this for golang and the tools out there can't handle large codebases.
Phil-bitplex 6y ago
I wrote one for Tcl years ago, as I started working on a really complicated product with around 50k lines of code (or more, distant memory). Maybe I should give it another crack for modern languages - there is an even greater need for it these days with dependency injection and microservices being common. The way I see it the need stems from needing to understand what is REALLY going on, as opposed to what the code is saying should be going on.
- blacksqr 6y ago
  Would be interested in obtaining your Tcl work if still available.
dsbyrne 6y ago
If you use VS Code with Ruby or Java, check out the AppMap extension. Its core function is similar to what you've described: diagramming execution flow and component relationships. It's dynamic analysis, so it captures data snapshots as well. https://marketplace.visualstudio.com/items?itemName=appland.appmap https://marketplace.visualstudio.com/items?itemName=appland....
mapme 6y ago
IntelliJ data flow analysis does this to a certain extent (paid feature though) https://www.jetbrains.com/help/idea/analyzing-data-flow.html https://www.jetbrains.com/help/idea/analyzing-data-flow.html
m463 6y ago
I think this is a fantasy (that many of us have). "I run this magical tool, and voila! a satellite view of the code! It clearly shows this river runs into this bay, and there's a dam over there! Now I'm miles ahead of everyone! Lookie, this river runs in circles..." Ha. What really happens is you run some tool and what comes out looks and smells like hair you pulled out of your clogged drain. I've tried this. Anything graphing shows ... well... It will look like this: https://upload.wikimedia.org/wikipedia/commons/9/9b/Social_Network_Analysis_Visualization.png https://upload.wikimedia.org/wikipedia/commons/9/9b/Social_N... I think the thing is - the code that gets thing done will confound automated visualization tools. It's basically like expecting decompiler output to be super helpful. It may help your understanding a little bit, but much will be lost. Yes there are heroic decompiler stories, but time is involved. Also, most code has macros or helper functions or automatic code generation or something that obfuscates what you're really looking for. You will have to develop a system to unblock this organically. What will help: Peruse the source code. Try to follow the flow. if you have tools to jump back and forth between a function call and definition use it. fix some bugs. Follow the stack traces up and down. ask people how stuff works. put in the time. and the other stuff mentioned in this article. The osmosis method is really how you'll get it.
- antpls 6y ago
  Maybe one could limit the graph to the happy path to begin with. The output of a run of profile-guided optimisation could be used to discover that graph, and only the touched functions would be drawn. This graph could be useful to start with the codebase, without being too overwhelming with all the edge cases.
  - m463 6y ago
    I think this depends on what you're graphing. If it's an application, I think it has a better chance of working. If it's something like a driver or something with a hal or tables of functions or callbacks it might be harder. If your codebase is large... hmm. Last time I tried something like this was a decade ago so things might have gotten better.
- kaba0 6y ago

gdoptimizer 6y ago

Sourcetrail?

lstamour 6y ago
Yep, Sourcetrail can do that, for the languages it supports. (It has an SDK so additional languages can be added, with effort.) Give it a method, another method, and it will draw a line from point A to B (with all the functions in between) using static analysis plus you can explore before and after to see what calls what. You can even see field usage though there it can be confused sometimes, understanding varies by language. But it’s still really useful. Doesn’t yet support cross-language integrations but it has a lot of potential now being open source. It works great for individual developer use, for team use I’d want to try my hand porting it to React or the web in order to more easily share views with others, and perhaps use a central database. For now you can make Sourcetrail projects as part of a CI system to share them with other team members. In addition to Sourcetrail, I also recommend adding OpenTelemetry for distributed projects or flame graphs for less distributed ones. Some of the videos Honeycomb.io put together really highlight the value of distributed tracing, such as this one: https://youtu.be/GuIWQ-EF7YE https://youtu.be/GuIWQ-EF7YE and the OpenTelemetry Collector makes it simple to filter telemetry, route it to services or drop a majority of traces which don’t have exceptions, for example. One day I hope OpenTelemetry tracing can be baked into any language the way flame graphs tend to enjoy first-class support in Java, and that tools like Sourcetrail can be baked into IDEs such that runtime metadata is available just by hovering your mouse over modules and functions. Kind of like CodeLens shown here, but for understanding the code: https://docs.microsoft.com/en-us/azure/azure-monitor/app/asp-net-exceptions#diagnosing-exceptions-using-visual-studio https://docs.microsoft.com/en-us/azure/azure-monitor/app/asp... Something like https://www.codestream.com/use-cases/code-documentation https://www.codestream.com/use-cases/code-documentation works as a social network and documentation hub but doesn’t necessarily bring in production telemetry or models/ontology from code (such as Lattix, but that’s specialized to code organization in a way…) Maybe Project Cortex but for source code? https://techcommunity.microsoft.com/t5/microsoft-365-blog/introducing-project-cortex/ba-p/966091 https://techcommunity.microsoft.com/t5/microsoft-365-blog/in... JetBrains Space or GitHub doesn’t yet analyze code beyond dependencies/security issues/CI but might in the future. Finally, there are tools like https://backstage.io/ https://backstage.io/ which hint at a future where developers build their own infra tools for the rest of the company to use… but that hasn’t extended much into the realm of modelling, documentation or telemetry yet. Folks might be lucky if they have a hosted copy of SourceGraph right now… the future, I think, builds on all of these ideas.
- aste-risk 6y ago
  Can you use sourcetrail on properitery codebases as well? I see it's GPL and according to my understanding, it's okay to use it on properitery software as long as you don't make any modifications to the sourcetrail software itself. Are there any hidden commercial licenses before I try it out on my company's codebase?
  - lstamour 6y ago
    I’m not a lawyer but if you’re not embedding GPL code output into your code, you’re fine. Using GPL code to write or reason about code under a different license is not the same thing as having GPL software output a copy of its own GPL-licensed code, for example: https://softwareengineering.stackexchange.com/questions/52211/open-source-ides-released-under-gpl https://softwareengineering.stackexchange.com/questions/5221... The only other risk is letting your company’s proprietary code be visible by third-parties but Sourcetrail runs locally on your computer and can run completely offline. As to Sourcetrail’s licensing— it previously had a closed license and was supported by a startup with a number of employees. It recently went open source and can be supported financially through Patreon: https://www.sourcetrail.com/blog/open_source/ https://www.sourcetrail.com/blog/open_source/

nyellin 6y ago

Came here to ask the same question. I've tried to find something like this for golang and the tools out there can't handle large codebases.

Phil-bitplex 6y ago

I wrote one for Tcl years ago, as I started working on a really complicated product with around 50k lines of code (or more, distant memory). Maybe I should give it another crack for modern languages - there is an even greater need for it these days with dependency injection and microservices being common. The way I see it the need stems from needing to understand what is REALLY going on, as opposed to what the code is saying should be going on.

blacksqr 6y ago
Would be interested in obtaining your Tcl work if still available.

dsbyrne 6y ago

If you use VS Code with Ruby or Java, check out the AppMap extension. Its core function is similar to what you've described: diagramming execution flow and component relationships. It's dynamic analysis, so it captures data snapshots as well. https://marketplace.visualstudio.com/items?itemName=appland.appmap https://marketplace.visualstudio.com/items?itemName=appland....

mapme 6y ago

IntelliJ data flow analysis does this to a certain extent (paid feature though) https://www.jetbrains.com/help/idea/analyzing-data-flow.html https://www.jetbrains.com/help/idea/analyzing-data-flow.html

m463 6y ago

I think this is a fantasy (that many of us have). "I run this magical tool, and voila! a satellite view of the code! It clearly shows this river runs into this bay, and there's a dam over there! Now I'm miles ahead of everyone! Lookie, this river runs in circles..." Ha. What really happens is you run some tool and what comes out looks and smells like hair you pulled out of your clogged drain. I've tried this. Anything graphing shows ... well... It will look like this: https://upload.wikimedia.org/wikipedia/commons/9/9b/Social_Network_Analysis_Visualization.png https://upload.wikimedia.org/wikipedia/commons/9/9b/Social_N... I think the thing is - the code that gets thing done will confound automated visualization tools. It's basically like expecting decompiler output to be super helpful. It may help your understanding a little bit, but much will be lost. Yes there are heroic decompiler stories, but time is involved. Also, most code has macros or helper functions or automatic code generation or something that obfuscates what you're really looking for. You will have to develop a system to unblock this organically. What will help: Peruse the source code. Try to follow the flow. if you have tools to jump back and forth between a function call and definition use it. fix some bugs. Follow the stack traces up and down. ask people how stuff works. put in the time. and the other stuff mentioned in this article. The osmosis method is really how you'll get it.

antpls 6y ago
Maybe one could limit the graph to the happy path to begin with. The output of a run of profile-guided optimisation could be used to discover that graph, and only the touched functions would be drawn. This graph could be useful to start with the codebase, without being too overwhelming with all the edge cases.
- m463 6y ago
  I think this depends on what you're graphing. If it's an application, I think it has a better chance of working. If it's something like a driver or something with a hal or tables of functions or callbacks it might be harder. If your codebase is large... hmm. Last time I tried something like this was a decade ago so things might have gotten better.
kaba0 6y ago