11 ms·

Show HN: Visualize the entropy of a codebase with a 3D force-directed graph

180 points by gabimtme 3y ago

Hi HN! I'm Gabriel, the author of dep-tree (https://github.com/gabotechs/dep-tree https://github.com/gabotechs/dep-tree), and I wanted to show off this tool and explain why it's being really useful at my current org for dealing with code complexity. I work at a startup where business evolves really fast, and requirements change frequently, so it's easy to end up with big piles of code stacked together without a clear structure, specially with tight deadlines. I made dep-tree [1] to help us maintain a clean code architecture and a logical separation of concerns between parts of the application, which is accomplished by: (1) Visualizing the source files and the dependencies between them using a 3D force-directed graph; and (2) Enforcing some dependency rules that allow/forbid dependencies between different parts of the application. The 3D force-directed graph visualization works like this: - It takes an entrypoint to the codebase, usually the main executable file or a library's entrypoint (index.js, main.py, etc...) - It recursively crawls import statements gathering other source files that are being depended upon - It creates a directed graph out of that, where nodes are source files and edges are the dependencies between them - It renders this graph in the browser using a 3D force-directed layout, where attraction/repulsion forces will be applied to each node depending on which other nodes it is connected to. With this, properly decoupled codebases will tend to form clusters of nodes, representing logical parts that live together and are clearly separated from other parts, and tightly coupled codebases will be rendered without clear clustering or without a clear structural pattern in the node placement. Some examples of this visualization for well-known codebases are: TypeScript: https://dep-tree-explorer.vercel.app/api?repo=https%3A%2F%2Fgithub.com%2Fmicrosoft%2FTypeScript&entrypoint=src%2Ftypescript%2Ftypescript.ts https://dep-tree-explorer.vercel.app/api?repo=https%3A%2F%2F... React: https://dep-tree-explorer.vercel.app/api?repo=https%3A%2F%2Fgithub.com%2Ffacebook%2Freact&entrypoint=packages%2Freact-dom%2Findex.js https://dep-tree-explorer.vercel.app/api?repo=https%3A%2F%2F... Svelte: https://dep-tree-explorer.vercel.app/api?repo=https%3A%2F%2Fgithub.com%2Fsveltejs%2Fsvelte&entrypoint=packages%2Fsvelte%2Fsrc%2Fcompiler%2Findex.js https://dep-tree-explorer.vercel.app/api?repo=https%3A%2F%2F... Langchain: https://dep-tree-explorer.vercel.app/api?repo=https%3A%2F%2Fgithub.com%2Flangchain-ai%2Flangchain&entrypoint=libs%2Flangchain%2Flangchain%2F__init__.py https://dep-tree-explorer.vercel.app/api?repo=https%3A%2F%2F... Numpy: https://dep-tree-explorer.vercel.app/api?repo=https%3A%2F%2Fgithub.com%2Fnumpy%2Fnumpy&entrypoint=numpy%2F__init__.py https://dep-tree-explorer.vercel.app/api?repo=https%3A%2F%2F... Deno: https://dep-tree-explorer.vercel.app/api?repo=https%3A%2F%2Fgithub.com%2Fdenoland%2Fdeno&entrypoint=cli%2Fmain.rs https://dep-tree-explorer.vercel.app/api?repo=https%3A%2F%2F... The visualizations are cool, but it's just the first step. The dependency rules checking capabilities is what makes the tool actually useful in a daily basis and what keeps us using it every day in our CI pipelines for enforcing decoupling. More info about this feature is available in the repo: https://github.com/gabotechs/dep-tree?tab=readme-ov-file#check https://github.com/gabotechs/dep-tree?tab=readme-ov-file#che.... The code is fully open-source.

deleted 3y ago
[deleted]
leetrout 3y ago
Off topic but... > I work at a startup where business evolves really fast, and requirements change frequently, so it's easy to end up with big piles of code stacked together without a clear structure, specially with tight deadlines That smells. It sounds like the team could benefit from better stack technologies and a bit more discipline in how it is applied to solutioning. > Enforcing some dependency rules that allow/forbid dependencies between different parts of the application. What is the alternative to this tool that lowers the cognitive barrier / builds the right muscles for the team to understand what they should / shouldnt depend on?
- gabimtme 3y ago
  > It sounds like the team could benefit from better stack technologies and a bit more discipline in how it is applied to solutioning. For our specific case it's actually pretty good, we've built a lot of discipline around maintainability, but in general this is a recurring problem in tech teams who might not be able to afford the time it takes to gain discipline. > What is the alternative to this tool that lowers the cognitive barrier / builds the right muscles for the team to understand what they should / shouldnt depend on? Some programming languages allow you to split the codebase into modular units (npm workspaces, cargo workspaces, etc..) which forces developers to modularize things, and dependencies between modules need to be explicitly declared. This is good, but usually not enough, as nothing prevents you to mess things up within a module/workspace. There's some other tooling with similar functionality to dep-tree, but language-specific and with visualizations not suitable for large codebases (.dot files, 2d svgs...)
  - jonmoore 3y ago
    Indeed, and tools like dep-tree provide a combination of 1) making module structure visible 2) making rules about this structure concrete and 3) automatically checking for rule violations. These all help to lower the cognitive barrier to learning and maintaining the code base effectively. For developers new to the code base they help with learning and for those more experienced they help with ongoing design and maintenance. Most long-lived code bases I've seen have adopted or built such tooling at some point, often with tools customized to the code base. For example in one large code base (c. 250 devs) we built tooling that simulated and helped optimize the changes to implement a major refactor of the overall module structure.

180 points by gabimtme 3y ago

deleted 3y ago

[deleted]

leetrout 3y ago

Off topic but... > I work at a startup where business evolves really fast, and requirements change frequently, so it's easy to end up with big piles of code stacked together without a clear structure, specially with tight deadlines That smells. It sounds like the team could benefit from better stack technologies and a bit more discipline in how it is applied to solutioning. > Enforcing some dependency rules that allow/forbid dependencies between different parts of the application. What is the alternative to this tool that lowers the cognitive barrier / builds the right muscles for the team to understand what they should / shouldnt depend on?

gabimtme 3y ago
> It sounds like the team could benefit from better stack technologies and a bit more discipline in how it is applied to solutioning. For our specific case it's actually pretty good, we've built a lot of discipline around maintainability, but in general this is a recurring problem in tech teams who might not be able to afford the time it takes to gain discipline. > What is the alternative to this tool that lowers the cognitive barrier / builds the right muscles for the team to understand what they should / shouldnt depend on? Some programming languages allow you to split the codebase into modular units (npm workspaces, cargo workspaces, etc..) which forces developers to modularize things, and dependencies between modules need to be explicitly declared. This is good, but usually not enough, as nothing prevents you to mess things up within a module/workspace. There's some other tooling with similar functionality to dep-tree, but language-specific and with visualizations not suitable for large codebases (.dot files, 2d svgs...)
- jonmoore 3y ago
  Indeed, and tools like dep-tree provide a combination of 1) making module structure visible 2) making rules about this structure concrete and 3) automatically checking for rule violations. These all help to lower the cognitive barrier to learning and maintaining the code base effectively. For developers new to the code base they help with learning and for those more experienced they help with ongoing design and maintenance. Most long-lived code bases I've seen have adopted or built such tooling at some point, often with tools customized to the code base. For example in one large code base (c. 250 devs) we built tooling that simulated and helped optimize the changes to implement a major refactor of the overall module structure.