3 ms·

Start at the main function. See how things get setup and walk through the code from there. Keep notes on the structure and flow of things (if any). There isn't

Start at the main function. See how things get setup and walk through the code from there. Keep notes on the structure and flow of things (if any). There isn't really an easy way to do this unless it had been documented properly before. AstroGrep is a good Windows based tool that allows you to search within file so you could use it to find which files spit out a particular output to screen. Not sure what you mean by using ASTs though.

jacquesm 12y ago
AST = abstract syntax tree. With 100M lines your process will take many years.
- eklavya 12y ago
  Am I missing something or are you saying that making an AST like a compiler will help you understand a huge codebase better and faster?
  - jacquesm 12y ago
    No, an AST will not help but a call graph certainly would (it shows you how the various routines are organized in graph form, who calls who). An AST for 100M lines would be absolute madness, a call graph just might work and I'm somewhat hoping that it turns out to be either a ton of generated or duplicated code. I also wonder if the OP isn't out of his depth based on the question(s) asked.
  - klibertp 12y ago
    "making an AST like a compiler" will give you a semantic - as contrasted with textual - understanding of the code. Especially with how much data Java encodes in its source, this makes it a very good base for running automated analysis and visualisation tools, perhaps written by yourself. In general having AST is always better than having a plain text file, unless you want to read it. But then you can easily dump AST back to text whenever you want. Yeah, making AST will help you analyse your codebase programmatically which in turn will let you understand the codebase better and faster. This is some very basic programming knowledge, I think. Or is it not? Some comments here don't know what AST even is - is this the state of PL knowledge in the mainstream? Lisp and Smalltalk people would be very, very sad if it was so.
    - jacquesm 12y ago
      I disagree an AST would help with a project this size, it's just unmanageable. You'd be better off to start with just the build scripts and build tools. ASTs are great for increasing understanding of much smaller projects but for something this size you'd likely end up with very little to show for your effort except the crashlogs of your tools. You need to go 'coarse' before you can go 'fine' on something this magnitude. This is not a 3 week project, just mapping the thing properly will take (man)years.
      - klibertp 12y ago
        > something this magnitude Yeah, I started commenting before the realization of how HUGE this thing would be hit me, sorry :)
        jacquesm 12y ago
        np. I think your sorry would be better directed at this guy: https://news.ycombinator.com/item?id=8257519 https://news.ycombinator.com/item?id=8257519
    - eklavya 12y ago
      Yeah I only came to know about AST around a year back. I guess I am not just that competent but I can not imagine someone trying to make sense of a codebase using ASTs. Mostly it's read the code/modify the code/debug the code for me. I have written some Scheme and I still can't say I need to screw around with ASTs. May be I will be enlightened some day? A UML diagram for this level of hugeness would be a really useful thing according to me, much much better than an AST.
      - klibertp 12y ago
        I probably overreacted, sorry for this - working on programming languages and programming tools is not what most of programmers do (obviously) and that's the only domain where you need to know about what you can use AST for. As for this: > A UML diagram for this level of hugeness would be a really useful thing we actually agree 100% here. What I mean is that having AST is meaningless by itself, but you need AST if you want to generate UML diagram from the code. Or generate a callgraph. Or find similarities or duplication in the code. Or indeed perform any kind of automatic code transformation. So extracting AST is a first step to developing your own tools for working with a codebase. And with a codebase of this size you just have to write your own tools, adapted to the nature of this particular codebase. So while "trying to make sense using ASTs" really is a bit hard to imagine, trying to make sense of a codebase using all the tools AST enables you to write is what I had in mind.

jacquesm 12y ago

AST = abstract syntax tree. With 100M lines your process will take many years.

eklavya 12y ago
Am I missing something or are you saying that making an AST like a compiler will help you understand a huge codebase better and faster?
- jacquesm 12y ago
  No, an AST will not help but a call graph certainly would (it shows you how the various routines are organized in graph form, who calls who). An AST for 100M lines would be absolute madness, a call graph just might work and I'm somewhat hoping that it turns out to be either a ton of generated or duplicated code. I also wonder if the OP isn't out of his depth based on the question(s) asked.
- klibertp 12y ago
  "making an AST like a compiler" will give you a semantic - as contrasted with textual - understanding of the code. Especially with how much data Java encodes in its source, this makes it a very good base for running automated analysis and visualisation tools, perhaps written by yourself. In general having AST is always better than having a plain text file, unless you want to read it. But then you can easily dump AST back to text whenever you want. Yeah, making AST will help you analyse your codebase programmatically which in turn will let you understand the codebase better and faster. This is some very basic programming knowledge, I think. Or is it not? Some comments here don't know what AST even is - is this the state of PL knowledge in the mainstream? Lisp and Smalltalk people would be very, very sad if it was so.
  - jacquesm 12y ago
    I disagree an AST would help with a project this size, it's just unmanageable. You'd be better off to start with just the build scripts and build tools. ASTs are great for increasing understanding of much smaller projects but for something this size you'd likely end up with very little to show for your effort except the crashlogs of your tools. You need to go 'coarse' before you can go 'fine' on something this magnitude. This is not a 3 week project, just mapping the thing properly will take (man)years.
    - klibertp 12y ago
      > something this magnitude Yeah, I started commenting before the realization of how HUGE this thing would be hit me, sorry :)
      - jacquesm 12y ago
        np. I think your sorry would be better directed at this guy: https://news.ycombinator.com/item?id=8257519 https://news.ycombinator.com/item?id=8257519
  - eklavya 12y ago
    Yeah I only came to know about AST around a year back. I guess I am not just that competent but I can not imagine someone trying to make sense of a codebase using ASTs. Mostly it's read the code/modify the code/debug the code for me. I have written some Scheme and I still can't say I need to screw around with ASTs. May be I will be enlightened some day? A UML diagram for this level of hugeness would be a really useful thing according to me, much much better than an AST.
    - klibertp 12y ago
      I probably overreacted, sorry for this - working on programming languages and programming tools is not what most of programmers do (obviously) and that's the only domain where you need to know about what you can use AST for. As for this: > A UML diagram for this level of hugeness would be a really useful thing we actually agree 100% here. What I mean is that having AST is meaningless by itself, but you need AST if you want to generate UML diagram from the code. Or generate a callgraph. Or find similarities or duplication in the code. Or indeed perform any kind of automatic code transformation. So extracting AST is a first step to developing your own tools for working with a codebase. And with a codebase of this size you just have to write your own tools, adapted to the nature of this particular codebase. So while "trying to make sense using ASTs" really is a bit hard to imagine, trying to make sense of a codebase using all the tools AST enables you to write is what I had in mind.