4 ms·

I navigate codebases by cat'ing all the files together (prefixed with filename) and piping it into vim. I was shocked how much I learn by this seemingly-horrib

I navigate codebases by cat'ing all the files together (prefixed with filename) and piping it into vim. I was shocked how much I learn by this seemingly-horrible technique. For example, the python files that actually get deployed are often quite different from the ones in source control. For tensorflow, at least. I regularly read 1M+ lines of code this way. Not an exaggeration; vim scales, nothing else does.

Gehinnn 6y ago
If you read/scan 10 lines a second, you still need over 24 hours non-stop to read a 1M+ code base. I doubt a random file ordering is helpful! Especially if you lose code navigation features like "go to definition".
- sillysaurusx 6y ago
  You'd be surprised. Suppose I want to "go to definition" for a class named Saver. /^class Saver\> 19 times out of 20, this works. It's also instant; my vim will likely get me there faster than your IDE's go to definition functionality. (Looking at you, pycharm!) Here's my flow. >>> import tensorflow as tf >>> tf.train.Saver <class 'tensorflow.python.training.saver.Saver'> >>> from tensorflow.python.training import saver >>> saver <module 'tensorflow.python.training.saver' from '/usr/local/lib/python3.7/site-packages/tensorflow_core/python/training/saver.py'> Then I open /usr/local/lib/python3.7/site-packages/tensorflow_core/python/training/saver.py. Suppose I want to know: Where are all the places that Saver is used in all of tensorflow? time find . -type f -name '*.py' | xargs merge | ft py Vim: Reading from stdin... real 0m0.930s user 0m0.243s sys 0m0.414s /\<Saver( :%v//d Boop: https://i.imgur.com/EJ5bZSW.png https://i.imgur.com/EJ5bZSW.png Literally every usage of Saver in all of Tensorflow. So let's say you're interested in a specific line. This one, for example: # being added to the GLOBAL_VARIABLES collection, so that Saver() Boop: https://i.imgur.com/YSAzhCJ.png https://i.imgur.com/YSAzhCJ.png I did that by highlighting "being added to the GLOBAL_VARIABLES", ctrl-c, then pressing "u" to undo the :%v//d, then / followed by ctrl-v. That might sound hard, but with muscle memory I don't even think about it -- it's like explaining how you open a can of food. Do you really think about where you place your fingers, or the pressure of your nail on the flap of the can? No, you just open it up. Same thing here; it's automatic. Way faster than IDEs, and I get just as much (or more) info. I'd love to use pycharm, but the slowness keeps pushing me back to this technique.
  - fraculus 6y ago
    I agree that things like "Go To Definition" can be pretty bad (especially when a codebase has code that is auto-generated, but you haven't figured out how yet). But I'm curious, what benefits do you see in your approach over just doing a grep in the folder (or e.g. Ctrl+Shift+F "Find in Files" in Pycharm)?
    - disgruntledphd2 6y ago
      grep type approaches are really good when part of your application is generated as SQL strings to make tables (which is a pathology sadly common in most data science codebases).
  - mejutoco 6y ago
    Nothing wrong with this (big fan of vim myself). I can see the appeal and simplicity. I just want to add that in Intellij you can do Ctrl + Shift + R "Saver" and it will search in all files (as dumb text matches, not usages), plus optional checks on extensions, etc. This is pretty fast too, and quite convenient, since it has a preview for each file. Not saying it is better, but it is an alternative.
    - sillysaurusx 6y ago
      I'm happy you mentioned that, because this highlights a very important difference: Your IDE would show you all "Saver" files in some checkout of Tensorflow, right? But you usually don't want to see the latest version of Tensorflow. You want to see the current version that's installed, which is somewhere under /usr/lib. I haven't found any IDE that can easily and effortlessly do that.
  - minusf 6y ago
    i do something similar but with tags and without the merging. vim's tag navigation is very powerful. split a window with the target tag and see side by side the class and its descendant for example. i use `venv`s for every project inside the project folder so uctags also generates tags for all libraries installed and i can "drill all the way up" to classes and definitions. it's also possible to have the system libraries show up in the tag database, it's just a matter of telling uctags which path's, files to include/exclude or alternatively use another tag file for that, vim can use multiple tag files. the real pain point is keeping the tag file update. gutentags makes this a bit more easy for me.
- disgruntledphd2 6y ago
  With the language server protocol, this works reasonably well in Python code bases for me in Emacs.
  - jidiculous 6y ago
    Yep, I didn't see why the author decided to snipe at Emacs and Vim at the end of the piece. Maybe 4 or 5 years ago, that criticism of Emacs would have been valid, but packages like flycheck and lsp-mode have brought Emacs at least to parity with many IDEs. In terms of Git alone, Magit has catapulted Emacs past any other editor, IDE, or Git GUI client.
coderdd 6y ago
I have great success using Zoekt, a Google codesearch reincarnation, for instant grepping.

Gehinnn 6y ago

If you read/scan 10 lines a second, you still need over 24 hours non-stop to read a 1M+ code base. I doubt a random file ordering is helpful! Especially if you lose code navigation features like "go to definition".

sillysaurusx 6y ago
You'd be surprised. Suppose I want to "go to definition" for a class named Saver. /^class Saver\> 19 times out of 20, this works. It's also instant; my vim will likely get me there faster than your IDE's go to definition functionality. (Looking at you, pycharm!) Here's my flow. >>> import tensorflow as tf >>> tf.train.Saver <class 'tensorflow.python.training.saver.Saver'> >>> from tensorflow.python.training import saver >>> saver <module 'tensorflow.python.training.saver' from '/usr/local/lib/python3.7/site-packages/tensorflow_core/python/training/saver.py'> Then I open /usr/local/lib/python3.7/site-packages/tensorflow_core/python/training/saver.py. Suppose I want to know: Where are all the places that Saver is used in all of tensorflow? time find . -type f -name '*.py' | xargs merge | ft py Vim: Reading from stdin... real 0m0.930s user 0m0.243s sys 0m0.414s /\<Saver( :%v//d Boop: https://i.imgur.com/EJ5bZSW.png https://i.imgur.com/EJ5bZSW.png Literally every usage of Saver in all of Tensorflow. So let's say you're interested in a specific line. This one, for example: # being added to the GLOBAL_VARIABLES collection, so that Saver() Boop: https://i.imgur.com/YSAzhCJ.png https://i.imgur.com/YSAzhCJ.png I did that by highlighting "being added to the GLOBAL_VARIABLES", ctrl-c, then pressing "u" to undo the :%v//d, then / followed by ctrl-v. That might sound hard, but with muscle memory I don't even think about it -- it's like explaining how you open a can of food. Do you really think about where you place your fingers, or the pressure of your nail on the flap of the can? No, you just open it up. Same thing here; it's automatic. Way faster than IDEs, and I get just as much (or more) info. I'd love to use pycharm, but the slowness keeps pushing me back to this technique.
- fraculus 6y ago
  I agree that things like "Go To Definition" can be pretty bad (especially when a codebase has code that is auto-generated, but you haven't figured out how yet). But I'm curious, what benefits do you see in your approach over just doing a grep in the folder (or e.g. Ctrl+Shift+F "Find in Files" in Pycharm)?
  - disgruntledphd2 6y ago
    grep type approaches are really good when part of your application is generated as SQL strings to make tables (which is a pathology sadly common in most data science codebases).
- mejutoco 6y ago
  Nothing wrong with this (big fan of vim myself). I can see the appeal and simplicity. I just want to add that in Intellij you can do Ctrl + Shift + R "Saver" and it will search in all files (as dumb text matches, not usages), plus optional checks on extensions, etc. This is pretty fast too, and quite convenient, since it has a preview for each file. Not saying it is better, but it is an alternative.
  - sillysaurusx 6y ago
    I'm happy you mentioned that, because this highlights a very important difference: Your IDE would show you all "Saver" files in some checkout of Tensorflow, right? But you usually don't want to see the latest version of Tensorflow. You want to see the current version that's installed, which is somewhere under /usr/lib. I haven't found any IDE that can easily and effortlessly do that.
- minusf 6y ago
  i do something similar but with tags and without the merging. vim's tag navigation is very powerful. split a window with the target tag and see side by side the class and its descendant for example. i use `venv`s for every project inside the project folder so uctags also generates tags for all libraries installed and i can "drill all the way up" to classes and definitions. it's also possible to have the system libraries show up in the tag database, it's just a matter of telling uctags which path's, files to include/exclude or alternatively use another tag file for that, vim can use multiple tag files. the real pain point is keeping the tag file update. gutentags makes this a bit more easy for me.
disgruntledphd2 6y ago
With the language server protocol, this works reasonably well in Python code bases for me in Emacs.
- jidiculous 6y ago
  Yep, I didn't see why the author decided to snipe at Emacs and Vim at the end of the piece. Maybe 4 or 5 years ago, that criticism of Emacs would have been valid, but packages like flycheck and lsp-mode have brought Emacs at least to parity with many IDEs. In terms of Git alone, Magit has catapulted Emacs past any other editor, IDE, or Git GUI client.

coderdd 6y ago

I have great success using Zoekt, a Google codesearch reincarnation, for instant grepping.