4 ms·

Are we anywhere close to being able to give a whole codebase to GPT-4 and ask various things about it?

by rane 3y ago

killthebuddha 3y ago
This of course depends entirely on how big the codebase is. But the short answer is "yes".
- smabie 3y ago
  Do you have any links / tools for this?
  - killthebuddha 3y ago
    Here's a couple (I apologize if some of these are already obvious to you): - https://python.langchain.com/docs/use_cases/code/code-analysis-deeplake - https://twitter.com/jerryjliu0/status/1636728577524916225?s=20 (GPT repo loader) FWIW there's tons of tools you could use to do this. GitHub Copilot is pretty close to being a comprehensive solution as well.
  - zora_goron 3y ago
    I wrote a simple ChatGPT plugin that exposes files within a local directory on my computer to ChatGPT [0]. It's been quite helpful for me, both for working with codebases as well as other material (supports docx and pdf as well). It does require plugin developer access to run, though. [0] https://github.com/samrawal/chatgpt-localfiles https://github.com/samrawal/chatgpt-localfiles
anotherpaulg 3y ago
My open source CLI tool can do this, as well as make changes to the code. Here's a chat transcript that shows how you can use it to explore and then modify a public github repo for a js game: https://aider.chat/examples/2048-game.html https://aider.chat/examples/2048-game.html Here a link to aider with more info: https://github.com/paul-gauthier/aider https://github.com/paul-gauthier/aider
- rane 3y ago
  Thanks for sharing. Looks very promising. I take it that asking things about the repository with gpt3.5-turbo-16k is not possible without adding each file separately? Just to test I generated ctags for my smallish Rust project and noticed it comes to around 10k tokens according to https://platform.openai.com/tokenizer https://platform.openai.com/tokenizer. I don't have GPT-4 API access yet but it appears it would get quite expensive with the 32k context.
  - anotherpaulg 3y ago
    Correct, right now aider does not provide the ctags repo map to any gpt-3.5 models. I still need to evaluate if those models can actually understand and use the map effectively. Improving and deepening support for the 3.5 models is probably my top priority next effort. Previously, they were barely useful for editing code. The 16k context window and functions support may have shifted that balance, so I need to dig in more. One other thing to note is that aider now distills the repo map to fit within a token budget. The default is 1k tokens, specified via `--map-tokens`. It tries to use the budget to convey the most important parts of the repo map to GPT. So in theory, even very large repos should get a useful map.

Are we anywhere close to being able to give a whole codebase to GPT-4 and ask various things about it?

by rane 3y ago

Are we anywhere close to being able to give a whole codebase to GPT-4 and ask various things about it?

killthebuddha 3y ago

This of course depends entirely on how big the codebase is. But the short answer is "yes".

smabie 3y ago
Do you have any links / tools for this?
- killthebuddha 3y ago
  Here's a couple (I apologize if some of these are already obvious to you): - https://python.langchain.com/docs/use_cases/code/code-analysis-deeplake - https://twitter.com/jerryjliu0/status/1636728577524916225?s=20 (GPT repo loader) FWIW there's tons of tools you could use to do this. GitHub Copilot is pretty close to being a comprehensive solution as well.
- zora_goron 3y ago
  I wrote a simple ChatGPT plugin that exposes files within a local directory on my computer to ChatGPT [0]. It's been quite helpful for me, both for working with codebases as well as other material (supports docx and pdf as well). It does require plugin developer access to run, though. [0] https://github.com/samrawal/chatgpt-localfiles https://github.com/samrawal/chatgpt-localfiles

anotherpaulg 3y ago

My open source CLI tool can do this, as well as make changes to the code. Here's a chat transcript that shows how you can use it to explore and then modify a public github repo for a js game: https://aider.chat/examples/2048-game.html https://aider.chat/examples/2048-game.html Here a link to aider with more info: https://github.com/paul-gauthier/aider https://github.com/paul-gauthier/aider

rane 3y ago
Thanks for sharing. Looks very promising. I take it that asking things about the repository with gpt3.5-turbo-16k is not possible without adding each file separately? Just to test I generated ctags for my smallish Rust project and noticed it comes to around 10k tokens according to https://platform.openai.com/tokenizer https://platform.openai.com/tokenizer. I don't have GPT-4 API access yet but it appears it would get quite expensive with the 32k context.
- anotherpaulg 3y ago
  Correct, right now aider does not provide the ctags repo map to any gpt-3.5 models. I still need to evaluate if those models can actually understand and use the map effectively. Improving and deepening support for the 3.5 models is probably my top priority next effort. Previously, they were barely useful for editing code. The 16k context window and functions support may have shifted that balance, so I need to dig in more. One other thing to note is that aider now distills the repo map to fit within a token budget. The default is 1k tokens, specified via `--map-tokens`. It tries to use the budget to convey the most important parts of the repo map to GPT. So in theory, even very large repos should get a useful map.