8 ms·
The author starts out with an excellent observation: Lately, I've been playing around with LLMs to write code. I find that they're great at generating smal
by anotherpaulg 3y ago
The author starts out with an excellent observation:
Lately, I've been playing around with LLMs to write code. I find
that they're great at generating small self-contained snippets.
Unfortunately, anything more than that requires a human...
I have been working on this problem quite a bit lately. I put together a writeup describing the solution that's been working well for me:
https://aider.chat/docs/ctags.html https://aider.chat/docs/ctags.html
The problem I am trying to solve is that it’s difficult to use GPT-4 to modify or extend a large, complex pre-existing codebase. To modify such code, GPT needs to understand the dependencies and APIs which interconnect its subsystems. Somehow we need to provide this “code context” to GPT when we ask it to accomplish a coding task. Specifically, we need to:
1. Help GPT understand the overall codebase, so that it can decifer the meaning of code with complex dependencies and generate new code that respects and utilizes existing abstractions.
2. Convey all of this “code context” to GPT in an efficient manner that fits within the 8k-token context window.
To address these issues, I send GPT a concise map of the whole codebase. The map includes all declared variables and functions with call signatures. This "repo map" is built automatically using ctags and enables GPT to better comprehend, navigate and edit code in larger repos.
The writeup linked above goes into more detail, and provides some examples of the actual map that I send to GPT as well as examples of how well it can work.
- freedomben 3y agoI've been hearing executives and "tech leaders" recently saying that 80% of the new code is now written by chatgpt, and that it will "10x" a developer, but that sure mismatches with my experience. I suspect there will be a lot of managers with much higher expectations than is reasonable, which won't be good.
- visarga 3y agoIt's actually 1.2x productivity. Writing code does not take most of the day. If GPT can be just as good at debugging as it is at writing code, maybe the speedup would increase a bit. The ultimate AI speed == human reading + thinking speed, not AI generation speed.
- blowski 3y agoI’ve found it’s very useful for understanding snippets of code, like a 200-line function. But systems are more than “lots of 200-line functions” - there’s a lot of context hidden in flags, conditional blocks, data, Git histories. Maybe one day, we’ll be able to run it over all the Git histories, Jira tickets, Confluence documentation, Slack conversations, emails, meeting transcripts, presentations, etc. Until then, the humans will need to stitch it all together as best they can.
- darkerside 3y agoI can't think of a reason you couldn't specifically train an AI on your own large code base. After all, current LLMs are trained on effectively the entire internet.
- zappchance 3y agoUnless your documentation is 20~100x the size of your codebase and written in a conversational tone, the LLM won't be able to be asked any questions about it using English. If your only aim is to use it like Copilot, sure, it's useful.
- sandos 3y agoSo the first step is to let the LLM write the documentation.... :)
- still_grokking 3y agoSure. Because it understands the code so well.
- oceanplexian 3y agoYou might be able fine tune a model on pull requests if they have really high quality descriptions, high quality commit messages, and the code is well documented and organized.
- epolanski 3y agoMost of the people I know don't use ChatGPT anymore. Many do use copilot as it is sometimes handy but far from life changing.
- derwiki 3y agoI use it pretty regularly for things like “write me a typed retry decorator in python, and write tests for it” or “parallelize this for loop using a ThreadPoolExecutor”
- ChatGTP 3y agoI work with people who have used it almost daily since it came out. None of them have 10x’d anything. Even their open source ChatGPT related projects are stalled and not going anywhere. Not to say it can’t help but I’ve not yet encountered this mythical 10x boost. As others have observed, maybe most have stopped using it. Maybe Ray Kurzweil is kind of on the money about computer brain interfaces, that’s when the fun really starts.
- still_grokking 3y agoNo worry. As soon as 80%+ of those managers will be replaced by ChatGPT such nonsense claims will stop; as ChatGPT knows things better than most managers. /s
- drvdevd 3y agoUsing ctags for this purpose is genius. Thanks for posting!
- fnordpiglet 3y agoThis is awesome, thanks. I wonder if complementing it with a vector store of the full repo further assists.
- anotherpaulg 3y agoYa, vector search is certainly the most common hammer to reach for when you're trying to figure out what subset of your full dataset to share with an LLM. And you're right, it's probably a piece of the puzzle for coding with GPT against a large codebase. But I think code has such a semantically useful structure that we should probably try and exploit that as much as possible before falling back to "just search for stuff that seems similar". Check out the "future work" section near the end of the writeup I linked above. I have a few possible improvements to this basic "repo map" concept that I'm experimenting with now.
- fnordpiglet 3y agoNo I agree the distilled map is most useful in context. However I wonder if providing a vector store of the total code base amplifies the effect. You could also pull in vector stores of all dependencies as well. Regardless amazing work and looking forward to seeing your future work as outlined.
- bredren 3y agoI did not know what Ctags were, here is the explanation from exuberant Ctags: >Ctags generates an index (or tag) file of language objects found in source files that allows these items to be quickly and easily located by a text editor or other utility. A tag signifies a language object for which an index entry is available (or, alternatively, the index entry created for that object). https://ctags.sourceforge.net/whatis.html https://ctags.sourceforge.net/whatis.html
- anotherpaulg 3y agoGood point, ctags is a bit old school! Most IDEs use Language Server Protocol now for similar purposes. I added a few sentences of explanation and background on ctags to my writeup.
- cyrialize 3y agoI'm a big fan of ctags. My old Emacs setup utilized Ctags when all else failed. So if I wanted to find a reference for something it would use LSP, then ctags if LSP returned nothing.
- RossBencina 3y agoI've been thinking that there needs to be a LangChain or similar in-prompt Tool that allows the model to automatically query a Language Server Protocol server. Maybe your tool does everything that could be done that way anyhow. aider looks interesting, I will try it out.
- anotherpaulg 3y agoAbsolutely, LSP is strictly more capable than ctags for this purpose. I mention it in the future work section of the writeup I linked above. The main reason I started with ctags is because the `universal ctags` tool supports a ton of languages right out of the box. Each LSP server implementation tends to only support a single language. So it would be more work for users to find, install and stand up the LSP server(s) they need for their particular projects. I plan to try some experiments with LSP in the future.
- joenot443 3y agoThis is a game changer. I’ve been doing something similar, relatively manually. I’ll give this a spin and report back.
- anotherpaulg 3y agoGlad to hear you're going to give it a try. Let me know how it goes.
- irrational 3y agoHow much time did it take to prepare all of that for ChatGPT? Won’t you have to redo all of that work every time you ask for more help since code bases are not static? Would it take less time and effort to just write the code on your own?
- anotherpaulg 3y agoYa, it would be tedious to do all of this manually. I guess it wasn't clear, but all of this is part of my open source GPT coding tool called "aider". aider is a command-line chat tool that allows you to write and edit code with GPT-4. As you are chatting, aider automatically manages all of this context and provides it to GPT as part of the chat. It all happens transparently behind the scenes.
- deleted 3y ago[deleted]
- linsomniac 3y agoI keep reading that you can upload data to one of the GPT interfaces, I'm not sure if it's a ChatGPT plugin, or one of the OpenAI API, but I wonder if you can include some source code that you can then prompt based on... I've been kind of tricking GPT, admittedly I'm currently limited to gpt-3.5-turbo, because with the API you can submit whole conversations and ask for the next reply from the assistant. I don't know how much of that whole conversation it re-reads, but I'll do things like: - Give it a system prompt with guidelines. - Give it my user prompt, which contains a description of what I want it to do. - Give it an "assistant response" which I've faked which is like: "Can you tell me more about the database schema?" - Give it my answer, which is the database schema. - Give it another faked assistant response where it is asking me about the API endpoints I need. - Give it my endpoints... At this point, I then ask it to generate a response. I've found that if I try to combine all the above information into a single prompt, I often run into token limits.
- andsoitis 3y ago> needs to understand LLMs don’t understand. They don’t understand because they have no model of reality.
- gmerc 3y agoCan you prove that? Because understanding ultimately is the ability to map to abstractions and GPT4 and other models are certifiably able to abstract.
- jen729w 3y agoThe onus is on you to prove that they do understand. One may not simply claim that a machine ‘understands’ because it appears to. Now we’re actually in the much-misunderstood Occam’s razor: that the theory which introduces the fewest new assumptions is the most likely correct. The assumption that a machine ‘understands’ — is capable of such a thing — is new, and requires extreme justification.
- andsoitis 3y ago> Can you prove that? Yes. One can easily construct a conversation where it is obvious that ChatGPT doesn't understand.
- gmerc 3y agoYa that’s not how proof works. People can do the same with humans. Not understanding things happens to be not a sign for the lack of capacity of understanding in humans.
- jiggawatts 3y agoLLMs can put their embeddings into a vector database, extending their short-term memory to encompass an entire codebase, every support ticket, PR, etc... I haven't seen any turnkey solutions yet good enough to use for development, but it's coming with months, not years.
- still_grokking 3y ago> Convey all of this “code context” to GPT in an efficient manner that fits within the 8k-token context window. > To address these issues, I send GPT a concise map of the whole codebase. The map includes all declared variables and functions with call signatures. Even if you could encode a function signature in only one token (which isn't be possible, of course) this would only work for small single developer programs, and not large projects. Mid-sized frameworks / libs have often teens of millions of lines of code, and hundred thousands of function signatures. With 8k tokens to spend you go nowhere even close to a large code base…