9 ms·
Lessons from Creating a VSCode Extension with GPT-4
- kevinslin 3y agoAuthor here. Happy to answer any questions or comments. As a teaser for a followup: I've been able to create a version of this that is able to generate an extension in one shot without any issues. This is done by giving GPT more context on the expected project layout as well as a checklist of constraints in order to create a valid app. Its not hard to envision a future where all software projects can be scaffolded by a LLM
- doodlesdev 3y agoTangential, but which model did you use to generate the picture at the start of the article?
- kevinslin 3y agodalle2 with the following prompt: Humanoid robots working on putting chips together in a factory, digital art
- rpastuszak 3y agoRandom plug, but I made a front end for d2, that is 6-10x cheaper than using their site: https://dall-e.sonnet.io/ https://dall-e.sonnet.io/
- kevinslin 3y agonice! will check it out
- mftb 3y agoI appreciate your write-up, it was well done. How long did you spend on this? I'd be most interested to know how long, not counting writing the blog post.
- operator-name 3y agoLooking forward to the follow up. I assume you set up a more capable gpt agent that can execute commands?
- kevinslin 3y agosame agent. just more constraints around the domain (of creating a vscode extension)
- ilrwbwrkhv 3y agoSo basically a waste of time. People are getting it wrong. It shouldn't be used for any generation. It should be used for compression.
- wahnfrieden 3y agoNah edit: simply check the update... managed to be done one-shot
- golol 3y agoI disagree. It is like 80% there.
- lcnPylGDnU4H9OF 3y ago"Prompt engineer" didn't exactly make sense to me until a coworker (graphic design) talked about the prompts that he'd see in Midjourney's discord server. Particularly when he mentioned specifications around camera lenses and perspective and other things I'm not familiar with. Very specific choices, continuing to be added and refined. Then seeing what guidance[0] is intended to do (LLM-prompt templating) it becomes obvious what people have in mind for this. It won't obviate understanding the lower-level code -- in fact, I expect the value of that skill to increase over time as fewer people bother to learn it -- but it will cut out the time it takes to scaffold an application. [0] https://github.com/microsoft/guidance https://github.com/microsoft/guidance
- ResearchCode 3y agoIf an LLM can learn to create a picture it's likely it can also learn to create a prompt.
- throwaway675309 3y agoIf anything, mid journey is an argument against prompt engineering. Each successive iteration of mid journey (currently at v5) has made it progressively easier and dispensed with the need for stable diffusion specific terminology.
- swyx 3y agovery cool to see usage of smol-developer in the wild! have been traveling so havent committed code but have a bunch of plans coming to level it up while staying smol. (see issues)
- kevinslin 3y agobig fan of smol-developer. it provides a great starting point for doing complicated things while being simple enough to reason about :)
- anotherpaulg 3y agoThe author starts out with an excellent observation: Lately, I've been playing around with LLMs to write code. I find that they're great at generating small self-contained snippets. Unfortunately, anything more than that requires a human... I have been working on this problem quite a bit lately. I put together a writeup describing the solution that's been working well for me: https://aider.chat/docs/ctags.html https://aider.chat/docs/ctags.html The problem I am trying to solve is that it’s difficult to use GPT-4 to modify or extend a large, complex pre-existing codebase. To modify such code, GPT needs to understand the dependencies and APIs which interconnect its subsystems. Somehow we need to provide this “code context” to GPT when we ask it to accomplish a coding task. Specifically, we need to: 1. Help GPT understand the overall codebase, so that it can decifer the meaning of code with complex dependencies and generate new code that respects and utilizes existing abstractions. 2. Convey all of this “code context” to GPT in an efficient manner that fits within the 8k-token context window. To address these issues, I send GPT a concise map of the whole codebase. The map includes all declared variables and functions with call signatures. This "repo map" is built automatically using ctags and enables GPT to better comprehend, navigate and edit code in larger repos. The writeup linked above goes into more detail, and provides some examples of the actual map that I send to GPT as well as examples of how well it can work.
- freedomben 3y agoI've been hearing executives and "tech leaders" recently saying that 80% of the new code is now written by chatgpt, and that it will "10x" a developer, but that sure mismatches with my experience. I suspect there will be a lot of managers with much higher expectations than is reasonable, which won't be good.
- visarga 3y agoIt's actually 1.2x productivity. Writing code does not take most of the day. If GPT can be just as good at debugging as it is at writing code, maybe the speedup would increase a bit. The ultimate AI speed == human reading + thinking speed, not AI generation speed.
- deleted 3y ago[deleted]
- pacifika 3y agoAsk it first how to break down the problem and then ask each step
- kevinslin 3y agoand keep enough shared context between the steps so that there is coherence
- jimsimmons 3y agoAnd to not reply in Klingon
- uludag 3y ago> To test GPT-4's ability to generate a complex program, ... I wonder how much the complexity of the various ecosystems we find ourselves in contribute to the lack of effectiveness of the language model. The task at hand really shouldn't be considered complex. Making and registering a command to do this in Emacs is essentially (defun inc-heading () (interactive) (save-excursion (search-backward-regexp "^\\# ") (insert "#"))). No project structure, no dependencies, no tooling: something a LLM should have no problem doing.
- esperent 3y agoThis is a great write-up, and I've had similar experiences in domain specific scaffolding (using GPT-4 directly rather than Smoll-AI and I was creating Odoo extensions rather than VSCode). That said, I think some of your takeaways are more criticisms of Smoll-AI than GPT-4: > there are no instructions on how to develop, use, or publish the extension ... Because you didn't ask for instructions. All it would take is one more prompt to get these. > there is no .gitignore for typescript/javascript/vscode artifacts > there is no launch.json file that configures running the extension in development Given the prompts it's fair to expect these should have been created. But again, since you already demonstrated some domain specific knowledge is required, when you see something like this is missing from the original prompt then you need to ask for it in a new prompt. > there are no tests I think we're seeing a pattern here. Did you ask for tests? In defense of the AI here, I think very few human coders would be writing tests in such an early stage of development. > there is no code reuse This one is fair. But I wonder if you fed all the original code in - I guess it would fit in one 32k context? - and asked it to find code reuse possibilities (and bugs while we're at it), what would you get?
- zdragnar 3y agoIt seems to me there are much better ways to get all of this stuff than chatting with an AI. Unless you already know what you want to do, relying on the AI seems like trying to dress yourself in the dark. If you do know what you want, use a more specific, better tool like (in this case) a project generator or starter template off of Github.
- esperent 3y agoIn my specific case, Odoo modules are pretty unusual, the default scaffolding is barely existant, the documentation is not great. I haven't found any useable alternative starter templates either. Beside that, it's also hard to use other Odoo modules s references because there's some weird thing where Odoo decided to recreate an old version of React that uses XML rather than JSX. They call it OWL. But they've only partially migrated so modules are mixed between that and I think at least two older styles. Oh, and some time in the past they decided to invent their own version of JS modules too. It's a mess. Oh yeah, and I'm using Odoo 16 while nearly everything else I can find is for Odoo 15, 14, or 13, and there have been a lot of technical changes between those versions. I've been programming professionally for about 15 years and this is one of the most tangled pieces of spaghetti I've ever had to unravel. There's just too many threads to easily keep them in my head at once, and very little in the way of tutorials. In this situation, for all it's faults, GPT-4 does a great job as an assistant. It genuinely is better than the alternatives I have available - even though it's also limited in knowledge of Odoo 16 which was released after the training date.
- andsoitis 3y agoThe most important, and which takes the most time, part of producing software is NOT the act of mapping solutions to code/architecture and writing said code. It is coming up with the solutions in the first place.
- linsomniac 3y agoThe author uses smol-developer for this. I've had mixed luck using smol-developer for actual development, but reading the code and prompts it's using, and just the general workflow is pretty fascinating and I've been playing around with adjusting it (Issue #34 https://github.com/smol-ai/developer/issues/34 https://github.com/smol-ai/developer/issues/34). In short: - smol-developer submits the user prompt and basically says: "Give me a list of files that would need to be written to accomplish this". - "Now give me a list of shared dependencies". - For each file: "Ok, write the code for $FILENAME, keeping in mind $SHARED_DEPENDENCIES" -- save file. One could imagine even breaking it further down ("what functions need to be in file X"), which might get it closer to what the author of this post says about GPT being better at smaller chunks. But you also kind of want some shared context as well. I broke out smol-developer's prompts and modified them to include "ask any clarifying questions you need to build the [list of files/dependencies/source]" and then was able to provide some more feedback during the development process. I also started off with that trick of "You are an AI prompt engineer, gather requirements and ask any follow up questions that are necessary to generate the perfect prompt, optimized to feed into GPT, to achieve the user's goals." trick, did some back and forth during which it asked some fascinating questions, including at least one that wasn't even on my radar. I then used that generated prompt in the following steps with smol-developer.
- RossBencina 3y ago> One could imagine even breaking it further down ("what functions need to be in file X") Based on my manual usage, I usually guide it through a top-down process, something like: - give an overview of how you would approach the problem - describe the main steps - do you see any problems with this approach? can you think of any alternatives? - taking into account everything that we have discussed so far, please carefully review and check your approach, propose a revised approach - only then would i get it to generate a list of source files and implement them one by one - now, given your implementation as a whole, review it and note any errors or improvements that could be made - fix the code ... Once I have code in hand I'll paste it into a new chat with a prompt like "You are our resident compiler guru. Please review the following code:..." It helps to be more specific about the reviewer's background and what you want. Rinse and repeat until it responds with high praise. Python works better than C++. GPT4 is much better than 3.5 of course.
- wodenokoto 3y agoI feel like GTP-4's training data cut-off is too old. I'm trying to get it to help me setup infrastructure in Azure. Blog posts describing my task are often from 2021 and drop down list and names of things simply don't match what they are now.