7 ms·
Yek: Serialize your code repo (or part of it) to feed into any LLM
- hbornfree 2y agoThanks for this! I have the exact use-case and have been using a Python script to do this for a while.
- TheTaytay 2y agoThis has some interesting ideas that I hadn’t seen in the other similar projects, especially around trying to sort files according to importance. (I’ve been using RepoPrompt for this sort of thing lately.)
- msoad 2y agoThis is really fast! Serialized 50k lines in 500ms on my Mac
- linschn 2y agoThat's neat ! I've built a transient UI to do this manually[0] within emacs, but with the context windows getting bigger ang bigger, being more systematic may be the way to go. The priorization mentioned in the readme is especially interesting. [0] https://rdklein.fr/bites/MyTransientUIForLocalLLMs.html https://rdklein.fr/bites/MyTransientUIForLocalLLMs.html
- widdershins 2y agoI'm a gptel addict, so I'll be looking at your approach. Thanks for posting it.
- pagekicker 2y agoError: yek: SHA256 mismatch Expected: 34896ad65e8ae7c5e93d90e87f15656b67ed5b7596492863d1da80e548ba7301 Actual: 353f4f7467af25b5bceb66bb29d9591ffe8d620d17bf40f6e0e4ec16cd4bd7e7 File: /Users/... Library/Caches/Homebrew/downloads/0308e13c088cb787ece0e33a518cd211773daab9b427649303d79e27bf723e0d--yek-x86_64-apple-darwin.tar.gz To retry an incomplete download, remove the file above. Removed & tried again this was the result. Is the SHA256 mismatch a security concern?
- mohsen1 2y agoOh totally forgot about homebrew installer. I'll fix it ASAP. Sorry about that. Edit: Working on a fix here https://github.com/bodo-run/yek/pull/14 https://github.com/bodo-run/yek/pull/14 You can use the bash installer on macOS for now. You can read the installer file before executing it if you're not sure if it is safe
- awestroke 2y agoThis looks promising. Hopefully much faster and less naive than Repomix
- mkagenius 2y agoI am doing something similar for my gitpodcast project: def get_important_files(self, file_tree): # file_tree = "api/backend/main.py api.py" # Send the prompt to Azure OpenAI for processing response = openai.beta.chat.completions.parse( model=self.model_name, messages=[ {"role": "system", "content": "Can you give the list of upto 10 most important file paths in this file tree to understand code architechture and high level decisions and overall what the repository is about to include in the podcast i am creating, as a list, do not write any unknown file paths not listed below"}, # Initial system prompt {"role": "user", "content": file_tree} ], response_format=FileListFormat, ) try: response = response.choices[0].message.parsed print(type(response), " resp ") return response.file_list except Exception as e: print("Error processing file tree:", e) return [] 1. https://gitpodcast.com https://gitpodcast.com - Convert any GitHub repo into a podcast.
- mg 2y agoI think this is where the future of coding is. It is still useful to be a coder, the more experienced the better. But you will not write or edit a lot of lines anymore. You will organize the codebase in a way AI can handle it, make architectural decisions and organize the workflow around AI doing the actual coding. The way I currently do this is that I wrote a small python file that I can start with llmcode.py /path/to/repo Which then offers a simple web interface at localhost:8080 where I can select the files to serialize and describe a task. It then creates a prompt like this: Look at the code files below and do the following: {task_description} Output all files that you need to change in full again, including your changes. In the same format as I provide the files below, that means each file starts with filename: and ends with :filename Under no circumstances output any other text, no additional infos, no code formatting chars. Only the code in the given format. Here are the files: somefile.py: ...code of somefile.py... :somefile.py someotherfile.py: ...code of someotherfile.py... :someotherfile.py assets/css/somestyles.css: ...code of somestyles.css... :assets/css/somestyles.css etc Then llmcode.py sends it to an LLM, parses the output and writes the files back to disk. I then look at the changes via "git diff". It's quite fascinating. I often only make minor changes before accepting the "pull request" the llm made. Sometimes I have to make no changes at all.
- shatrov1 2y agoWould you be kind to share your script? Thanks!
- mike_hearn 2y agoThere is an open source project called Aider that does that.
- KronisLV 2y ago> You will organize the codebase in a way AI can handle it, make architectural decisions and organize the workflow around AI doing the actual coding. This might sound silly, but I feel like it has the potential of resulting in more readable code. There have been times where I split up a 300 line function just so it’s easier to feed into an LLM. Same for extracting things into smaller files and classes that individually do more limited things, so they’re easier to change. There have been times where I pay attention to the grouping of code blocks more or even leave a few comments along the way explaining the intent so LLM autocomplete would work better. I also pay more attention to naming (which does sometimes end up more Java-like but is clear, even if verbose) and try to make the code simple enough to generate tests with less manual input. Somehow when you understand the code yourself and so can your colleagues (for the most part) a lot of people won’t care that much. But when the AI tools stumble and actually start slowing you down instead of speeding you up and the readability of your code results in a more positive experience (subjectively) then suddenly it’s a no brainer.
- wiradikusuma 2y agoSorry if it's not very obvious, where does Yek fit with existing coding assistants such as Copilot or Continue.dev? Is it purpose-built for code, or any text (e.g., Obsidian vault) would work?
- mohsen1 2y agoThis can be a piece of your own AI automation. Every task has a different need so being able to program your own AI automation is great for programmers. Any text based document works with this tool. It's rather simple, just stitching fils together with a dash of priority sorting
- CGamesPlay 2y agoWhat is the use-case here? What is a "chunk"? It looks like it's just an arbitrary group of files, where "more important" files get put at the end. Why is that useful for LLMs? Also, I see it can chunk based on token count but... what's a token? ChatGPT? Llama? Note, I understand why code context is important for LLMs. I don't understand what this chunking is or how it helps me get better code context.
- mohsen1 2y agotoken counting is done by crate that I'm using. I agree that not all LLMs use the same tokenizer but they are mostly similar. Chunking is useful because in chat mode you can feed more than context max size if you feed in multiple USER messages LLMs pay more attention to the last part of conversation/message. This is why sorting is very important. Your last sentence in a very long prompt is much more important the first. Use case: I use this to run an "AI Loop" with Deepseek to fix bugs or implement features. The loop steers the LLM by not letting it go stray in various rabbit holes. Every prompt reiterates what the objective is. By loop I mean: Serialize repo, run test, feed test failure and repo to LLM, get a diff, apply the diff and repeat until the objective is achieved.
- CGamesPlay 2y agoGot it, thanks. > in chat mode you can feed more than context max size if you feed in multiple USER messages Just so you know, this is false. You might be using a system that automatically deletes or summarizes older messages, which would make you feel like that, and would also indicate why you feel that the sorting is so important (It is important! But possibly not critically important). For future work, you might be interested in seeing how tools like Aider do their "repo serializing" (they call it a repomap), which tries to be more intelligent by only including "important lines" (like function definitions but not bodies).
- kruxigt 2y ago[dead]
- ycombiredd 2y agoi guess I shouldn’t be surprised that many of us have approached this in different ways. it’s neat to see already multiple replies of the sort I’m going to make too, which is to share the approach I’ve been taking, which is to concatenate or to “summarize” the code, with particular attention on dependency resolution. [chimeracat](https://github.com/scottvr/chimeracat https://github.com/scottvr/chimeracat) It took the shape that it has because it started as a tool to concatenate a library i had been working on into a single ipynb file so that I didn’t need to install the library on the remote colab, thus the dependency graph was born (as was the ascii graph plotter ‘phart’ that it uses) and then as I realized this could be useful to share code with an LLM, started adding the summarization capabilities, and in some sort of meta-recursive-irony, worked with Claude to do so. :-) I’ve put a collection of ancillary tools I use to aid in the pairing with LLM process up at https://github.com/scottvr/LLMental https://github.com/scottvr/LLMental
- sitkack 2y agoThis is hilarious https://github.com/scottvr/retree https://github.com/scottvr/retree
- ycombiredd 2y agowhile I hope you mean it is hilarious in the same spirit that I write most of my stuff (“ludicrous” is a common phrase even in my documentation), I did want to ask that if you meant that in more of a disparaging way, that you could flesh out any criticism. Of course, if you meant “hilarious” similarly to how I mean “ludicrous”, thanks! And thank you for taking the time to look at it. :-)
- sitkack 2y agoNot disparaging in anyway, in fact, it made made me think of bijective lenses and what other tools could have an inverse, even if lossy. Your project is amazing, you are ahead of me in your thinking.
- 2y ago
- mohsen1 2y agoAdded some benchmarking to show how fast it is: Here is a benchmark comparing it to [Repomix][1] serializing the Next.js project: time yek Executed in 5.19 secs fish external usr time 2.85 secs 54.00 micros 2.85 secs sys time 6.31 secs 629.00 micros 6.31 secs time repomix Executed in 22.24 mins fish external usr time 21.99 mins 0.18 millis 21.99 mins sys time 0.23 mins 1.72 millis 0.23 mins yek is 230x faster than repomix [1] https://github.com/jxnl/repomix https://github.com/jxnl/repomix
- amelius 2y agoMaybe I don't understand the usecase but I'm curious why speed matters given that LLMs are so slow (?)
- kruxigt 2y ago[dead]
- yani 2y agoYou can do this all in the browser: https://dropnread.io/ https://dropnread.io/
- Alifatisk 2y agoThere is also https://repo2txt.simplebasedomain.com/local.html https://repo2txt.simplebasedomain.com/local.html which doesn't require to download anything
- endofreach 2y agoI have a very simple bash function for this (filecontens), including ignoring files based on gitignore & binary files etc. Piped to clipboard and done. All these other ways seem unnecessarily complicated...
- imiric 2y agoI also feel like this can be done in a few lines of shell script. Can you share your function, please?
- verghese 2y agoHow does this compare to a tool like RepoPrompt? https://repoprompt.com https://repoprompt.com
- lordofgibbons 2y agoDoes anyone know of a more semantically meaningful way of chunking code in a generalizable way? Token count seems like it'd leave out meaningful context, or include unrelated context.
- energy123 2y agoI have an LLM summarize each file, and feed in a list of summaries categorized by theme, along with the function signatures and a tree of the directory. I only paste the full code of the files that are very important for the task.
- sitkack 2y agoI have to add https://github.com/simonw/files-to-prompt https://github.com/simonw/files-to-prompt as a marker guid. I think "the part of it" is key here. For packaging a codebase, I'll select a collection of files using rg/fzf and then concatenate them into a markdown document, # headers for paths ```filetype <data>``` for the contents. The selection of the files is key to let the LLM focus on what is important for the immediate task. I'll also give it the full file list and have the LLM request files as needed.
- zurfer 2y agoHas anyone build a linter that optimizes code for an LLM? The idea would be to make it more token efficient and (lower accidental perplexity), e.g. by renaming variable names, fixing typos and shortening comments. It should probably run after a normal linter like black.
- grajaganDev 2y agoGood idea - couldn't the linter also be an LLM?
- layer8 2y ago> accidental perplexity I have to remember this.
- foxhop 2y ago`tree --gitignore && cat .py && cat templates/`
- msoad 2y agowhy Dropbox when you can rsync, huh? ;)
- claireGSB 2y agoAdding my take to the mix, which has been working well for me: https://github.com/ClaireGSB/project-context https://github.com/ClaireGSB/project-context. It outputs both a file tree of your repo, a list of the dependancies, and a select list of files you want to include in your prompt for the LLM, in a single xml file. The first time you run it, it generates a .project-context.toml config file in your repo with all your files commented out, and you can just uncomment the ones you want written in full in the context file. I've found this helps when iterating on a specific part of the codebase - while keeping the full filetree give the LLM the broader context; I always ask the LLM to request more files if needed, as it can see the full list. The files are not sorted by priority in the output though, curious what the impact would be / how much room for manual config to leave (might want to order differently depending on the objective of the prompt).
- jamesponddotco 2y agoLike many people, I built something similar, llmctx[1], but the chunking feature seems really interesting, I have to look into that for llmctx. One thing I have with llmctx that I think is missing in yek is a “Claude mode”, as it outputs the concatenation in a format more suitable to provide context for the Anthropic LLM. [1]: https://sr.ht/~jamesponddotco/llmctx/ https://sr.ht/~jamesponddotco/llmctx/
- kauegimenes 2y agoI have a feeling theres some native unix commands that should cover this. So looking at the current scope of this tool, i think it would need more features. Anyone has a bash script that covers this use case?
- HarHarVeryFunny 2y agoSure, find, cat and split would do the job.
- paradite 2y agoThere are a lot of them. I collected a list of cli tools that does this: https://prompt.16x.engineer/cli-tools https://prompt.16x.engineer/cli-tools I also built a GUI tool that does this: https://prompt.16x.engineer/ https://prompt.16x.engineer/
- gatienboquet 2y agohttps://gitingest.com/ https://gitingest.com/
- frays 2y agoInteresting thread with other links.
- johnisgood 2y agohttps://github.com/simonw/files-to-prompt/ https://github.com/simonw/files-to-prompt/ seems to work fine especially for Claude as it follows the recommendations and format.
- Bao12534 2y agobảo
- neuraldenis 2y agoI made quite similar web tool some time ago, client side, no registration required: https://shir-man.com/txt-merge/ https://shir-man.com/txt-merge/
- dwrtz 2y agoi wrote something similar https://github.com/dwrtz/sink https://github.com/dwrtz/sink it has some nice features: - follows .gitignore patterns - optional file watcher to regenerate the snapshot when files are changed - supports glob patterns for filter/exclude - can report token usage - can use a config yaml file if you have complicated filter/exclude patterns i find myself using it all the time now. it's great with claude projects
- __chefski__ 2y agoThe token estimator of this package is quite inaccurate, since it appears to just take the number of characters, minus whitespace. This can lead to it being overly conservative, which would degrade an LLM's performance. That said, it can be improved in subsequent versions to properly integrate with a model's tokenizer so it can know the true token count. https://github.com/bodo-run/yek/blob/17fe37fbd461a8194ff612e26692ef884981f136/src/lib.rs#L374 https://github.com/bodo-run/yek/blob/17fe37fbd461a8194ff612e...