7 ms·
Chunking 2M files a day for code search using syntax trees
- deleted 3y ago[deleted]
- yding 3y agoThis is really cool and a much needed contribution to helping LLMs run better on large code bases.
- kevinlu1248 3y agoThanks! Would love to see this algorithm in LlamaIndex.
- thomasfromcdnjs 3y agoAlso thank you! I had an idea of how this all worked but this definitely concreted my thoughts.
- kevinlu1248 3y agoI'm glad to hear! Let me know if you need more questions or are interested in implementing this chunking algo.
- sandGorgon 3y agooh really ? Thats awfully kind. I'll take that in for EdgeChains as well. https://github.com/arakoodev/EdgeChains/issues/172 https://github.com/arakoodev/EdgeChains/issues/172
- kevinlu1248 3y agoFeel free to use the algo! Happy to help and reach out if you need help with implementation!
- kevinlu1248 3y agoFor others in this thread, this algo is now publicly accessible in LlamaIndex at https://github.com/jerryjliu/llama_index/blob/e567e6a20cf89bfdc7e1c62fd456648cc59d7480/llama_index/langchain_helpers/text_splitter.py#L476-L559C1 https://github.com/jerryjliu/llama_index/blob/e567e6a20cf89b...
- intalentive 3y agoNext step is to train models directly on syntax trees. Higher probability of correct output.
- kevinlu1248 3y agoThat's interesting, I've seen a few papers about this. I'm personally curious about editing syntax trees using language models, since it would prevent syntax errors altogether.
- vladf 3y agoWhich papers?
- kevinlu1248 3y agoHere's some of the papers: https://arxiv.org/abs/1911.09983 https://arxiv.org/abs/1911.09983 and https://aclanthology.org/2021.findings-acl.384.pdf https://aclanthology.org/2021.findings-acl.384.pdf
- madeofpalk 3y agoIn my limited use, I've never seen these models (chatgpt, and github copilot) generate invalid syntax. I don't see much to improve there I do see them generate code that fails the type checker though.
- kevinlu1248 3y agoFor editing code there's a decent chance of syntax errors or undefined variables since it's only modifying a subset of the code.
- Zambyte 3y agoJohn McCarthy was right
- 3y ago
- deleted 3y ago[deleted]
- sdesol 3y agoCongrats! Your project is off to a very good start as shown at https://devboard.gitsense.com/sweepai https://devboard.gitsense.com/sweepai What is interesting to me is the sharp increase in forks, which is a good indicator that others will contribute code in the near future. Full Disclosure: This is my tool
- kevinlu1248 3y agoHey thanks for showing this dashboard. There's some crazy analytics in here, the tool looks awesome!
- sdesol 3y agoThanks. For privacy reasons, I'm not showing a lot, but in the future when auth is in place, I can show deeper insights for repo members.
- kevinlu1248 3y agoThat would be awesome. Let me know when it's out and I'd love to try it out.
- sdesol 3y agoSure. Update your profile for a way to reach out or send me an email (in my profile).
- kevinlu1248 3y agoShot an email. Thanks!
- alj032 3y agoI just wanted to say the site looks a little odd on mobile Edit: I guess it is just the app I am using, it looks fine on my mobile browser but odd in the app
- kartoolOz 3y agoWould see great improvments in retrieval accuracy by finetuning e5-base-v2 or the newer leaders on mteb benchmark.
- kevinlu1248 3y agoDefinitely. I prefer the sentence-transformers ones since they have been fine-tuned on codesearchnet. I'm also really excited about the latest gte models by Alibaba, their smallest model is the size of MiniLM L6 but beats MPNet.
- alchemist1e9 3y agoAwesome first step. Next is to figure out how to apply syntax trees for diffs and then train the LLM on code and diffs but all in syntax trees somehow.
- kevinlu1248 3y agoYup, saw a few papers about this over the past two years, using graph neural networks for code generation. There's also another thread below on this topic. Edit: Here's some of the papers: https://arxiv.org/abs/1911.09983 https://arxiv.org/abs/1911.09983 and https://aclanthology.org/2021.findings-acl.384.pdf https://aclanthology.org/2021.findings-acl.384.pdf
- alchemist1e9 3y agoVery interesting! Thank you. I’ll try to explain something I’m thinking, it comes down to a type of agglutination. https://en.wikipedia.org/wiki/Agglutination https://en.wikipedia.org/wiki/Agglutination The ASTs need to become sequences of tokens for LLMs to work well. Also the embedding space is related, this should all be specifically optimized for code, not based on general human language . Of course the description of the code would be english. But my point is that a subspace of the encoding should be specifically designed for an agglutination based sequential expression of ASTs. Not sure any of that makes sense to someone with more expertise in this space than me.
- kevinlu1248 3y agoThis is really interesting, will take a look. So is this basically English to code translation?
- alchemist1e9 3y agoYeah what I’m thinking about is that and how an LLM could use ASTs expressed in some type of compact encoding and with that code tied to english descriptions, perhaps it will learn really well the semantic cross space.
- wanderingmind 3y agoTangent, are there any other similar alternatives to sweep that is not restricted to github but can be installed in other places like gitlab, bitbucket or even self hosted.
- kevinlu1248 3y agoNot a great answer but we are open-source so forking us is an option.
- wanderingmind 3y agoYes I had that thought, but your license looked custom so was not sure if that was allowed.
- kevinlu1248 3y agoUpdate: this algo is now publicly accessible in LlamaIndex at https://github.com/jerryjliu/llama_index/blob/e567e6a20cf89bfdc7e1c62fd456648cc59d7480/llama_index/langchain_helpers/text_splitter.py#L476-L559C1 https://github.com/jerryjliu/llama_index/blob/e567e6a20cf89b...
- d4rkp4ttern 3y agocurious why it is under "langchain_helpers"... I assume there is nothing specific to langchain here?
- yding 3y agoCurrently all of LlamaIndex's text splitters were/are Langchain compatible.
- mellosouls 3y agoOT: I'm not clear on whether or not Sweep is fully open source - I mean, can you run it fully self-hosted (apart from the GPT4 engine obv), or is the repo essentially a client to a Sweep API/binary? Cool project btw!
- kevinlu1248 3y agoThanks! The repo is just the backend that runs the GitHub webhooks. We used to have a "chat with your code" client but stopped supporting it. Now it's only the GitHub interface with creating tickets and comments.
- mellosouls 3y agoThanks for the clarification.