3 ms·

For helix, I notice that GitHub is listed as a data source, but there's nothing in the docs about this. I'd really love to see what a model trained on my common

by joshka 3y ago

For helix, I notice that GitHub is listed as a data source, but there's nothing in the docs about this. I'd really love to see what a model trained on my commonly used git repos (which generally are newer than The Stack etc), and in particular their commit history. Ideally these would make it easier for code completion to have the historical context as well as the current code to play with in determining what to write next. I often wonder how you'd go about organizing training data for a full historic github repo in a way that makes sense for training (or RAG)? The vast majority of the data is previous changes to the repo. I think this would generally mean that it would outweigh the current information and cause problems (i.e. old method names before refactoring etc.) Also, perhaps being able to expand that out to doing the same thing for a bunch of consumers of the library that I'm maintaining would be neat. Sprinkle in the PR and Issue history, docs website, API docs, and discord history and I think you'd have a helluva model.

binocarlos 3y ago
This is spot on, the thing we've not yet done is make it easy to import a repo(s) code and the associated metadata into a fine tuning session easily. > I often wonder how you'd go about organizing training data for a full historic github repo in a way that makes sense for training (or RAG)? This is the hard part :-) But you are right - it would be intriguing to see what the output of a fune-tuned & RAG model would look like for this use-case. We are currently experimenting with adding RAG alongside the fine tuned model (so it's both, not either or) to see if it produces better results. I will make sure we take a look at the gihub repo use case because it feels like that would be an interesting experiment to do! disclaimer: I work on Helix
- joshka 3y ago
  Reading through the dataprep stuff, I wonder if doing more RAG during the prep stage might help this sort of task on structured daa. E.g. pre-indexing related parts and using those to build summaries / QA pairs. I took a look at the current prompts that are very research focused ("professors" creating questions), and could extrapolate from that to a dev mindset nicely.

by joshka 3y ago

binocarlos 3y ago

This is spot on, the thing we've not yet done is make it easy to import a repo(s) code and the associated metadata into a fine tuning session easily. > I often wonder how you'd go about organizing training data for a full historic github repo in a way that makes sense for training (or RAG)? This is the hard part :-) But you are right - it would be intriguing to see what the output of a fune-tuned & RAG model would look like for this use-case. We are currently experimenting with adding RAG alongside the fine tuned model (so it's both, not either or) to see if it produces better results. I will make sure we take a look at the gihub repo use case because it feels like that would be an interesting experiment to do! disclaimer: I work on Helix

joshka 3y ago
Reading through the dataprep stuff, I wonder if doing more RAG during the prep stage might help this sort of task on structured daa. E.g. pre-indexing related parts and using those to build summaries / QA pairs. I took a look at the current prompts that are very research focused ("professors" creating questions), and could extrapolate from that to a dev mindset nicely.