Disclaimer: I'm the founder of GitSense (https://gitsense.com https://gitsense.com) that indexes code and Git history. Indexing and retrieving code at scale is

Disclaimer: I'm the founder of GitSense (https://gitsense.com https://gitsense.com) that indexes code and Git history. Indexing and retrieving code at scale is actually a really challenging problem due to the fact that there is a lot of code, on a lot of branches, in a lot forked repos. With GitSense, it doesn't even try to determine the authoritative source (repo/branch), since I personally think this is a lost cause, given current AI technology. With GitSense, everything is context driven, which is how you can reasonably remove duplication. To search, you have to define what branches/repositories to consider, which can a be a few to a few thousand. Note, once a search context has been defined, it can be reused, so this isn't something you have to create every time, if you want to search. I sort of envision a Yahoo type (the first incarnation) approach to searching for code. The basic idea is provide a curated search experience, where domain experts can share what they believe to be relevant branches to consider, for a given problem. Without some human intervention, I think duplication is a given and as you point out, can lead to useless results.